ArXiv: 2501.19324

🎯 Pitch

Conventional speculative decoding stalls on complex reasoning because its strict unbiasedness requirement forces wasteful rollbacks whenever draft and target models disagree—yet RSD deliberately biases token acceptance toward high-reward outputs, achieving up to 4.4× fewer FLOPs and +3.5 accuracy gains by trading off exact distributional matching for reasoning-specific quality improvements.

1. Executive Summary

This paper introduces Reward-Guided Speculative Decoding (RSD), a novel framework that accelerates LLM inference for reasoning tasks by dynamically combining a lightweight draft model with a more capable target model, using a process reward model (PRM) to evaluate intermediate decoding steps and decide whether to accept draft outputs or invoke the target model for refinement (a threshold-based acceptance criterion rather than strict token-matching). Evaluated across challenging reasoning benchmarks—including MATH500, GSM8K, Olympiad Bench, and GPQA—using Qwen2.5-Math, Qwen2.5, and Llama-3.1 model families with Skywork-o1-Open-PRM as the reward model, RSD achieves up to 4.4× fewer FLOPs compared to decoding with the target model alone while improving accuracy over standard speculative decoding by up to +3.5 points on average, establishing that controlled bias toward high-reward outputs can simultaneously improve both efficiency and reasoning quality in a way that strict unbiasedness cannot match.

2. Context and Motivation

The Core Problem: Speculative Decoding's Efficiency Paradox on Reasoning Tasks

The fundamental problem this paper tackles is that speculative decoding—the dominant paradigm for accelerating LLM inference—is poorly suited to long-horizon reasoning tasks like mathematical problem-solving, where generating a single solution requires many sequential steps of multi-step logical deduction. The issue is not that speculative decoding fails to produce correct outputs (it is theoretically unbiased and therefore lossless in quality), but rather that its efficiency gains collapse when the draft model and target model diverge systematically in their output distributions, which is precisely what happens on complex reasoning tasks.

To understand why this matters, we need to appreciate what speculative decoding achieves in standard text generation. In the canonical formulation (Leviathan et al., 2023; Chen et al., 2023a), a small draft model proposes candidate tokens autoregressively, and a large target model verifies them in parallel—checking whether each proposed token would have been sampled under its own distribution. When the draft model's predictions align closely with the target model's, many tokens are accepted in a single forward pass, yielding substantial speedups. The key mathematical property is unbiasedness: the acceptance criterion is designed so that the final token distribution exactly matches what the large model would have produced alone. This guarantee makes speculative decoding a safe acceleration technique—you get the exact same quality as the large model, just faster.

However, this guarantee comes with a hidden cost that becomes severe on reasoning tasks. The acceptance mechanism operates at the token level: a draft token is rejected if its probability under the target model falls below a rejection threshold derived from the draft model's own confidence. When the two models have systematically different behaviors—for instance, when the draft model tends to produce plausible-sounding but mathematically incorrect reasoning steps—the rejection rate climbs. Each rejection means the target model must generate a new token from scratch, consuming the very computation that speculative decoding was designed to avoid. The paper's Figure 1 illustrates this concretely: in standard speculative decoding, "mismatched tokens" and "wrong draft tokens" force the target model to redo work, while RSD accepts useful draft outputs even when they don't exactly match.

This is not a minor edge case. The authors observe that "strict unbiasedness requirement... restricts flexibility in exploring diverse completions" (Section 1), and in practice, "high-quality tokens (e.g., those favored by a process reward) may still be rejected if their probabilities under the large model are too low, leading to wasted computation and negating potential speedups." On reasoning tasks, the mismatch between small and large model distributions is systematic rather than random—it reflects genuine capability differences in how the models approach multi-step reasoning—so the efficiency penalty compounds with every step in the reasoning chain.

The significance of this problem sits at the intersection of two major trends in contemporary LLM deployment:

First, inference costs dominate the total cost of ownership for production LLMs. While training a large model is a one-time capital expenditure (though substantial), serving billions of inference queries incurs recurring costs that grow with usage. The paper cites Patterson et al. (2021) to establish that "inference at scale is even costlier [than training], requiring vast computational infrastructure and energy to serve billions of queries." This is not merely an economic concern—it has direct environmental implications and creates deployment barriers for latency-sensitive applications. Any technique that reduces per-query FLOPs without sacrificing quality therefore has outsized practical impact.

Second, reasoning tasks are increasingly important as deployment use cases for LLMs. General-purpose models (like Qwen2.5, Llama-3.1) are increasingly evaluated on their ability to solve math problems (GSM8K, MATH500), reason through scientific questions (GPQA), and handle Olympiad-level competition problems—not merely because these benchmarks are challenging, but because they proxy for the kind of structured, multi-step thinking that real-world applications demand (financial analysis, legal reasoning, scientific research assistance). However, these tasks are precisely where speculative decoding underperforms because they require generating long, coherent reasoning trajectories where draft-target divergence is most pronounced.

The paper explicitly notes this gap: "Despite advancements in parallel decoding, speculative decoding remains underutilized for complex reasoning tasks, particularly multi-step generation" (Section 1). The word "underutilized" is carefully chosen—it's not that speculative decoding fails in principle on reasoning tasks, but that its efficiency advantages diminish to the point where practitioners may not bother deploying it, preferring instead to run the target model directly (which guarantees quality but at high cost) or to try other acceleration techniques that compromise quality.

This creates a deployment trilemma for reasoning workloads: you can have high quality (target model only, expensive), high efficiency (draft model only, unreliable), or unbiased acceleration (speculative decoding, efficient in theory but often not in practice on reasoning tasks). The paper positions RSD as breaking this trilemma by showing you can have both efficiency and quality—and sometimes even better quality than the target model alone—by introducing controlled, reward-guided bias.

Where Prior Approaches Fall Short

The paper identifies specific limitations in three categories of existing work:

Standard speculative decoding enforces unbiasedness at the cost of flexibility. The theoretical guarantee that the output distribution exactly matches the large model's is a double-edged sword. It provides a safety net—you can't accidentally degrade quality—but it also means the method is forced to reject any draft token that the target model wouldn't have produced with sufficient probability, even if that token is objectively correct or useful. On reasoning tasks, this is a critical flaw. The draft model might produce a reasoning step that is mathematically sound and advances toward the correct answer, but if the target model would have phrased it differently or approached the problem from a different angle, that step gets rejected. The paper gives a concrete scenario: "If a draft token is correct but does not match the large model's distribution exactly, strict rejection is counterproductive" (Section 1). Worse, the target model's own alternative step might actually be incorrect, leading to a final answer that is worse than what the draft model would have produced alone. This is the "efficiency paradox"—the mechanism designed to preserve quality ends up wasting computation on unnecessary corrections while potentially degrading accuracy.

A specific empirical manifestation appears in the results: the authors note that "in cases where a draft model outperforms the target model... SD's strict unbiasedness leads to worse performance compared to the draft model" (Section 3.1). This is a damning observation—speculative decoding's guarantee is that you get the target model's quality, but if the target model is worse than the draft on certain problem types, you're guaranteed to get worse results. The paper shows this concretely on CN Middle School 24 and College Math (Appendix B, Table B.1), where the draft model (1.5B Qwen2.5-Math) outperforms the target (7B Qwen2.5-Math), and SD's accuracy (73.3 and 46.9) falls between the two, while RSD (78.2 and 48.2) exceeds both.

Test-time scaling methods that rely on the draft model alone hit a capability ceiling. The paper evaluates majority voting (maj@N) and Best-of-N (with a PRM scoring N candidates and selecting the highest-scoring one) as baselines that try to extract maximum performance from a small model through sampling. The results in Table 2 are decisive: across all three model families tested (Qwen2.5-Math, Qwen2.5, Llama-3.1), these methods consistently underperform the target model alone by substantial margins—for example, in the Llama-3.1 setting, majority voting (maj@16) achieves only 30.8 average accuracy compared to 41.4 for the target model. The paper's interpretation is that "this finding highlights the importance of a larger model for reasoning tasks, as its performance cannot be easily matched by a smaller model with increased computation" (Section 3.1). In other words, scaling inference compute with a small model hits fundamental capability limits that no amount of sampling can overcome—the small model simply cannot produce correct solutions for hard problems, so selecting among its outputs cannot create capability where it doesn't exist. This mirrors findings from the test-time compute scaling literature (Snell et al., 2024) that compute can amplify existing capability but cannot generate it from scratch.

Search-based methods using PRMs (beam search, process Best-of-N) suffer from combinatorial explosion. The paper compares against beam search and process Best-of-N (where N candidate next-steps are sampled and the one with the highest PRM score is selected at each reasoning step). Table 3 shows that while these methods outperform the 1.5B base model alone (78.2% vs. 73.8% on MATH500 for beam search with beam size 4), they still dramatically underperform RSD (82.6% on MATH500), which uses the 1.5B draft but can invoke a 7B target model for difficult steps. The paper's diagnosis is incisive: "for certain complex or 'hard' reasoning steps, search-based methods struggle to find optimal solutions due to the combinatorial explosion of potential candidates, leading to suboptimal performance" (Section 3.2). Beam search over the space of possible reasoning steps with a small model is fundamentally limited by the quality of the search space itself—if the small model cannot generate the correct next step in its top-N proposals, no amount of search can find it. This is the same "proposal distribution" bottleneck that Snell et al. (2024) analyzed in the context of PRM-guided search. RSD's insight is that when the draft model's proposal distribution fails, you should fall back to a larger model's generation capabilities rather than search more aggressively within the small model's limited space.

No existing method dynamically allocates compute between draft and target models based on output quality. This is perhaps the most fundamental gap. Standard speculative decoding always uses both models for every sequence—the draft proposes and the target verifies, with the target always engaged. Search-based methods always use only the draft model. Neither adapts to the specific difficulty of the reasoning step at hand. The paper's key conceptual move is to recognize that not all steps require the same level of capability—some steps are easy (the draft model gets them right with high confidence), and some are hard (the draft model produces low-quality or incorrect reasoning). A method that can route easy steps through the cheap draft model and hard steps through the expensive target model can achieve both better efficiency (fewer expensive inferences) and potentially better quality (the target model catches and corrects draft errors that search would miss).

How This Paper Positions Itself

The paper frames RSD as a controlled relaxation of speculative decoding's unbiasedness requirement that replaces strict token-probability matching with reward-guided acceptance. The core conceptual device is a dynamic mixture distribution: at each generation step, the output is sampled either from the draft model or from the target model, with the mixing weight determined by a reward function that evaluates the quality of the draft's output. This is formalized through the distribution:

PRSD(yi∣zi)=ω(r(yi∣zi))Pm(yi∣zi)+νPM(yi∣zi)P_{\text{RSD}}(y_i | z_i) = \omega(r(y_i|z_i)) P_m(y_i|z_i) + \nu P_M(y_i|z_i)

where ω(r)\omega(r) is a weighting function that maps the reward score (from a PRM evaluating the draft's proposed step) to a weight in [0,1][0,1], and ν\nu is a normalizing constant that ensures the target model serves as a reliable fallback. When the draft's step is high-quality (high reward), ω\omega approaches 1 and the draft distribution dominates, avoiding expensive target model invocation. When quality is low, ω\omega approaches 0 and the target model takes over.

This is a fundamentally different operating principle from standard speculative decoding. Standard SD asks: "Would the target model have produced this token?" RSD asks: "Is this reasoning step good, regardless of whether the target model would have produced it?" The shift from distributional matching to quality evaluation is what unlocks the ability to accept useful draft outputs that don't match the target's exact token preferences, and to reject plausible-sounding but incorrect draft steps even if they align with the target's distribution.

The paper positions this work at the intersection of two lines of research that have previously been pursued independently. On one side, speculative decoding and its variants (Leviathan et al., 2023; Chen et al., 2023a; Miao et al., 2024; Sun et al., 2024b; Li et al., 2024b; Xia et al., 2024; Fu et al., 2024) have focused on accelerating inference while maintaining exact distributional fidelity through draft-then-verify mechanisms, with innovations in tree-based speculation, self-speculation, and optimal transport-based acceptance criteria—but always within the unbiasedness framework and almost exclusively evaluated on standard text generation rather than multi-step reasoning. On the other side, PRM-guided reasoning (Lightman et al., 2023; Chen et al., 2024a; Snell et al., 2024; Brown et al., 2024) has used process rewards to score intermediate reasoning steps for search, verification, or selection during inference, but has focused on extracting maximum capability from a single model through test-time compute rather than on accelerating inference by combining models of different sizes.

The paper's positioning is thus that it is "a novel application of PRMs to accelerate reasoning during inference" (Section 5), bridging speculative decoding's efficiency focus with PRM-based reasoning's quality evaluation paradigm. This is a natural synthesis that, as of this paper's writing, had not been attempted in prior work—the authors note that "unlike previous speculative decoding methods, our approach utilizes process rewards to perform stepwise speculative reasoning" (Section 5).

The paper also explicitly situates itself relative to the broader efficiency landscape. Unlike quantization (Frantar et al., 2022; Lin et al., 2024), pruning (Sun et al., 2023), or KV-cache compression (Li et al., 2024a; Xu et al., 2024)—which reduce per-token computation—RSD reduces the number of tokens that must be processed by the expensive target model. This makes it complementary rather than competing: a quantized target model in RSD would compound both forms of efficiency. Similarly, unlike model merging or distillation, which produce a single model of intermediate quality and cost, RSD maintains both extremes (cheap draft, expensive target) and dynamically routes between them per step, allowing finer-grained allocation of compute.

A subtle but important positioning choice: the paper does not claim RSD is theoretically unbiased (it explicitly abandons that guarantee). Instead, it argues that controlled bias toward high-reward outputs is a feature, not a bug—it can lead to better accuracy than the target model alone if the reward signal correctly identifies good reasoning steps that the target model might have missed. Proposition 2.2 formalizes this: under mild conditions (the weighting function is non-decreasing in reward, and the target model has higher expected reward than the draft), RSD's expected reward is guaranteed to exceed the draft model's, and can exceed the target model's if the weighted draft distribution concentrates on higher-reward regions than the target alone would sample. The empirical results in Section 3 bear this out—RSD consistently outperforms the target model alone on average accuracy across benchmarks.

The paper also differentiates RSD from search-based methods that use PRMs (beam search, process Best-of-N) through a key architectural distinction: search exhaustively explores a small model's generation space; RSD selectively invokes a large model's generation space. Table 3 empirically validates that this distinction matters substantially on complex benchmarks, with RSD outperforming beam search by 4.4 points on MATH500 and 5.6 points on GSM8K. The paper's framing here connects to a broader theme in the test-time compute literature—the proposal distribution (what the model can generate) and the verifier (how outputs are scored) are complementary axes, and improving the proposal distribution by falling back to a larger model can be more effective than optimizing the verifier's search over a limited proposal space (Snell et al., 2024).

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems-and-algorithms paper that proposes a new inference-time decoding strategy combining ideas from speculative decoding and reward-guided reasoning. The core idea is to replace speculative decoding's strict token-level probability matching with a step-level quality check: instead of asking "would the target model have produced this token?", RSD asks "is this reasoning step good according to a process reward model, regardless of which model produced it?" This shift from distributional fidelity to reward-based acceptance is what enables simultaneous improvements in both efficiency (less unnecessary target model computation) and accuracy (accepting correct draft steps that the target model would have rejected, while rejecting incorrect draft steps that the target model might have accepted).

3.2 Big-Picture Architecture (Diagram in Words)

The RSD system has four major components that interact at every step of autoregressive generation:

  1. Draft Model ($m$) — a small, fast, inexpensive LLM (e.g., Qwen2.5-Math-1.5B or Llama-3.1-1B) that generates candidate reasoning steps by conditioning on the prompt and all previously accepted steps. It serves as the default proposal engine; its outputs are cheap but may contain errors.

  2. Process Reward Model (PRM) — a learned verifier (e.g., Skywork-o1-Open-PRM-7B) that takes the current context (prompt + previous steps) and the draft model's proposed next step, and outputs a scalar score $r \in [0, 1]$ estimating the quality of that step. It provides the signal that decides whether the draft step is good enough to keep.

  3. Target Model ($M$) — a large, powerful, expensive LLM (e.g., Qwen2.5-Math-72B or 7B) that serves as a fallback generator. When the PRM deems the draft step insufficiently good, the target model generates a replacement step from scratch, using its superior reasoning capability. The target model's output is always accepted—it is the "ground truth" fallback.

  4. Acceptance Criterion ($A_\omega$) — a decision rule (Algorithm 2 in the paper) that takes the PRM score $r$, applies a weighting function $\omega(r)$, and probabilistically decides whether to accept the draft step or invoke the target model. This is the core control mechanism that balances efficiency and quality.

Information flow at each step: The draft model proposes a candidate step → the PRM scores it → the acceptance criterion decides accept/reject → if accepted, the draft step is appended to the sequence; if rejected, the target model generates a replacement step which is appended instead → the process repeats for the next step until EOS or max length.

The critical difference from standard speculative decoding: In standard SD, acceptance depends on whether the target model would have sampled the draft token (probability ratio check). In RSD, acceptance depends on whether an external reward model judges the step to be good. This means RSD can accept a draft step that the target model would have rejected (if the step is correct but phrased differently) and can reject a draft step that the target model would have accepted (if the step is plausible but wrong). The reward model, not the distributional match, is the arbiter of quality.

3.3 Roadmap for the Deep Dive

  • First, the mathematical framework (Section 2): Understand the mixture distribution $P_{\text{RSD}}$, the weighting function $\omega(r)$, and the reward assumption—this is the theoretical foundation for why RSD works and what guarantees it provides.
  • Second, the acceptance criterion and the binary step function: Understand Algorithm 2 and why Proposition 2.3 proves that a threshold-based binary function is the optimal weighting strategy under a sampling budget constraint—this is the algorithmic core that connects theory to practice.
  • Third, the RSD algorithm itself (Algorithm 1): Walk through the per-step procedure, including how the draft step is generated, how the PRM scores it, how the acceptance decision is made, and how the target model fallback works.
  • Fourth, the experimental configuration: Understand what models are used for draft, target, and PRM; what hyperparameters control the system ($\delta$, temperature, step segmentation); and how the system is evaluated.
  • Fifth, the computational cost model (FLOPs): Understand how efficiency is measured and why RSD achieves up to 4.4× fewer FLOPs than the target model alone.
  • Sixth, design choices and their justifications: Why a binary step function? Why $\delta = 0.7$ as the default? Why step-level rather than token-level? Why an external PRM rather than self-verification? Why is the target model always accepted?

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems-and-algorithms paper whose core contribution is a novel inference-time decoding strategy that replaces speculative decoding's unbiased token-matching with reward-guided step-level acceptance, enabling both higher efficiency (fewer target model invocations) and higher accuracy (accepting good draft steps regardless of distributional match).

The Mixture Distribution: What RSD Fundamentally Computes

The central mathematical object in RSD is the distribution from which each reasoning step is sampled. At step $i$ of generation, given context $z_i = [x, y_{1:i-1}]$ (the prompt $x$ concatenated with all previously generated steps), the next step $y_i$ is drawn from:

PRSD(yi∣zi)=ω(r(yi∣zi)) Pm(yi∣zi)+ν PM(yi∣zi)P_{\text{RSD}}(y_i \mid z_i) = \omega(r(y_i \mid z_i)) \, P_m(y_i \mid z_i) + \nu \, P_M(y_i \mid z_i)

where $P_m(y_i \mid z_i)$ is the draft model's conditional distribution over the next step given context $z_i$; $r(y_i \mid z_i)$ is the reward function that scores the quality of step $y_i$ in context $z_i$, producing a scalar in $[0, 1]$; $\omega(r(y_i \mid z_i))$ is the weighting function that maps the reward score to a confidence weight in $[0, 1]$, dictating how much to trust the draft model for this specific output; $\nu$ is a normalizing constant defined as $\nu = 1 - \mathbb{E}_{y \sim P_m}[\omega(r(y \mid z_i))]$, ensuring that $P_{\text{RSD}}$ integrates to 1; and $P_M(y_i \mid z_i)$ is the target model's conditional distribution over the next step.

What it computes: The RSD distribution is a weighted mixture of two component distributions. The first component, $\omega(r) P_m$, is the draft model's distribution modulated by the reward—in regions where the draft output is high-quality (high reward), this term is close to $P_m$ (weight near 1), meaning the draft model's predictions dominate. The second component, $\nu P_M$, is the target model's distribution scaled by a constant $\nu$, ensuring that the target model always contributes some probability mass to the mixture. The constant $\nu$ is not arbitrary—it equals one minus the expected weight assigned to the draft model across all possible outputs, so the total probability mass sums to 1 (the draft's weighted distribution plus the target's contribution fills out the remaining probability).

Operationally, what happens at inference time: The distribution $P_{\text{RSD}}$ is not explicitly constructed—no one computes the full mixture over all possible steps. Instead, the algorithm samples from it via rejection sampling. A candidate step $\hat{y}_i$ is drawn from the draft model $P_m$. The weight $\omega(r(\hat{y}_i \mid z_i))$ is computed. With probability $\omega(r)$, the candidate is accepted (sampled from the first component). With probability $1 - \omega(r)$, the candidate is rejected, and a new step is drawn from the target model $P_M$ (sampled from the second component). This rejection sampling procedure produces samples exactly from $P_{\text{RSD}}$ without ever materializing the full distribution.

Why this form: The mixture formulation is chosen over alternatives for three reasons. First, it cleanly separates the roles of the draft and target models—the draft provides cheap proposals, and the target provides expensive fallbacks, with the reward function as the arbiter. Second, the additive mixture (as opposed to a multiplicative reweighting like importance sampling) ensures that the target model always has non-zero influence, preventing degenerate cases where the draft model's distribution is simply reweighted without the target ever contributing fresh generations. Third, the form is analyzed in Proposition 2.1 and Proposition 2.2 to have desirable properties: under the assumption that the target model achieves higher expected reward than the draft (Equation 1: $\mathbb{E}_{y \sim P_M}[r(y \mid z)] \geq \mathbb{E}_{y \sim P_m}[r(y \mid z)]$), and assuming $\omega(r)$ is non-decreasing in $r$, the expected reward under $P_{\text{RSD}}$ is guaranteed to be at least as high as the expected reward under $P_m$ alone (Proposition 2.2). This guarantee means RSD cannot make accuracy worse than just using the draft model—it either matches or improves.

The Weighting Function: How Reward Scores Map to Acceptance Probabilities

The weighting function $\omega(r)$ is the control mechanism that translates a reward score into a decision about how much to trust the draft model's output. The paper considers several functional forms (Table 1), but the primary variant used in all experiments is the binary step function:

ω(r)=1(r≥δ)\omega(r) = \mathbf{1}(r \geq \delta)

where $\mathbf{1}(\cdot)$ is the indicator function (1 if the condition is true, 0 otherwise), and $\delta \in [0, 1]$ is a threshold hyperparameter.

What it computes: For a given reward score $r$, the weighting function outputs 1 if $r \geq \delta$ (the draft step is good enough) and 0 otherwise (the draft step is insufficient). This is the simplest possible weighting scheme—it makes a hard binary decision rather than a soft interpolation. When $\omega(r) = 1$, the draft step is always accepted (the first component of the mixture dominates). When $\omega(r) = 0$, the draft step is always rejected and the target model is always invoked (the second component takes over).

Why the binary function is optimal (Proposition 2.3): The paper proves that under a constrained sampling budget—specifically, the constraint that the expected fraction of steps sampled from the target model $\nu = 1 - \mathbb{E}_{y \sim P_m}[\omega(r(y \mid z))]$ is bounded above by some $\gamma \in (0, 1)$—the optimal weighting function that maximizes expected reward is a binary step function. The intuition is clean: given a limited budget for expensive target model invocations, you want to use them exactly on the lowest-quality draft outputs (where they provide the most improvement), and conserve them on high-quality outputs (where the draft is already good). A continuous weighting function would waste some target model budget on intermediate-quality outputs where the marginal improvement is small, at the expense of not having budget left for very low-quality outputs where the improvement would be large. The optimal strategy is to set a threshold $\delta_\gamma$ (dependent on the budget $\gamma$) such that all draft outputs with reward below the threshold get the target model, and all above get the draft model.

Other weighting functions considered (Table 1): The paper lists five variants, though only the binary function is used in the main experiments:

  • Constant: $\omega(r) = p$ for some fixed $p \in (0, 1)$. This ignores the reward signal entirely and accepts draft steps with constant probability. Figure 6 shows this performs the worst, confirming that reward information is crucial.

  • Clipping: $\omega(r) = \min(1, \max(0, r))$. This clips the raw reward score to $[0, 1]$. It provides a linear relationship between reward and acceptance probability but doesn't distinguish sharply between good and bad steps.

  • Sigmoidal transformation: $\omega(r) = \max(0, \frac{r}{1+r})$. This maps rewards through a sigmoid-like curve, providing smooth interpolation but again without a sharp decision boundary.

  • Logistic function: $\omega(r) = \frac{1}{1 + e^{-\alpha(r - \delta)}}$. This is the classic sigmoid with tunable steepness $\alpha$ and center $\delta$, providing a smooth approximation to the binary step function.

The paper's empirical finding (Figure 6) is that the binary step function achieves the best accuracy at comparable inference cost, validating Proposition 2.3's theoretical prediction.

The Acceptance Criterion: How the Binary Decision Is Implemented

The acceptance criterion $A_\omega$ (Algorithm 2) is the stochastic mechanism that translates the weighting function's output into an accept/reject decision. Its logic depends on the value of $\omega(r)$:

  • If $\omega(r) = 0$: Reject deterministically (the draft step is always discarded, target model invoked).
  • If $\omega(r) = 1$: Accept deterministically (the draft step is always kept, target model not invoked).
  • If $0 < \omega(r) < 1$: Sample a uniform random variable $u \sim U(0, 1)$. Accept if $\omega(r) \geq u$, reject otherwise. This makes acceptance probabilistic with exactly the probability specified by the weighting function.

When the binary step function is used ($\omega(r) = \mathbf{1}(r \geq \delta)$): The weighting function only outputs 0 or 1, so Algorithm 2 becomes fully deterministic—it simply checks whether $r \geq \delta$. The paper notes this explicitly: "when a binary function is used, Algorithm 2 almost surely degenerates into a deterministic procedure, producing a definite acceptance or rejection outcome" (Section 2.2).

Why this matters for implementation: The binary function with a deterministic threshold makes RSD trivially simple to implement. There is no need for random number generation, no probability ratio computation, no calibration between draft and target model distributions. The system just runs the draft model, runs the PRM, compares the PRM score to $\delta$, and either keeps the draft step or runs the target model. This simplicity is a practical advantage over standard speculative decoding, which requires careful management of rejection sampling probabilities and can suffer from floating-point errors (Chen et al., 2023a, cited in the paper).

The probability of invoking the target model: Under the binary step function, the probability that the target model is called at a given step is exactly $\mathbb{E}_{y \sim P_m}[\mathbf{1}(r(y \mid z) < \delta)]$—the fraction of draft outputs whose reward falls below the threshold. This is directly controlled by $\delta$: a higher threshold means more steps are rejected, more target model invocations, higher computational cost, and potentially higher quality. A lower threshold means fewer rejections, less target model usage, lower cost, and potentially lower quality. Figure 5 empirically demonstrates this tradeoff.

The RSD Algorithm: Step-by-Step Execution

Algorithm 1 describes the full per-sequence generation loop. Let's trace through what happens for a single question:

Inputs: A prompt $x$, a draft model $m$, a target model $M$, a process reward model (PRM) $r$, the acceptance criterion $A_\omega$, an end-of-sequence token $s$, and a maximum sequence length $N$.

Initialization: The sequence of previously generated steps $y_{1:0}$ is set to empty.

Loop: For each step index $i$ from 1 to $N-1$:

  1. Generate draft step: The draft model $m$ produces a candidate next step $\hat{y}_i$ by conditioning on the prompt $x$ and all previously accepted steps $y_{1:i-1}$. This is a standard autoregressive generation from the small model—it outputs a complete reasoning step (defined as a sequence of tokens ending with \n\n, as specified in the default settings), not just a single token. This is a critical difference from standard speculative decoding: RSD operates at the step level (multi-token reasoning segments), not the token level.

  2. Compute reward: The PRM scores the candidate step. It takes as input the full context $[x, y_{1:i-1}]$ and the proposed step $\hat{y}_i$, and outputs a scalar $r_i \in [0, 1]$ estimating the quality of this step in context. The PRM evaluates whether the reasoning in $\hat{y}_i$ is correct, consistent with the previous steps, and likely to lead to the right final answer.

  3. Apply acceptance criterion: $A_\omega(r_i)$ is evaluated. For the binary step function with threshold $\delta$, this is simply $r_i \geq \delta$.

    • If accepted: The draft step $\hat{y}_i$ is assigned to $y_i$ and appended to the sequence. No target model computation occurs.
    • If rejected: The target model $M$ generates a new step $y_i$ from scratch, conditioning on the same context $[x, y_{1:i-1}]$. This step is always accepted—the target model serves as the infallible fallback. The rejected draft step is discarded and never used.

  4. Check for termination: If the end-of-sequence token $s$ appears in $y_i$, generation stops. Otherwise, the loop continues.

Output: The complete response $y_{1:i}$, consisting of the sequence of accepted steps.

What happens at the step level vs. token level: The paper explicitly defines a reasoning step as "a generation ended with \n\n" (Section 3, default setting). This means the draft model generates multiple tokens (an entire reasoning sentence or paragraph) before the PRM evaluates it. The PRM scores the entire multi-token step as a unit. This step-level grain is crucial because it allows the reward model to assess semantically meaningful chunks—individual tokens (e.g., "the", "=", "3") carry no reasoning quality signal, but complete sub-steps (e.g., "Step 1: Evaluate $i^5$. $i^5 = i$") do. Figure 1 illustrates this with step-level reward scores (0.6, 0.7, 0.9, 1.0) assigned to entire reasoning segments, not individual tokens.

Why the target model is never rejected: The paper's assumption (Equation 1) is that the target model strictly dominates the draft model in expected reward: $\mathbb{E}_{y \sim P_M}[r(y \mid z)] \geq \mathbb{E}_{y \sim P_m}[r(y \mid z)]$. The target model is treated as "sufficiently robust and reliable" (Section 2), so its outputs are always accepted without PRM evaluation. This is a design choice that simplifies the algorithm—if the target model were also subject to PRM checking, you could get into an infinite regress (if both are rejected, what generates the step?). In practice, the paper validates this assumption in Figure 3 (middle and right panels): for correctly answered questions within RSD, the target model consistently achieves higher reward scores than the draft model, and wins in reward comparison 62.7% of the time across all questions.

Relationship to standard speculative decoding: In standard SD, when a draft token is rejected, the target model generates a replacement token and then the draft model resamples from that point, meaning the draft model always conditions on target-generated tokens after a rejection. In RSD, the paper does not specify that the draft model conditions on the target's output—the target simply generates the replacement step and it is appended. The draft model then conditions on the entire history (including target-generated steps) when proposing the next step. This means target-generated steps become part of the permanent context, influencing all future draft proposals—a form of implicit error correction that propagates forward.

The Reward Model: What Gets Scored and How

The PRM is not trained in this paper—it is used off-the-shelf. The primary PRM is Skywork-o1-Open-PRM (available in 1.5B and 7B parameter versions), described as "the most advanced open-source PRM available during our experiments" (Section 3). The authors also test Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B (Zhang et al., 2025) for robustness analysis (Table 5).

What the PRM outputs: A scalar $r \in [0, 1]$ where higher values indicate better reasoning quality. The paper states "the reward score ranges from 0 to 1, the higher the better" (Section 3). The PRM evaluates the step in context—it sees the full prompt and all preceding steps, so it can assess whether the current step is logically consistent with what came before.

How the PRM is invoked: At each step in Algorithm 1, the PRM takes $r(y_i \mid x, y_{1:i-1})$—the candidate step $y_i$ conditioned on the original prompt $x$ and all previously generated steps $y_{1:i-1}$. This means the PRM's judgment is path-dependent: the same step could receive different scores depending on what reasoning preceded it. For example, a step that says "therefore $x = 3$" might score highly if the previous steps correctly derived that value, but poorly if the previous steps made an arithmetic error.

PRM overhead is minimal: The paper quantifies this in Section 4: "In MATH500, the average number of reasoning steps per question is 18, suggesting that the PRM is invoked 18 times per question, akin to generating 18 tokens." Since the PRM only needs to run a single forward pass to score a step (it doesn't generate anything), its computational cost is comparable to processing a small number of tokens—dwarfed by the cost of the target model generating full reasoning steps when invoked.

PRM size tradeoff (Table 2, discussed in Section 3.1): The paper compares a 1.5B PRM against a 7B PRM. The larger PRM generally improves accuracy (e.g., 84.6 vs. 82.6 on MATH500 for the 7B target, 1.5B draft setting), particularly on complex datasets like GPQA (33.8 vs. 38.4) and MATH500 (84.6 vs. 82.6). However, the 1.5B PRM still provides sufficient signal for RSD to outperform the target model alone (65.9 vs. 65.3 average), demonstrating that even a small reward model can effectively guide the accept/reject decisions.

Robustness to PRM choice (Table 5): The paper tests RSD with three different PRMs (Skywork-o1-Open-PRM-7B, Qwen2.5-Math-PRM-7B, Qwen2.5-Math-PRM-72B) and finds consistent improvements over SD and the target model alone. The average accuracy with the 72B PRM (66.5) is slightly higher than with the 7B PRMs (66.1, 65.9), but all substantially exceed SD (64.6). This robustness check is important because it demonstrates that RSD does not depend on a specific PRM—it works as long as the PRM provides a reasonable quality signal. The paper notes that "training or fine-tuning a specialized PRM that is closely aligned with the draft model could further enhance performance" (Section 4), suggesting that the off-the-shelf PRM represents a lower bound on what a tailored reward model could achieve.

Process reward vs. outcome reward for general-domain tasks: In Section 4, the paper explores using an outcome reward model (ORM) instead of a PRM for non-reasoning tasks where step-level annotation doesn't exist. On AlpacaEval with 805 prompts, using Skywork-Reward-Llama-3.1-8B-v0.2 as the ORM and treating text segments ending with \n\n as steps, RSD achieves a 18.85% win rate against GPT-4-Turbo, compared to 7.09% for the draft model alone and 24.47% for the target model alone. The paper acknowledges this is preliminary—the ORM used "ranges from $-\infty$ to $\infty$" and the threshold was set to $\delta = 0$ without tuning—but it demonstrates that the RSD framework generalizes beyond PRMs to any reward signal, though more sophisticated reward models and tuning would be needed for strong performance.

The Threshold $\delta$: The Primary Control Knob

The threshold $\delta$ is the single hyperparameter that controls the efficiency-quality tradeoff in RSD when using the binary step function. It is the value that the PRM score must meet or exceed for a draft step to be accepted.

How $\delta$ affects behavior (Figure 5): The paper sweeps $\delta$ from 0.0 to 1.0 on MATH500 using RSD (1.5B draft, 7B target, 7B PRM):

  • $\delta = 0.0$: All draft steps are accepted regardless of reward. This is equivalent to using only the draft model—100% of questions solved by draft alone, but accuracy is only ~74% (the draft model's standalone performance).

  • $\delta = 0.1$ through $\delta = 0.5$: Some low-quality draft steps are rejected and the target model takes over. Accuracy climbs gradually, and the percentage of questions solved purely by the draft model drops—with $\delta = 0.5$, about 65% of questions are still solved by the draft model alone (meaning for those questions, every single step had reward ≥ 0.5).

  • $\delta = 0.6$ through $\delta = 0.8$: Accuracy peaks in this region, with $\delta = 0.7$ achieving the maximum of ~84.6%. At $\delta = 0.7$, approximately 48% of questions are solved entirely by the draft model (the target model is never invoked for those questions), while the remaining 52% require at least one target model intervention. The paper highlights this as a key advantage: "This eliminates the need for the target model to engage with these 48% of questions, distinguishing RSD from SD, which always involves both the draft and target models for every question" (Section 3.4).

  • $\delta = 0.9$: Accuracy declines to ~83.4%, slightly below $\delta = 0.7$. The high threshold means many reasonable draft steps are rejected, forcing unnecessary target model invocations that may not improve (or may slightly degrade) accuracy.

  • $\delta = 1.0$: The threshold is so high that virtually all draft steps are rejected. This is equivalent to using only the target model—accuracy is ~83.2%, matching the single target model baseline.

The peak at $\delta = 0.7$ is significant: It means RSD with a moderate threshold achieves higher accuracy (84.6%) than either the draft model alone (74%, at $\delta = 0$) or the target model alone (83.2%, at $\delta = 1$). This is the "controlled bias as a feature" phenomenon—by selectively mixing draft and target outputs based on reward, RSD produces a final distribution that is better than either individual model. The improvement over the target model alone (+1.4 points on MATH500) demonstrates that the target model is not optimal on every step—sometimes the draft model produces better reasoning, and the PRM can identify those cases.

Why $\delta = 0.7$ is the default in the paper: The authors state it "offers a good trade-off between accuracy and efficiency for all tasks" (Table 2 caption). It is not the absolute optimum for every benchmark—Table B.2 shows that slight adjustments (e.g., $\delta = 0.8$ or $\delta = 0.9$) can yield small further gains on specific tasks, and the paper includes an "optimized threshold" $\delta^*$ column in Table 2 that is tuned per-task. However, $\delta = 0.7$ is robust across all six benchmarks and all model configurations tested, making it a practical default that avoids per-task tuning.

Task-specific optimal thresholds (Table B.2): The paper reports per-task $\delta^*$ values implicitly through the accuracy numbers. For the math model configuration (Qwen2.5-Math, 1.5B draft, 7B target, 7B PRM), the optimal thresholds vary: MATH500 peaks at $\delta = 0.7$ (84.6%); GSM8K peaks at $\delta = 0.8$ or $\delta = 0.9$ (95.8%); GPQA peaks at $\delta = 0.7$ or $\delta = 0.8$ (34.3% or 34.3%). The variation is moderate, confirming that $\delta = 0.7$ is near-optimal across tasks while a per-task tuning provides marginal additional gains. The paper suggests that "different tasks have a different complexity of reasoning" (Table 2 caption), leading to different reward distributions and thus slightly different optimal thresholds.

Automatic compute allocation via $\delta$ (Section 3.4, Appendix B.2): The paper frames the threshold as an automatic difficulty-aware compute allocation mechanism. At $\delta = 0.7$, the draft model alone solves 84% of Level 1 questions (easiest), 67% of Level 2, 58% of Level 3, 44% of Level 4, and only 19% of Level 5 (hardest) on MATH500's five difficulty levels (Figure B.1). This means RSD automatically allocates more compute (target model invocations) to harder problems and less to easier ones, without requiring an explicit difficulty estimator. The mechanism is implicit: harder problems produce lower PRM scores on the draft's steps, triggering more rejections and target model fallbacks. The paper explicitly contrasts this with speculative decoding: "RSD can be considered as a method for automatic compute allocation, less compute for easy questions and more compute for hard questions, which is different from SD that always needs both target and draft models for every question" (Appendix B.2).

The Target Model's Invocation and Its Guaranteed Acceptance

A crucial design choice in RSD is that the target model's output is never evaluated by the PRM and never rejected. When the draft step's reward falls below the threshold, the target model generates a replacement step unconditionally—it is always appended to the sequence.

Why is this safe? The paper's foundational assumption (Equation 1) is that the target model is strictly better than the draft model in expected reward: $\mathbb{E}_{y \sim P_M}[r(y \mid z)] \geq \mathbb{E}_{y \sim P_m}[r(y \mid z)]$. This assumption is empirically validated in Figure 3: the target model's reward scores are consistently higher than the draft model's, especially for correctly answered questions (middle panel), where the target model's reward curve is shifted noticeably upward. The target model wins the reward comparison 62.7% of the time across all questions (right panel) and 56.6% of the time on correctly answered questions.

What if the target model produces a worse step? The assumption can fail on specific instances—the target model wins only 62.7% of reward comparisons, meaning it loses (has lower reward) 37.3% of the time. The paper does not address this failure mode directly in the main text, but the empirical results speak: RSD with $\delta = 0.7$ achieves 84.6% on MATH500 vs. 83.2% for the target model alone, demonstrating that the net effect of target model invocations is positive on average, even if some individual target steps are worse than the draft steps they replaced. The PRM's role is to invoke the target model only when the draft step is probably bad (reward < $\delta$), which means the target model is operating in a regime where it has high expected improvement—the draft step was likely wrong, so even an imperfect target step is likely better.

An alternative would be to also score the target model's output: You could imagine a symmetric approach where the PRM scores the target's step too, and if it's also below threshold, some third mechanism kicks in (majority voting across multiple target generations, for example). The paper does not explore this, likely because it would increase computational cost (each target invocation would require a PRM evaluation) without a clear theoretical benefit—the target model is assumed to be the upper bound of available capability, so rejecting its outputs would be inherently self-defeating without a stronger model to fall back to.

Step Segmentation: How the System Knows Where One Step Ends

RSD operates at the step level, not the token level, which requires a definition of what constitutes a "step." The paper's default setting (Section 3) defines a reasoning step as a generation that ends with the delimiter \n\n (two newline characters). This is a natural segmentation for math reasoning: models trained to produce step-by-step solutions typically separate steps with blank lines.

How this works in practice: The draft model generates autoregressively until it produces \n\n, at which point the generation is treated as a complete step. The entire multi-token segment is then passed to the PRM for scoring. If accepted, the step (including its trailing newlines) is appended to the context. If rejected, the target model generates its own step, also terminated by \n\n.

Why step-level rather than token-level: The paper's central insight is that reward models evaluate reasoning quality, which is a property of complete logical units, not individual tokens. A token-level PRM would need to judge whether "the" or "=" or "3" constitutes good reasoning, which is meaningless. The step-level grain also aligns with how process reward models are trained and used in the literature (Lightman et al., 2023; Chen et al., 2024a)—they score semantically complete reasoning sub-steps.

Implications for the acceptance criterion: Because the acceptance decision happens at step boundaries, the draft model generates multiple tokens "for free" before a decision is made. If the step is rejected, all those draft tokens are discarded, which represents wasted computation—but the alternative (checking at every token) would require the PRM to score partial, incomplete reasoning fragments, which it is not designed to do and which would likely produce noisy, unreliable scores. The step-level grain is thus a pragmatic tradeoff: slightly more wasted computation per rejection, but much more reliable quality signals.

The step delimiter is configurable: The paper uses \n\n as the delimiter for all math reasoning tasks because the models being used (Qwen2.5-Math, etc.) naturally produce step-separated output. For other tasks or model families, a different delimiter might be appropriate—for the general-domain AlpacaEval experiment (Section 4), the paper still uses \n\n, acknowledging that this is a heuristic adaptation.

Computational Cost Model: How Efficiency Is Measured

The paper quantifies efficiency using FLOPs (floating-point operations), following the standard approximation from the transformer scaling literature (Vaswani et al., 2017; Kaplan et al., 2020):

FLOPs=2N\text{FLOPs} = 2N

where $N$ is the number of model parameters, and the factor 2 accounts for one multiplication and one addition per parameter per token (the forward pass cost for a single token in a transformer). This is stated in Section 3.3: "we adopt the standard approximation of FLOPs for transformers with $N$ parameters, i.e. $2N$ per inference token."

Total FLOPs for a generation: For a sequence that generates $T$ tokens using a model with $N$ parameters, the total inference FLOPs is $2N \times T$. In RSD, tokens can be generated by three different models, each with its own parameter count:

  • Draft model tokens: $2N_{\text{draft}} \times T_{\text{draft}}$ FLOPs, where $N_{\text{draft}}$ is the draft model's parameter count and $T_{\text{draft}}$ is the number of tokens generated by the draft model (both accepted steps and the tokens of rejected steps that were discarded).

  • Target model tokens: $2N_{\text{target}} \times T_{\text{target}}$ FLOPs, where $N_{\text{target}}$ is the target model's parameter count and $T_{\text{target}}$ is the number of tokens generated by the target model (only for rejected steps—target tokens are always kept).

  • PRM tokens: $2N_{\text{PRM}} \times T_{\text{PRM}}$ FLOPs, where $N_{\text{PRM}}$ is the PRM's parameter count and $T_{\text{PRM}}$ is the number of tokens processed by the PRM. The paper explicitly notes that "the inference cost for PRMs is also included in the calculations" (Section 3.3). The PRM processes each draft step (equivalent to processing that step's tokens), plus the entire context up to that point—but the paper's FLOPs accounting treats PRM cost as proportional to the number of tokens scored.

How RSD achieves $4.4\times$ fewer FLOPs (Figure 4): The key to RSD's efficiency is that the expensive target model is invoked only for a fraction of steps. In the configuration RSD (7B/72B/7B)—1.5B or 7B draft, 72B target, 7B PRM—the draft model (with far fewer parameters) generates most tokens cheaply, and the 72B target model (with $\sim 10\times$ more parameters than the 7B draft) is only called when the PRM deems it necessary. The paper reports that RSD (7B/72B/7B) achieves 88.0% accuracy on MATH500 compared to 85.6% for the target model (72B) alone, with "nearly $4.4\times$ fewer FLOPs" (Section 3.3). This factor comes from the ratio of FLOPs required by the 72B model generating all tokens versus the weighted sum of draft, target, and PRM FLOPs in RSD.

Comparison with speculative decoding (Figure 4): RSD (1.5B/7B/7B) achieves higher accuracy (84.6%) than SD (1.5B/7B) at 83.4% while using fewer FLOPs. The FLOPs advantage comes from two sources: (1) RSD sometimes accepts draft steps that SD would reject (because they don't match the target model's token distribution), avoiding target model invocations; and (2) RSD never runs the target model for steps where the draft produces high-reward output, whereas SD always runs at least the verification pass.

Comparison with Best-of-N (Figure 4): RSD (7B/72B/7B) achieves 88.0% accuracy vs. BoN (7B/7B, N=64) at 86.2%, with "significantly lower computational cost." Best-of-N with N=64 requires generating 64 complete solutions from the 7B model (64 × solution length in tokens), scoring all of them with the 7B PRM, and selecting the best. RSD generates one solution but selectively invokes the 72B model for hard steps, resulting in a small fraction of the total tokens being produced by the expensive model.

Why FLOPs rather than wall-clock time: FLOPs are a hardware-independent measure of computational work. In practice, RSD's efficiency gains in wall-clock time depend on implementation details—the draft model and PRM are smaller and faster than the target model, but the sequential nature of the algorithm (draft → PRM → decision → potentially target → next draft) may limit parallelism. The paper runs experiments on NVIDIA A100 GPUs using vLLM (Kwon et al., 2023) as the backend, but does not report wall-clock latency numbers.

Experimental Configuration: Hyperparameters and Settings

Models used (Section 3, "Models" paragraph): The paper evaluates three model families:

  • Qwen2.5-Math-Instruct family: Draft models at 1.5B parameters, target models at 7B and 72B parameters. This is the primary configuration; the 7B target represents a moderate-capability reasoning model, and the 72B target represents a high-capability one.

  • Qwen2.5-Instruct family (general-purpose): Draft at 1.5B, target at 7B. Used to test generalization beyond math-specialized models.

  • Llama-3.1-Instruct family (general-purpose): Draft at 1B (Llama-3.2-1B-Instruct), target at 8B. Used to test cross-family generalization.

PRM configurations: The primary PRM is Skywork-o1-Open-PRM, available at 1.5B and 7B. Table 5 additionally tests Qwen2.5-Math-PRM-7B and -72B for robustness.

Decoding parameters (Section 3, "Default Setting"): The paper distinguishes two temperature regimes:

  • For majority voting, (process) Best-of-N, and beam search: temperature = 0.7, top-p = 0.8. These methods benefit from diversity because they sample multiple candidates and need variation to explore the solution space.

  • For all other methods (single model, SD, RSD): temperature = 0, top-p = 1. This is greedy decoding—the model always selects the most likely token. The paper uses greedy decoding for RSD, SD, and single-model baselines to isolate the effect of the decoding strategy from the effect of stochastic sampling. With temperature = 0, the draft model always produces the same step for a given context, making the system deterministic modulo the binary accept/reject decisions.

Step delimiter: "For process Best-of-N, beam search and RSD, we define a generation ended with \n\n as a reasoning step, and then apply a PRM to rate this step" (Section 3). This means the system monitors the generated text for the pattern \n\n (double newline), and when detected, treats the preceding text as a complete step to be scored.

Speculative decoding configuration: For the SD baseline, the paper uses "a number of speculative tokens as 7" (Section 3). This means the draft model generates up to 7 tokens ahead, and the target model verifies them in parallel—standard speculative decoding with a fixed lookahead window.

Hardware and software: All experiments on NVIDIA A100 GPUs, using vLLM (Kwon et al., 2023) as the inference backend. vLLM provides efficient KV-cache management and continuous batching, which is important for serving multiple queries efficiently.

Number of samples for baselines: Majority voting and Best-of-N use $\text{maj}@16$ and $N = 16$ for the 7B target configurations, and $\text{maj}@64$ and $N = 64$ for the 72B target configurations. The larger sample counts for the 72B target baseline are chosen "to show their converged performance" (Section 3), meaning the paper wants to compare RSD against the best possible performance achievable by sampling-based methods, even though those methods use more compute than RSD. This makes the comparison conservative—RSD outperforms baselines that use more total FLOPs.

The Theoretical Guarantee: RSD Cannot Be Worse Than the Draft Model

Proposition 2.2 provides the central theoretical result: under two assumptions, the expected reward of RSD is at least as high as the expected reward of the draft model alone.

Assumption 1: The weighting function $\omega(r)$ is non-decreasing in $r$. This means higher-reward steps are at least as likely to be accepted as lower-reward steps. The binary step function satisfies this trivially (it's a step from 0 to 1 at $\delta$), as do all other variants in Table 1.

Assumption 2: The target model has higher expected reward than the draft model: $\mathbb{E}_{P_M}[r] \geq \mathbb{E}_{P_m}[r]$. This is Equation 1 in the paper and is empirically validated in Figure 3.

The proof structure: The expected reward under $P_{\text{RSD}}$ is decomposed into two terms:

EPRSD[r]=EPm[ω(r)⋅r]+(1−EPm[ω(r)])⋅EPM[r]\mathbb{E}_{P_{\text{RSD}}}[r] = \mathbb{E}_{P_m}[\omega(r) \cdot r] + (1 - \mathbb{E}_{P_m}[\omega(r)]) \cdot \mathbb{E}_{P_M}[r]

The first term is the expected reward from accepted draft steps (weighted by acceptance probability). The second term is the expected reward from target model fallbacks (the rejection probability times the target's expected reward).

The proof then shows that $\mathbb{E}_{P_{\text{RSD}}}[r] - \mathbb{E}_{P_m}[r]$ equals:

CovPm(ω(r),r)+(1−EPm[ω(r)])⋅(EPM[r]−EPm[r])\text{Cov}_{P_m}(\omega(r), r) + (1 - \mathbb{E}_{P_m}[\omega(r)]) \cdot (\mathbb{E}_{P_M}[r] - \mathbb{E}_{P_m}[r])

Both terms are non-negative: the covariance term is non-negative because $\omega(r)$ is non-decreasing in $r$ (higher reward means higher weight), and the second term is non-negative because $\mathbb{E}_{P_m}[\omega(r)] \leq 1$ (it's a probability) and $\mathbb{E}_{P_M}[r] \geq \mathbb{E}_{P_m}[r]$ (Assumption 2).

What this guarantee means operationally: RSD is "safe" in the sense that it can never make the average reward worse than just using the draft model. If the target model is genuinely better than the draft (Assumption 2), and if the weighting function correctly identifies good steps (is non-decreasing in reward), then RSD's mixture is provably better than or equal to the draft model's performance. This guarantee does not say RSD beats the target model—that's an empirical finding, not a theorem—but it says RSD is at least as good as the cheaper model, which is the minimum bar for a method that uses both models.

When can RSD exceed the target model? The paper acknowledges this possibility in Section 2.4: "since $P_{\text{RSD}}$ is a mixture of $\omega_r P_m$ and $P_M$, its reward is a weighted sum of their expected rewards. Thus, $P_{\text{RSD}}$ can exceed $P_M$ in expected reward when we use an aggressive $\omega$ that only assigns 1 to high-reward regions." If the draft model, in the regions where it produces very high-reward outputs (above the threshold), has a higher expected reward than the target model's average performance, the mixture can outperform both. The target model contributes to low-reward regions (replacing bad draft outputs), while the draft model handles high-reward regions where it excels. This is exactly what the empirical results show—RSD at $\delta = 0.7$ outperforms the target model alone because it uses the draft model's best outputs and the target model's general robustness, getting the best of both.

Design Choices and Their Justifications: A Summary

Why a binary step function rather than a smooth weighting? Proposition 2.3 proves optimality under a budget constraint—given that you can only afford to run the target model for a fraction of all steps, you should allocate those invocations to the worst draft outputs. A binary threshold implements this exactly. Smooth weighting functions waste some target model budget on intermediate-quality steps where the marginal improvement over the draft is small. Figure 6 empirically confirms that the binary function achieves the best accuracy at comparable cost.

Why $\delta = 0.7$ as the default? It emerges empirically as a robust near-optimal threshold across tasks. The PRM scores on correctly answered questions tend to be above 0.7 (Figure 3, middle panel, shows target model reward scores predominantly in the 0.4–1.0 range for correct answers, with a concentration above 0.6). Setting the threshold at 0.7 accepts most correct draft steps and rejects most incorrect ones. The threshold is not tuned per-task in the main results—the paper shows that even without tuning, RSD outperforms SD and the target model.

Why step-level rather than token-level? Reward models evaluate reasoning quality, which is a semantic property of complete logical sub-units, not individual tokens. Step-level operation also means the PRM is invoked fewer times (once per step, ~18 times per question on MATH500) rather than once per token (potentially hundreds of times), keeping PRM overhead negligible. The tradeoff is that a rejected step discards more generated tokens, but the paper's efficiency results show this is worthwhile.

Why use an external PRM rather than the target model's own probabilities? This is the fundamental departure from speculative decoding. The target model's token probabilities measure "would I have said this?" not "is this correct?" On reasoning tasks, correctness and model-internal probability are imperfectly correlated—the target model can assign high probability to plausible-but-wrong reasoning. An external PRM trained specifically for correctness evaluation provides a signal that is better aligned with the true objective (producing correct answers). The empirical results in Table 2 validate this: RSD with a PRM consistently outperforms SD (which uses only model probabilities) and even outperforms the target model alone.

Why is the target model always accepted, never rejected? The target model is treated as the upper bound of available capability—if we reject its output, what would we replace it with? There's no stronger model to fall back to. The assumption that the target model dominates the draft model in expected reward (Equation 1, empirically validated in Figure 3) justifies always trusting the target model when it is invoked. The target model is only invoked when the draft step is probably bad (reward < threshold), so even occasional target model errors are likely better than the draft steps they replace.

Why temperature = 0 for RSD? Greedy decoding makes the system deterministic modulo the accept/reject decisions, which simplifies analysis and isolates the effect of the reward-guided mechanism. With temperature > 0, the draft model would produce different steps on different runs, introducing variance that would confound evaluation. The paper notes that temperature > 0 is used for baselines (majority voting, Best-of-N, beam search) that specifically benefit from diversity, but RSD's strength is in selective refinement, not diversity exploitation.

Model Merging: Reducing the Number of Served Models

An important practical consideration for RSD is that it requires serving three separate models (draft, target, PRM), which increases deployment complexity and memory requirements. Section 4 addresses this through model merging experiments.

How merging works (Appendix B.3): The paper uses linear interpolation (weighted averaging of parameters) via MergeKit (Goddard et al., 2024). Since the PRM and the policy models have different architectures—"the PRM includes a projection layer atop the final transformer layer, which projects the hidden dimension to a scalar output, whereas the policy model employs an lm head" (Appendix B.3)—only the shared transformer layers are merged. The PRM's projection layer and the policy model's language model head are kept separate.

Two merging configurations tested: (1) Merge the draft model with the PRM (so the draft model also serves as its own reward model); (2) Merge the target model with the PRM (so the target model also serves as reward model for draft outputs). The interpolation weights tested are [0.6, 0.4] and [0.5, 0.5], with the policy model receiving 0.6 and the PRM 0.4, which performed slightly better.

Results (Table 4, Appendix Table B.3): Merging models "does not necessarily degrade performance and remains superior to SD" (Section 4). Specifically:

  • RSD (7B target, 1.5B draft, 1.5B PRM merged with draft): average accuracy 65.0, compared to 65.9 for the unmerged version. A small drop (-0.9) but still above SD (64.6).
  • RSD (7B target merged with 7B PRM, 1.5B draft): average accuracy 66.7, compared to 66.1 for the unmerged version. Merging larger models actually improves performance (+0.6), consistent with findings by Yadav et al. (2024) that model merging can yield beneficial synergies.

Why this matters: The ability to merge models means RSD can be deployed with 2 models (merged draft+PRM or target+PRM) instead of 3, reducing memory footprint and simplifying serving infrastructure. The merged configurations still outperform standard speculative decoding, preserving the key advantages of RSD while being more practical to deploy.

Relationship to Standard Speculative Decoding: A Mathematical Connection

The paper notes in Section 2.4 that "Reward-based weighting functions $\omega_r$ are not the only option" and explicitly draws a connection to standard speculative decoding:

w(y∣z)=min⁡(1,αPM(y∣z)Pm(y∣z))w(y \mid z) = \min\left(1, \alpha \frac{P_M(y \mid z)}{P_m(y \mid z)}\right)

where $\alpha > 0$ is a hyperparameter. When $\alpha = 1$, this is exactly the acceptance probability used in standard speculative decoding (Leviathan et al., 2023): the draft token is accepted with probability $\min(1, P_M/P_m)$, which guarantees the final distribution matches $P_M$. When $\alpha < 1$, the acceptance criterion is more conservative, rejecting more draft tokens; when $\alpha > 1$, it's more aggressive, accepting more draft tokens but introducing bias.

The key insight connecting the two formulations: The likelihood ratio $P_M / P_m$ can be interpreted as a measure of "how much the target model agrees with the draft." If this ratio is high, the draft token is very likely under the target model, so it's safe to accept. The reward function $r(y \mid z)$ can be interpreted as a more general quality measure—it doesn't just check whether the target model agrees, but whether an independent evaluator thinks the output is good.

Hybrid approaches: The paper suggests that "hybrid approaches can combine both the reward function $r$ and the ratio $P_M / P_m$" (Section 2.4), for example:

w(y∣z)=min⁡(1,β r(y∣z)PM(y∣z)Pm(y∣z))w(y \mid z) = \min\left(1, \beta \, r(y \mid z) \frac{P_M(y \mid z)}{P_m(y \mid z)}\right)

or by normalizing the two quantities. Such hybrids could be useful when the reward model is noisy or the distributional mismatch between draft and target is extreme—the likelihood ratio provides a "sanity check" that the draft output is at least plausible under the target model, while the reward provides a quality assessment.

Why RSD doesn't use the likelihood ratio: The paper's experiments use purely reward-based weighting. The rationale, though not explicitly stated as such, is clear from the results: on reasoning tasks, the reward signal is more reliable than the distributional match for identifying correct reasoning. The target model can assign high probability to wrong answers (if they follow a plausible reasoning pattern), and low probability to correct answers (if they use an unconventional approach). RSD's reward-based acceptance sidesteps this issue by evaluating correctness directly, while SD's likelihood-ratio test is vulnerable to it.

RSD and SD are complementary, not mutually exclusive: Section 4 suggests they "can be seamlessly combined to enhance efficiency. For instance, during a rejected step, SD (draft+target) can be utilized to regenerate the step." In this combined approach, RSD's PRM-based acceptance would be the first-stage filter, and standard SD would be used within the target model invocation—the target model would verify draft tokens using distributional matching rather than generating from scratch. This could further reduce the cost of target model invocations, though it is not explored in the paper.

4. Key Insights and Innovations

Innovation 1: Relabeling Speculative Decoding's Core Guarantee from "Safety Net" to "Straitjacket" on Reasoning Tasks

The paper's most important intellectual move is not algorithmic but conceptual: it identifies that the unbiasedness guarantee in standard speculative decoding—universally treated as a virtue in the literature—is in fact a binding constraint that prevents speculative decoding from working effectively on the very tasks where inference acceleration matters most.

Every prior speculative decoding paper (Leviathan et al., 2023; Chen et al., 2023a; Miao et al., 2024; Sun et al., 2024b) treats distributional fidelity to the target model as the defining property of the method. The guarantee is: whatever the draft model proposes, the final output distribution exactly matches what the large model would have produced alone. This is framed as a safety property—you can deploy speculative decoding without worrying about quality degradation. The dominant assumption in the field has been that this guarantee is both necessary and sufficient: necessary because users won't accept decoding strategies that might degrade output quality, and sufficient because it preserves everything the large model can do.

This paper argues—and demonstrates empirically—that on reasoning tasks, this guarantee is neither necessary nor sufficient. It is not necessary because a controlled bias toward high-reward outputs can produce better quality than the target model alone (RSD at δ = 0.7 achieves 84.6% on MATH500 vs. 83.2% for the target model only, Table 2). It is not sufficient because on tasks where the draft model outperforms the target model—which empirically happens on certain math benchmarks (CN Middle School 24, College Math in Appendix Table B.1)—speculative decoding's unbiasedness forces the system to output the target model's inferior distribution, making it worse than just using the draft model alone (73.3% for SD vs. 75.2% for draft-only on CN Middle School 24).

This reframing has implications that extend beyond this paper. It suggests that the speculative decoding literature has been optimizing for the wrong objective on reasoning workloads. The goal should not be to match the large model's distribution but to produce the highest-quality output at the lowest computational cost—and these two objectives diverge exactly when the large model's distribution is not perfectly aligned with correctness (which, on reasoning tasks, it is not). The paper's diagnosis is that "high-quality tokens (e.g., those favored by a process reward) may still be rejected if their probabilities under the large model are too low" (Section 1), and this is not a rare edge case but the dominant failure mode on multi-step reasoning.

Prior work had noted that speculative decoding's efficiency degrades on certain tasks due to low acceptance rates, but the standard remedy has been to improve the draft model (tree-based speculation, better draft architectures, self-speculation) or improve the acceptance criterion within the unbiasedness framework (optimal transport, improved calibration). The paper's insight is more fundamental: the problem is not that the draft model is insufficiently good at mimicking the target; it is that mimicking the target is the wrong standard altogether.

This is a fundamental conceptual shift, not an incremental refinement. It opens a new category of decoding methods—biased speculative decoding with external quality signals—that the field had not previously explored because the unbiasedness guarantee was treated as definitional to speculative decoding. It also provides a diagnostic lens for understanding where speculative decoding will and won't work: any task where correctness and model-internal probability are poorly correlated is a task where reward-guided methods should outperform distribution-matching methods.

Innovation 2: The Adoption of a Reward Model as a Decoding-Time Arbiter Between Models of Different Scales

The paper's second distinctive contribution is repurposing the process reward model from a search/selection tool into a dynamic routing mechanism for heterogeneous model ensembles. This is a genuinely new role for PRMs that sits outside both of their established use cases in the literature.

Prior work on PRMs has used them in two ways. First, as a search heuristic: Lightman et al. (2023) and subsequent tree-search methods (Chen et al., 2024a; Yao et al., 2024; Qi et al., 2024) use per-step reward scores to guide beam search or MCTS over the generation space of a single model—the PRM tells the search algorithm which partial solutions are promising, and the search explores accordingly. The PRM improves selection from within a fixed proposal distribution. Second, as a selection criterion: Best-of-N (Brown et al., 2024; Cobbe et al., 2021a) uses the PRM (or ORM) to score complete solutions and pick the best one. Again, the PRM operates on outputs from a single model; it selects rather than routes.

RSD uses the PRM in a fundamentally different role: as a model selector. The PRM does not guide search within a model's generation space, nor does it select among multiple complete solutions. Instead, it decides, at every reasoning step, which model should generate the next step—the cheap draft model or the expensive target model. This is routing, not search or selection.

The significance of this shift is that it transforms the PRM from a capability multiplier into an efficiency multiplier. In prior work, PRMs are used to extract more capability from a given model by spending more compute (searching over many candidates). The paper's experimental results in Table 3 dramatize the limitation of this approach: beam search with a 1.5B model and a 1.5B PRM achieves 78.2% on MATH500, while RSD with the same 1.5B draft and PRM but a 7B target model achieves 82.6%—using less total FLOPs. The PRM-as-router achieves better results with less compute than the PRM-as-search-heuristic because routing can draw on the target model's qualitatively different generation capabilities, while search can only rearrange the draft model's limited outputs.

This insight connects to a broader theme that has been emerging in the test-time compute literature but hadn't been realized in the context of speculative decoding: improving the proposal distribution (what can be generated) is often more effective than improving the search over a fixed proposal distribution (Snell et al., 2024). RSD's architecture implements this insight directly—the target model is a capability escalator that kicks in when the draft model's proposal space is insufficient, rather than an optimization tool to be applied uniformly.

The paper's AlpacaEval experiment (Section 4, Table 6) further demonstrates the generality of this concept: using an outcome reward model (ORM) instead of a PRM, on an open-ended generation task with no step-level structure, the routing mechanism still functions (65% of tokens generated by the draft model alone, win rate of 18.85% vs. GPT-4-Turbo compared to 7.09% for draft-only and 24.47% for target-only). This suggests that reward-guided model routing is not tied to reasoning tasks or step-level reward signals—it generalizes to any setting where a quality signal can distinguish when the small model is adequate from when the large model is needed.

Innovation 3: The Empirical Discovery That Selective Draft-Target Mixing Can Exceed the Performance of Either Model Alone

The paper presents a striking empirical finding that is not guaranteed by any of its theoretical results and that challenges a default assumption: RSD achieves higher accuracy than the target model alone on average across benchmarks, despite the target model being larger, more capable, and serving as the fallback when the draft model's output is judged insufficient.

This result appears consistently across configurations. In Table 2, for the Qwen2.5-Math family with a 7B target, RSD at δ = 0.7 achieves 65.9 average accuracy vs. 65.3 for the target alone (with the 7B PRM configuration, it rises to 66.1). For the Qwen2.5 general-purpose family, the gain is more dramatic: 61.8 vs. 59.9. For the 72B target configuration, RSD achieves 72.9 vs. 71.8. On the challenging GPQA benchmark specifically, RSD (1.5B/7B/1.5B) scores 38.4 vs. 32.8 for the 7B target alone—a 5.6 percentage point improvement.

The mechanism behind this counterintuitive result is not explicitly theorized in the paper, but can be reconstructed from the architecture and the evidence. The target model is not omniscient—it makes mistakes, and its mistakes are not randomly distributed. The PRM, trained independently to evaluate step quality, can sometimes recognize when a target model step is likely wrong. By only invoking the target model when the draft step has low reward (below δ), RSD ensures that the target model is operating in a regime where its expected improvement over the draft is high. Meanwhile, when the draft model produces a high-reward step, it is preserved—even if the target model would have produced a different (and potentially worse) step. The system thus selectively amplifies the strengths of each model while suppressing their weaknesses: the draft model's best outputs (which can be very good on certain problem types—recall from Appendix Table B.1 that the 1.5B math model outperforms the 7B on some benchmarks) are kept; the draft model's worst outputs are replaced by the target model; and the target model is never called upon to "fix" steps that were already good.

This is a form of implicit ensembling through rejection sampling—the final output distribution is a mixture that concentrates probability mass on high-reward regions of the draft model's distribution while filling in low-reward regions with the target model. Standard model ensembling (averaging outputs, majority voting) combines models symmetrically. RSD does so asymmetrically and adaptively based on a quality signal, which can extract more value from the combination than either uniform ensembling or using either model alone.

This finding is significant beyond the raw accuracy numbers because it refutes the implicit assumption in the speculative decoding literature that the target model represents a performance ceiling that acceleration techniques must preserve but cannot exceed. The paper shows that a system combining a small model and a large model with a reward signal can outperform the large model alone—not in spite of the small model's involvement, but because of it. This opens the door to architectures where models of different scales are not just alternatives to be chosen between, but complementary components to be dynamically composed.

Innovation 4: A Theoretical Framework That Characterizes the Optimal Form of Quality-Based Routing Under a Compute Budget

The paper provides a theoretical result (Proposition 2.3) that, while simple, has important conceptual implications: under a constraint on how often the expensive model can be invoked, the optimal strategy is a hard threshold on reward—use the expensive model exactly on the worst draft outputs, and never on the good ones.

This result is not mathematically deep (it follows from a straightforward Lagrangian optimization), but its conceptual contribution is to formalize an intuition that the field had not previously articulated: compute allocation between models of different capabilities should be reward-monotonic and threshold-based, not probability-matching or confidence-weighted. Standard speculative decoding's acceptance criterion—accept with probability min(1, P_M/P_m)—is a continuous function of the likelihood ratio; it sometimes accepts low-probability tokens and sometimes rejects high-probability ones, proportional to the target model's relative confidence. Proposition 2.3 says this is suboptimal when you have a budget constraint on target model usage. You should draw a hard line: everything above a quality threshold gets the cheap model; everything below gets the expensive one. No probabilistic mixing, no soft interpolation.

The paper tests this prediction empirically in Figure 6: among five weighting functions (constant, binary, clipping, sigmoidal, logistic), the binary step function achieves the highest accuracy at comparable inference cost (84.6% on MATH500 vs. 81.8% for clipping, 81.4% for sigmoidal, and 79.2% for logistic). This validates the theory and provides practical guidance: when building reward-guided routing systems, use a hard threshold, not a soft weighting.

This result also connects RSD to a broader set of optimal stopping and resource allocation problems. The "use the expensive resource on the hardest cases" principle appears in many domains (triage systems, multi-tier customer support, compute-optimal test-time scaling as in Snell et al., 2024), but it had not been formalized in the context of model routing during autoregressive generation. The paper's formulation—maximize expected reward subject to a constraint on the expected fraction of target model invocations—provides a clean theoretical foundation for this class of problems.

One limitation of the theory is that it assumes the constraint on target model usage is known and fixed (captured by γ). In practice, RSD controls target model usage indirectly through the threshold δ, and the relationship between δ and the expected invocation rate depends on the reward distribution of the draft model's outputs on the specific task distribution. The paper does not provide a method for setting δ to achieve a target invocation rate; it treats δ as a hyperparameter to be tuned empirically. Bridging this gap—providing a principled way to set δ based on a desired compute budget—would strengthen the connection between theory and practice.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on seven reasoning benchmarks: GSM8K (Cobbe et al., 2021b), MATH500 (Hendrycks et al., 2021), MMLU STEM (Hendrycks et al., 2020), OlympiadBench (He et al., 2024), GaoKao-2023-En (Liao et al., 2024), GPQA Diamond (Rein et al., 2023), and Minerva Math (Lewkowycz et al., 2022). All are standard benchmarks; MATH500 uses 500 test questions from the MATH dataset, GSM8K uses its standard test split, and the others are used with their public evaluation sets. Exact split sizes are not specified for all benchmarks. The paper also includes two additional math benchmarks in Appendix B (CN Middle School 24 and College Math) and a general-domain evaluation on AlpacaEval (Dubois et al., 2023) with 805 prompts in Section 4.

  • Base model(s). The paper uses three model families to establish generality: Qwen2.5-Math-Instruct (Yang et al., 2024b) at scales 1.5B (draft), 7B (target), and 72B (target); Qwen2.5-Instruct (Yang et al., 2024a) at 1.5B (draft) and 7B (target); and Llama-3.1-Instruct (Dubey et al., 2024; Grattafiori et al., 2024) at 1B (draft, specifically Llama-3.2-1B-Instruct) and 8B (target). The Qwen2.5-Math family is the primary configuration because it provides math-specialized models with non-trivial but far-from-saturated performance, leaving room for test-time strategies to show improvement. The general-purpose families (Qwen2.5-Instruct, Llama-3.1) test whether RSD's benefits transfer to models not specialized for math. For the PRM, the paper uses Skywork-o1-Open-PRM (o1 Team, 2024) at 1.5B and 7B scales as the primary reward model, with additional robustness tests using Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B (Zhang et al., 2025). The choice of Qwen2.5-Math models is well-motivated: the 1.5B draft achieves 73.8% on MATH500 (Table 3), the 7B target achieves 83.2%, and the 72B reaches 85.6%, providing clear capability gaps across which RSD can route.

  • Metrics. The primary metric is accuracy (%) — the fraction of test questions for which the model's final answer exactly matches the ground truth, following the standard grading protocols of each benchmark. For AlpacaEval, the metric is win rate (%) against GPT-4-Turbo as judged by the AlpacaEval 2.0 evaluator (Li et al., 2023). For computational efficiency, the paper uses FLOPs per question, computed using the standard transformer approximation of 2N FLOPs per token per forward pass for a model with N parameters (Vaswani et al., 2017; Kaplan et al., 2020), with the total FLOPs summed across all models involved (draft, target, and PRM) for all tokens generated or processed. The paper emphasizes this metric over wall-clock time because it is hardware-independent and captures total computational work, though it does report that all experiments run on NVIDIA A100 GPUs using vLLM (Kwon et al., 2023).

  • Baselines. The paper evaluates against four categories of baselines:

    1. Single target model only: The large model decoding independently, serving as the quality upper bound that RSD aims to match or exceed while using less compute. This is listed as "Single Model" in Table 2 with the target model size indicated.
    2. Draft model with test-time scaling: Majority voting at N samples (maj@N) and Best-of-N (BoN) with a PRM scoring N complete solutions and selecting the highest-scoring final answer (Brown et al., 2024; Cobbe et al., 2021a). These baselines test whether simply scaling inference compute with the small model can match RSD's efficiency-quality tradeoff. The paper uses maj@16 and N = 16 for the 7B target configurations, and maj@64 and N = 64 for the 72B target configurations, "to show their converged performance" (Section 3).
    3. Standard speculative decoding (SD): The method of Leviathan et al. (2023) with a speculative token count of 7, using temperature = 0 and top-p = 1. This is the most direct point of comparison, as SD uses the same draft and target models as RSD but enforces unbiasedness through token-level probability matching rather than reward-guided acceptance.
    4. Search-based methods: Beam search (Chen et al., 2024a) and process Best-of-N, both using a PRM to select optimal steps or complete solutions. These baselines appear in Table 3 and test whether search over a small model's generation space can match RSD's approach of falling back to a larger model for difficult steps. Beam search sweeps beam sizes of 4 and 8; process Best-of-N sweeps N = 8 and N = 16. These methods use the draft model (1.5B) as the base generator and a 1.5B PRM for scoring, making the comparison to RSD (which uses the same draft and PRM but adds a 7B target model) a test of whether search or model routing is more effective.

  • Generation budget / compute accounting. Compute is measured in FLOPs, not generations or wall-clock time, because different methods invoke models of vastly different sizes. The FLOPs accounting is: 2N_draft × T_draft (all tokens generated by the draft model, including those from rejected steps) + 2N_target × T_target (all tokens generated by the target model for replacement steps) + 2N_PRM × T_PRM (all tokens processed by the PRM, which scores each draft step). The paper explicitly states that "the inference cost for PRMs is also included in the calculations" (Section 3.3). This comprehensive accounting prevents RSD from hiding PRM overhead. The 4.4× FLOPs reduction (Figure 4) compares RSD's total FLOPs against the target model generating all tokens independently.

  • Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, standard deviations, or statistical significance tests. All results in the main tables are single-point estimates from one evaluation run on each benchmark's standard test set. For the threshold tuning experiments (Table B.2), the paper sweeps δ values from 0.6 to 0.9 in increments of 0.1 for each configuration and reports the accuracy at each setting. The "optimized threshold" δ* in Table 2 is per-task tuning—the threshold that achieves the best accuracy on that specific benchmark—which means it is a form of test-set optimization and may not represent out-of-distribution performance. The paper acknowledges this implicitly by reporting both δ = 0.7 (fixed) and δ* (tuned) in the main table, with the fixed threshold serving as the fair, non-overfit baseline. The paper does not discuss or control for the multiple comparisons problem inherent in testing across six benchmarks, multiple model families, multiple PRM configurations, and multiple δ values, making it difficult to assess whether individual benchmark improvements are statistically reliable or could arise from variance in a 500-question test set.

Main Quantitative Results

Reasoning Benchmark Performance (Table 2)

The central experimental result is that RSD consistently outperforms both the single target model and standard speculative decoding on average accuracy across benchmarks, while using less compute. The headline numbers span three model families and multiple PRM configurations:

Math model family (Qwen2.5-Math, 7B target). With δ = 0.7 and a 7B PRM, RSD achieves 66.1 average accuracy across six benchmarks, compared to 65.3 for the 7B single target model (+0.8) and 64.6 for SD (−0.7 below target). With per-task optimized δ*, RSD reaches 66.5 (+1.2 over target, +1.9 over SD). The gains are not uniform: RSD shows the largest improvement on GPQA Diamond (33.8 vs. 32.8 with δ = 0.7, 34.3 with δ*) and Olympiad Bench (42.1 vs. 41.2 with δ = 0.7, 43.6 with δ*), but slightly degrades on GSM8K (94.5 vs. 95.7 with δ = 0.7) and MMLU STEM (72.3 vs. 71.8 with δ = 0.7). This heterogeneity matters: the target model is already near-ceiling on GSM8K (95.7%), leaving little room for improvement, while the harder benchmarks show larger gains from reward-guided routing.

Math model family (Qwen2.5-Math, 72B target). With a 7B draft model and δ = 0.7 with a 7B PRM, RSD achieves 72.7 average accuracy vs. 71.8 for the 72B target alone (+0.9) and 71.1 for SD (−0.7 below target). With δ*, RSD reaches 72.9 (+1.1 over target). The MATH500 result is particularly strong: 88.0 for RSD vs. 85.6 for the target alone (+2.4). The 72B configuration provides evidence that RSD's benefits scale to larger target models—the gains are comparable in absolute terms to the 7B configuration, though the percentage improvement is smaller since the 72B baseline is higher (71.8 vs. 65.3).

General-purpose model family (Qwen2.5-Instruct, 7B target). With a 1.5B draft, δ = 0.7, and a 1.5B PRM, RSD achieves 61.8 average accuracy vs. 59.9 for the target alone (+1.9) and 59.1 for SD (−0.8 below target). With δ*, RSD reaches 62.6 (+2.7 over target). The GPQA result is notable: 31.3 for RSD at δ = 0.7 vs. 28.8 for the target alone (+2.5), and the GaoKao result improves from 64.9 to 64.2 (+0.7). However, for the 7B PRM configuration at δ = 0.7, RSD drops to 59.4 (−0.5 below target), suggesting that the larger PRM may not always help with general-purpose models—a non-obvious interaction. The δ* configuration recovers to 62.1.

General-purpose model family (Llama-3.1-Instruct, 8B target). With a 1B draft, δ = 0.7, and a 7B PRM, RSD achieves 41.9 average accuracy vs. 41.4 for the target alone (+0.5) and 41.1 for SD (−0.3 below target). The gains are more modest than for Qwen models, but the pattern holds: RSD outperforms both SD and the target model. With δ*, RSD reaches 42.5 (+1.1 over target), with the largest per-benchmark improvement on Olympiad Bench (18.1 vs. 14.5) and College Math (from Table B.1, though not in the average).

The SD underperformance is systematic. Across all eight configurations in Table 2 where SD is compared to the single target model, SD achieves lower accuracy than the target alone in every case except the math model 1.5B draft configuration on MATH500 (83.4 vs. 83.2, a negligible +0.2). The average SD-to-target gap ranges from −0.3 (Llama-3.1, 8B target) to −0.8 (Qwen2.5-Instruct, 7B target). The paper attributes this to floating-point errors (citing Chen et al., 2023a) and to cases where "a draft model outperforms the target model... SD's strict unbiasedness leads to worse performance compared to the draft model" (Section 3.1). The latter point is empirically demonstrated in Appendix B (Table B.1): on CN Middle School 24, the draft Qwen2.5-Math-1.5B achieves 75.2% alone, the target 7B achieves 72.3%, and SD (7B/1.5B) achieves 73.3%—better than the target but worse than the draft. RSD with δ = 0.7 reaches 78.2%, exceeding both.

The PRM size tradeoff is task-dependent. Comparing the 1.5B PRM and 7B PRM columns in Table 2 (math model, 7B target, δ = 0.7): the 7B PRM yields higher accuracy on MATH500 (84.6 vs. 82.6), GSM8K (95.5 vs. 94.5), Olympiad Bench (42.1 vs. 39.6), and MMLU STEM (72.3 vs. 71.4), but lower on GPQA (33.8 vs. 38.4) and GaoKao (68.3 vs. 68.8). The GPQA result is the most dramatic—the 1.5B PRM actually outperforms the 7B PRM by 4.6 points on this benchmark. The paper does not explain this anomaly, suggesting that the larger PRM may be overfitting to math-specific reasoning patterns at the expense of the science reasoning tested by GPQA, or that the 1.5B PRM's reward distribution better matches the draft model's output distribution for these types of questions.

The best-of-N and majority voting baselines strongly underperform. Across all configurations, majority voting and Best-of-N with the draft model consistently fall below the target model alone by substantial margins: in the math model family with a 72B target, majority voting at maj@64 achieves 69.8 average vs. 71.8 for the target (−2.0), and Best-of-N at N = 64 with a 7B PRM achieves 68.5 (−3.3). Even with 64× the sampling budget, these methods cannot close the capability gap. The paper interprets this as evidence that "a larger model for reasoning tasks, as its performance cannot be easily matched by a smaller model with increased computation" (Section 3.1). This reinforces the paper's central thesis: scaling test-time compute with a small model hits fundamental capability ceilings, making efficient routing to a larger model more effective than exhaustive exploitation of the small model.

Comparison with Search-Based Methods (Table 3)

The paper compares RSD against beam search and process Best-of-N, both using the same draft model (1.5B Qwen2.5-Math) and PRM (1.5B) as RSD, but without access to a target model. The results are decisive across all three benchmarks evaluated:

  • MATH500: RSD (1.5B/7B/1.5B, δ = 0.7) achieves 82.6%, compared to 78.2% for beam search (beam size 4), 78.2% for beam search (beam size 8), 76.0% for process Best-of-N (N = 16), and 73.8% for the draft model alone. RSD outperforms the best search-based method by 4.4 points.

  • GSM8K: RSD achieves 94.5%, compared to 88.9% for beam search (beam size 4), 88.4% for beam search (beam size 8), and 85.0% for the draft model alone. The gap is 5.6 points over the best search method.

  • Minerva Math: RSD achieves 34.6%, compared to 33.5% for beam search (beam size 4), 32.4% for beam search (beam size 8), and 29.0% for the draft model alone. The gap is 1.1 points—smaller than on MATH500 and GSM8K but still favorable.

The paper's diagnosis is that "for certain complex or 'hard' reasoning steps, search-based methods struggle to find optimal solutions due to the combinatorial explosion of potential candidates" (Section 3.2). The beam search results exhibit diminishing returns with increased beam size: beam size 8 does not improve over beam size 4 on MATH500 (both 78.2%) and actually degrades on GSM8K (88.4% vs. 88.9%), suggesting that the search space quality—rather than search depth—is the bottleneck. RSD addresses this by falling back to a larger model when the search space (the draft model's output distribution) is insufficient, rather than searching more aggressively within it.

Computational Efficiency Analysis (Figure 4)

Figure 4 plots FLOPs per question (log scale, x-axis) against MATH500 accuracy (y-axis) for eight configurations: SD (1.5B/7B), RSD (1.5B/7B/1.5B), RSD (1.5B/7B/7B), SD (7B/72B), RSD (7B/72B/1.5B), RSD (7B/72B/7B), BoN (1.5B/7B, N = 16), BoN (7B/7B, N = 64), and the single target models at 7B and 72B.

RSD achieves the Pareto frontier. The four RSD configurations occupy the upper-left region of the plot—higher accuracy at lower FLOPs than the alternatives. Specifically:

  • RSD (1.5B/7B/7B) achieves ~84.6% accuracy with approximately 3 × 10^4 FLOPs per question, compared to SD (1.5B/7B) at ~83.4% with roughly the same or slightly higher FLOPs, and the single 7B target model at ~83.2% with substantially higher FLOPs (~1.2 × 10^5). RSD thus matches or exceeds the target model's accuracy at a fraction of the FLOPs.

  • RSD (7B/72B/7B) achieves 88.0% accuracy with approximately 8 × 10^4 FLOPs per question, compared to the single 72B target model at 85.6% with approximately 3.5 × 10^5 FLOPs. The paper reports this as "nearly 4.4× fewer FLOPs" (Section 3.3). This is the most dramatic efficiency gain—RSD with a 7B draft achieves higher accuracy than the 72B model alone while using less than a quarter of the computation.

  • BoN (7B/7B, N = 64) achieves 86.2% accuracy—higher than the 72B target alone—but at enormous FLOPs (~1.5 × 10^6, or roughly 4.3× more than RSD (7B/72B/7B) and ~19× more than the 7B target baseline). The paper notes that RSD "delivers 1.8 points higher accuracy at a significantly lower computational cost" compared to BoN (7B/7B, N = 64).

  • BoN (1.5B/7B, N = 16) achieves 82.2% accuracy at moderate FLOPs (~3 × 10^5), which is slightly below the 7B target alone (83.2%) and substantially below RSD (1.5B/7B/7B) at 84.6% with ~10× fewer FLOPs.

The efficiency gains are not solely from using a smaller model more often. The FLOPs advantage comes from two sources: (1) the draft model generates most tokens (cheap), and (2) the target model is only invoked for a fraction of steps (avoiding expensive computation). The PRM adds some overhead—invoked ~18 times per MATH500 question (Section 4, "PRM Overheads")—but since the PRM only processes one step's worth of tokens per invocation (no generation), this cost is negligible compared to the target model generating full reasoning steps. The paper's explicit inclusion of PRM FLOPs in the calculation makes the efficiency gains credible rather than an artifact of ignoring the reward model's cost.

The qualitative difference between RSD and SD in Figure 4 is revealing. SD (1.5B/7B) and SD (7B/72B) are both close to their respective target model accuracies (as expected from unbiasedness) but at FLOPs values that are only modestly lower than the target model alone. This visualizes the paper's central critique: SD's efficiency gains are limited because it must run the target model for every question and often rejects draft tokens, leading to substantial target model computation. RSD's FLOPs advantage comes from entirely skipping the target model for high-reward steps—a capability that SD's unbiasedness guarantee explicitly prevents.

Ablation Studies and Robustness Checks

Threshold δ (Figure 5, Table B.2): Sweeping δ from 0.0 to 1.0 on MATH500 with RSD (1.5B/7B/7B), accuracy increases from ~74% at δ = 0 (draft only) to a peak of ~84.6% at δ = 0.7, then declines to ~83.2% at δ = 1.0 (target only). The percentage of questions solved by the draft model alone (no target model invocation) decreases monotonically from 100% at δ = 0 to ~0% at δ = 1.0. At δ = 0.7, the draft model alone solves approximately 48% of questions—meaning RSD achieves its accuracy improvement over the target model while invoking the expensive target model for only about half of all questions. The paper emphasizes this as "automatic compute allocation": harder questions, where the draft model's steps receive lower rewards, naturally trigger more target model invocations. Table B.2 provides per-task δ sweeps: δ = 0.7 is near-optimal across benchmarks, with some tasks benefiting from δ = 0.8 (GPQA) or δ = 0.9 (GSM8K). The optimal δ varies slightly by task because "different tasks have a different complexity of reasoning" (Table 2 caption), leading to different reward score distributions from the PRM.

Weighting function (Figure 6, Table 1): Five weighting function variants are compared at similar inference cost using RSD (1.5B/7B/7B). The binary step function (δ = 0.7) achieves the highest accuracy at 84.6%. The constant function (p = 0.6, ignoring reward) achieves 77.6%—better than the draft model alone (73.8%) but worse than all reward-aware variants, confirming that PRM feedback is essential. The clipping function achieves 81.8%, the sigmoidal achieves 81.4%, and the logistic (α = 1, δ = 0.6) achieves 79.2%. All are below the binary function, validating Proposition 2.3's prediction that a hard threshold is optimal under a budget constraint. However, the gaps between the binary function and the smooth variants (2.8–5.4 points) are larger than one might expect from the theoretical optimality result alone, suggesting that smooth weighting functions may suffer from noise amplification—intermediate weights partially accept intermediate-quality steps that the binary function would correctly reject, accumulating errors over long reasoning chains.

PRM choice robustness (Table 5): RSD is tested with three PRM architectures (Skywork-o1-Open-PRM-7B, Qwen2.5-Math-PRM-7B, Qwen2.5-Math-PRM-72B) using the math model configuration (1.5B draft, 7B target, δ = 0.7). All PRM variants outperform SD (64.6 average): Skywork-7B achieves 66.1, Qwen2.5-Math-PRM-7B achieves 65.9, and Qwen2.5-Math-PRM-72B achieves 66.5. The improvement from the larger PRM (72B) is modest (+0.4 over Skywork-7B), but the 72B PRM shows particular strength on MMLU STEM (75.6 vs. 72.3 for Skywork-7B). This robustness check is important because it rules out the possibility that RSD's gains are specific to the Skywork PRM. The consistent improvement over SD across PRMs with different training procedures and architectures suggests that any reasonable quality signal enables RSD's routing mechanism to outperform unbiased decoding.

Model merging (Table 4, Table B.3): To test deployment feasibility, the paper linear-merges the PRM with either the draft model or the target model using MergeKit (Goddard et al., 2024), interpolating shared transformer layers with weights [0.6, 0.4] (policy model receiving 0.6). Merging the 1.5B draft model with the 1.5B PRM yields RSD (7B/1.5B*/1.5B*) with average accuracy 65.0—slightly below the unmerged RSD (65.9) but still above SD (64.6). Merging the 7B target model with the 7B PRM yields RSD (7B*/1.5B/7B*) with average accuracy 66.7—actually exceeding the unmerged RSD (66.1). The paper notes this is "consistent with observations reported by Yadav et al. (2024)" that model merging can produce beneficial synergies. The practical implication is that RSD can be deployed with 2 models (merged target+PRM or merged draft+PRM) rather than 3, reducing memory and serving complexity. The merged configurations all outperform SD, preserving RSD's advantages in a more deployment-friendly form.

Difficulty-level behavior (Figure B.1): Using MATH500's human-annotated difficulty levels (1 = easiest, 5 = hardest), the paper examines how RSD (1.5B/7B/7B) behaves across difficulty tiers. The key finding: for the same δ, the proportion of questions solved by the draft model alone decreases with difficulty—at δ = 0.7, 84% of Level 1 questions are solved by draft alone, vs. 67% for Level 2, 58% for Level 3, 44% for Level 4, and only 19% for Level 5. This demonstrates implicit, automatic compute allocation: harder questions naturally trigger more target model invocations because the draft model's steps receive lower PRM scores on difficult problems. The accuracy improvement from involving the target model (δ > 0 vs. δ = 0) also increases with difficulty: +4.7 points for Level 1, +5.6 for Level 2, +6.7 for Level 3, +16.4 for Level 4, +15.7 for Level 5. This validates the paper's claim that RSD automatically allocates computes "less compute for easy questions and more compute for hard questions" (Appendix B.2).

Combining RSD with SD (Section 4, discussion only—not experimentally evaluated): The paper notes that RSD and SD are complementary: during a rejected step where the target model is invoked, standard SD could be used to generate the replacement step more efficiently (draft proposes, target verifies at token level) rather than having the target model generate from scratch. This combination is not evaluated in the paper, but the observation suggests a path to further efficiency improvements.

General-domain task with ORM (Section 4, Table 6): As a preliminary exploration of general-domain applicability, the paper tests RSD with an outcome reward model (Skywork-Reward-Llama-3.1-8B-v0.2) on AlpacaEval using Llama-3.2-1B-Instruct as draft and Llama-3.1-8B-Instruct as target. With δ = 0 (set empirically without tuning), RSD achieves 18.85% win rate vs. GPT-4-Turbo, compared to 7.09% for draft-only and 24.47% for target-only. The paper reports that "65% tokens are generated by the draft model only without any intervention of the target model." This is a limited experiment (δ is not tuned, the ORM output range is −∞ to ∞ making thresholding heuristic, and the "step" delimiter \n\n is unnatural for open-ended generation), but it demonstrates that the RSD framework can function with outcome rewards rather than process rewards, and on tasks without natural step boundaries.

Computational cost of draft model rejections: A notable negative result that the paper does not explicitly quantify is the wasted computation from rejected draft steps. When the PRM rejects a draft step, all tokens generated by the draft model for that step are discarded, and the target model generates a replacement from scratch. For a configuration with δ = 0.7 where roughly 52% of questions require at least one target model intervention, the draft model generates tokens that are thrown away on rejected steps. The paper accounts for these in the FLOPs calculation (all draft model tokens count toward total FLOPs, regardless of acceptance), but does not report the rejection rate or the fraction of total draft tokens that are wasted. This is a genuine efficiency cost that partially offsets the savings from not invoking the target model on accepted steps, and its omission from detailed analysis is a gap.

Critical Assessment

Does RSD genuinely improve reasoning accuracy over the target model alone, or are the gains attributable to benchmark noise and per-task threshold tuning?

The paper's central claim is that RSD simultaneously improves both efficiency and accuracy over both standard speculative decoding and the single target model. The accuracy claim requires careful scrutiny because the absolute gains over the target model are modest in many configurations: +0.8 average for Qwen2.5-Math 7B with δ = 0.7 and 7B PRM, +0.9 for the 72B configuration, +0.5 for Llama-3.1, and +0.4 for Qwen2.5-Math 72B with 7B draft. These are small margins on test sets of unspecified size for most benchmarks (MATH500 has 500 questions; GSM8K's test set has 1,319 questions; sizes for Olympiad Bench, GaoKao, GPQA, and MMLU STEM are not reported in the paper). Without confidence intervals, standard deviations, or statistical tests, it is impossible to determine whether a +0.8 average improvement across six benchmarks is reliably above zero or could reflect variance from a single evaluation run. The paper reports only point estimates—no error bars appear in any figure or table.

This concern is partially mitigated by the consistency of the direction of improvement: RSD outperforms the target model in 7 of 8 (δ = 0.7) and 8 of 8 (δ*) configurations in Table 2. If the null hypothesis were that RSD equals the target model, seeing RSD win in 15 of 16 comparisons would be unlikely by chance. However, the lack of formal statistical reporting remains a weakness—particularly given that the per-task δ* values in Table 2 involve optimizing the threshold on the test set, which introduces an overfitting risk that inflates the reported numbers relative to what would be achieved on truly unseen data.

The GPQA Diamond result is the most striking: RSD (1.5B/7B/1.5B) at δ = 0.7 achieves 38.4% vs. 32.8% for the target alone (+5.6). If this is reliable, it is a substantial improvement on a challenging benchmark. But GPQA Diamond is a small dataset—the paper does not report its exact size, but the original GPQA paper describes the Diamond subset as 198 questions. A 5.6-point swing on ~200 questions could arise from RSD correcting the target model on ~11 questions, which is plausible but also within the range of sampling variability. Without replication or confidence intervals, the magnitude of this gain should be treated as suggestive rather than definitive.

Does the 4.4× FLOPs reduction claim hold up under realistic deployment assumptions?

The 4.4× FLOPs reduction for RSD (7B/72B/7B) compared to the 72B target model alone (Figure 4) is the paper's headline efficiency number. This comparison is valid on its own terms: the total FLOPs consumed by RSD (draft + target + PRM) is approximately 8 × 10^4 per question vs. approximately 3.5 × 10^5 for the 72B model alone, while achieving 88.0% accuracy vs. 85.6%. The calculation includes all models, accounts for rejected draft tokens, and uses the standard transformer FLOPs approximation.

However, several deployment-relevant costs are not captured by this FLOPs accounting:

  1. Memory footprint: RSD requires loading three models (draft, target, PRM) into GPU memory simultaneously, or swapping them in and out. The 7B + 72B + 7B configuration requires approximately 88B parameters' worth of model weights to be resident (assuming 16-bit precision, this is ~176 GB, plus KV-cache memory). A single 72B model requires ~144 GB. If GPU memory is the bottleneck rather than FLOPs, RSD's memory overhead could limit batch sizes or require more GPUs, partially offsetting the FLOPs savings. The model merging experiments (Section 4) partially address this by reducing the count to 2 models, but even 7B + 72B (merged with PRM) is ~158 GB vs. ~144 GB for the 72B alone.

  2. Latency and sequential dependencies: RSD's per-step loop (draft → PRM → decision → potentially target → next step) is inherently sequential. The draft model must finish generating a complete step before the PRM can score it; the PRM must finish before the accept/reject decision; if rejected, the target model must generate a full step before the draft model can propose the next step. This serial dependency means RSD's wall-clock latency may be higher than the FLOPs reduction suggests, especially if the target model is invoked for multiple steps. The paper does not report any latency measurements, which is a significant omission for a method whose primary claimed benefit is efficiency. A 4.4× FLOPs reduction will not translate to a 4.4× latency reduction if the draft model and PRM steps cannot be parallelized and if target model invocations create sequential bottlenecks.

  3. Batch processing: The FLOPs comparison in Figure 4 appears to be per-question, not batched. In high-throughput serving scenarios, the target model can process multiple queries simultaneously via continuous batching, potentially amortizing its cost. RSD's per-question routing decisions create divergent execution paths (some queries need target model, some don't), which could reduce batching efficiency. The paper uses vLLM (Kwon et al., 2023) as the backend, which supports continuous batching, but does not analyze batching efficiency or throughput in queries-per-second.

These caveats don't invalidate the 4.4× FLOPs claim—it's a valid measure of total computational work—but they suggest the claim should be understood as an upper bound on achievable speedup in practice, with actual wall-clock improvements likely smaller and dependent on serving infrastructure and workload characteristics.

Are the baselines fairly configured, or does RSD benefit from weak baselining?

Several baseline configurations warrant scrutiny:

  • Standard speculative decoding uses 7 speculative tokens. The paper does not sweep this parameter; 7 tokens may not be optimal for the step-based reasoning tasks where RSD operates. If the draft model generates reasoning steps of 20–50 tokens before the \n\n delimiter, a 7-token speculative window means many draft tokens are generated before verification even begins, potentially wasting computation on long incorrect steps. This is actually a weakness of standard SD on reasoning tasks (which the paper's motivation correctly identifies), but it also means the SD baseline might be stronger with a different speculative token count. The paper does not ablate this parameter to show that RSD outperforms SD across a range of SD configurations, not just at the default setting.

  • Beam search and process Best-of-N use the 1.5B draft model. In Table 3, these search-based baselines operate entirely within the draft model's generation space. RSD, by contrast, has access to the 7B target model. The comparison demonstrates that routing to a larger model is more effective than search within a small model, but it is not a fair FLOPs-matched comparison—RSD uses a 7B model that the search baselines don't have access to. A more revealing baseline would be beam search or process Best-of-N using the 7B model as the generator and the 1.5B PRM as the scorer, to test whether RSD's routing mechanism still outperforms search when both have access to the same model scales. This experiment is not in the paper.

  • The target model baseline uses greedy decoding (temperature = 0). This is the standard baseline for quality evaluation, but it means RSD's comparison point is the minimum rather than the maximum achievable target model performance. The paper uses temperature = 0.7 and top-p = 0.8 for majority voting and Best-of-N, which benefit from diversity. If the target model were also sampled with non-zero temperature and combined with majority voting or a PRM-verifier (as is common practice in reasoning benchmarks), its accuracy would likely be higher, raising the bar that RSD must clear. The paper's FLOPs comparison in Figure 4 does include BoN (7B/7B, N = 64) as a strong target-model-based baseline (achieving 86.2% at high FLOPs), and RSD still outperforms it, but this is at the cost of 64× sampling—a more moderate N (e.g., N = 4 or N = 8) with the target model could provide a tighter efficiency-quality comparison.

  • No comparison to target model + speculative decoding with reward-guided acceptance. The paper discusses (Section 2.4) but does not evaluate a hybrid weighting function that combines the likelihood ratio P_M/P_m with the reward signal r. Such a hybrid could potentially outperform pure reward-guided RSD by using distributional matching as a consistency check on the reward model's judgments. Without this ablation, it's unclear whether the reward signal alone is sufficient or whether combining it with the target model's own probability assessments would be better.

Do the experiments demonstrate generalization beyond math reasoning?

The paper includes general-purpose model families (Qwen2.5-Instruct, Llama-3.1-Instruct) and a preliminary AlpacaEval experiment, but the bulk of the evidence is on math reasoning benchmarks. Four of the six main benchmarks (MATH500, GSM8K, GaoKao-2023-En, Olympiad Bench) are math-focused; MMLU STEM includes science problems but still has a substantial math component; GPQA tests graduate-level science reasoning. The evaluation suite covers reasoning tasks well, but does not cover non-reasoning generation tasks (summarization, translation, dialogue, creative writing) where the step-level PRM paradigm is least applicable.

The AlpacaEval experiment (Table 6) is a first step toward broader evaluation, but it is preliminary: the threshold was set to δ = 0 without tuning, the outcome reward model's score range is not calibrated for thresholding, and the win rate (18.85%) is far below the target model alone (24.47%). This suggests that RSD in its current form does not transfer straightforwardly to open-ended generation without further development—specifically, better reward models (PRMs for general generation do not yet exist), better step segmentation, and task-appropriate threshold tuning. The paper is upfront about this, noting that "a general-domain PRM and dedicated tuning of δ could further boost the performance" (Section 4), but the current empirical evidence for RSD's generality is thin.

Are the experiments on a single evaluation run, or do they account for variance?

All results appear to be from single evaluation runs—the paper does not mention multiple random seeds, cross-validation, or standard deviations. This is standard practice for LLM benchmarking papers but reduces the reliability of small-margin claims. For MATH500 (500 questions), a 1.4-point accuracy difference (RSD vs. target at 84.6 vs. 83.2) corresponds to 7 questions out of 500. Whether this difference would replicate on a different set of questions or with different random seeds (if temperature > 0 were used) is unknown. The Appendi

6. Limitations and Trade-offs

Limitation 1: The PRM's Quality Defines an Implicit Upper Bound That Cannot Be Exceeded

The assumption or constraint. RSD's acceptance criterion reduces to a single scalar comparison: r(draft step) ≥ δ. Every decision—whether to keep the draft output or invoke the expensive target model—hinges on the PRM's judgment. The paper acknowledges this dependence indirectly when it notes that "different tasks have a different complexity of reasoning" (Table 2 caption) leading to different optimal thresholds, but it does not formalize what happens when the PRM is systematically wrong. The PRM is used off-the-shelf (Skywork-o1-Open-PRM); it is not fine-tuned on the draft model's output distribution and was not trained with RSD's specific routing objective in mind.

The consequence. There are two distinct failure modes. First, false acceptances: the PRM assigns a high score to an incorrect draft step (reward ≥ δ), the step is accepted, and the error propagates through the rest of the reasoning chain. Because RSD never revisits accepted steps—the target model only generates forward from the current context, it does not retroactively correct earlier accepted errors—a single false acceptance can derail the entire solution. Second, false rejections: the PRM assigns a low score to a correct draft step (reward < δ), the target model is invoked unnecessarily, and the target model may produce a worse step (recall from Figure 3, right panel, that the target model loses the reward comparison 37.3% of the time). In this case, the system spends expensive target model compute and potentially degrades quality.

The severity of this dependence is compounded by the binary step function's hardness: at δ = 0.7, a step with reward 0.69 is rejected while a step with reward 0.71 is accepted. If the PRM's scores are noisy or miscalibrated near the threshold, many decisions will be effectively random, accumulating errors over long reasoning trajectories (MATH500 averages 18 steps per question). The paper provides no calibration analysis of the PRM's scores—no reliability diagrams, no breakdown of false positive/negative rates near δ—so the practitioner has no way to assess whether the specific PRM they deploy is reliable enough for the binary threshold to be sensible.

What evidence exists in the paper. The GPQA Diamond anomaly in Table 2 is suggestive: RSD with a 1.5B PRM achieves 38.4% vs. 33.8% with a 7B PRM (both at δ = 0.7, math model family, 7B target). The larger, presumably more capable PRM produces worse routing decisions on this benchmark. The paper offers no explanation, but the most natural hypothesis is that the 7B PRM's reward distribution is miscalibrated for GPQA's scientific reasoning, leading to systematic false rejections (good draft steps scored below δ, forcing unnecessary and potentially harmful target model interventions) or false acceptances (bad steps scored above δ, letting errors through). This single data point does not establish a general pattern, but it demonstrates that PRM quality interacts with task type in ways that can reverse RSD's benefits. The robustness check across three PRM architectures (Table 5) shows consistent improvement over SD, which is reassuring, but all three PRMs were trained for math reasoning—none is specialized for science (GPQA) or general knowledge (MMLU STEM), leaving open the question of what happens with a genuinely mismatched PRM.

Mitigation status. The paper does not address PRM calibration, does not propose methods for detecting or recovering from PRM errors, and does not ablate sensitivity to PRM quality beyond the Table 5 architecture comparison (which tests different PRM variants but does not systematically degrade PRM quality to find the breaking point). The suggestion to train "a specialized PRM that is closely aligned with the draft model" (Section 4) acknowledges the limitation implicitly but provides no method or results. For a practitioner, the takeaway is that RSD's performance is upper-bounded by PRM quality, and that off-the-shelf PRMs not aligned with the specific draft model's error patterns may yield unpredictable routing behavior—particularly on tasks that differ from the PRM's training distribution.

Limitation 2: Wall-Clock Latency and Serial Dependencies Are Not Measured or Analyzed, and the FLOPs Reduction Is an Upper Bound on Practical Speedup

The assumption or constraint. The paper measures efficiency exclusively in FLOPs (Section 3.3), using the standard approximation of 2N FLOPs per token for a model with N parameters. FLOPs are a hardware-independent measure of total computational work, but they abstract away all latency, memory, and parallelism considerations. The paper acknowledges this implicitly by noting experiments were run on NVIDIA A100 GPUs using vLLM (Kwon et al., 2023), but reports no wall-clock timing, throughput, or latency numbers for any configuration. Section 4 mentions that "the average number of reasoning steps per question is 18" and that "adding an additional PRM for RSD incurs minimal overhead compared to SD," but this is stated qualitatively without latency quantification.

The consequence. RSD's per-step execution is inherently sequential in a way that standard autoregressive decoding is not:

  1. The draft model must generate a complete reasoning step (multiple tokens) before any evaluation occurs.
  2. The PRM must process the entire step (one forward pass over the step's tokens plus the full context) before the accept/reject decision.
  3. If rejected, the target model must generate a complete replacement step from scratch before the draft model can begin the next step.

This creates a serial dependency chain: draft → PRM → (potentially) target → draft. In contrast, the single target model baseline just generates continuously without these handoffs. Standard speculative decoding also has a draft-then-verify serial dependency, but it operates at the token level (the target model verifies a window of K draft tokens in one parallel forward pass), which can be pipelined more efficiently. RSD's step-level grain means the draft model generates 20–50 tokens before any verification, creating longer idle periods for the target model and PRM.

The practical consequence is that a 4.4× FLOPs reduction (RSD 7B/72B/7B vs. 72B target alone on MATH500, Figure 4) is extremely unlikely to translate to a 4.4× latency reduction. If the 72B target model can generate tokens continuously with high GPU utilization, while RSD's 7B draft model and 7B PRM have lower utilization (they must wait for each other), the wall-clock speedup could be much smaller—potentially below 2×, or even below 1× for certain batch sizes or hardware configurations. The worst case is when the target model is frequently invoked: the draft generates a step (cheap but takes time), the PRM scores it (cheap but takes time), the step is rejected, and the target model generates a replacement (expensive and takes time). The draft tokens from the rejected step are wasted work that consumed latency budget without contributing to the output.

What evidence exists in the paper. No latency or throughput measurements exist anywhere in the paper. Figure 4 plots FLOPs only. The paper does not discuss GPU utilization, memory bandwidth, batch size constraints, or the overhead of model switching (loading different model weights for draft vs. target vs. PRM). The model merging experiments (Section 4, Table 4) partially address the memory footprint concern—merging reduces the number of distinct models from three to two—but do not address the sequential dependency problem. The vLLM backend (Kwon et al., 2023) provides continuous batching and efficient KV-cache management, but RSD's per-query divergent execution paths (some queries need target model, some don't; some steps within a query need target model, some don't) could reduce batching efficiency compared to uniform decoding where all queries follow the same execution pattern. None of these batching dynamics are studied.

Mitigation status. The paper does not attempt to mitigate the latency limitation. Section 4's suggestion to combine RSD with standard SD for rejected steps—"during a rejected step, SD (draft+target) can be utilized to regenerate the step"—could reduce the cost of target model invocations but does not address the fundamental serial dependency. For latency-sensitive deployments (interactive assistants, real-time applications), a practitioner would need to run their own latency benchmarks with their specific hardware, batch sizes, and model configurations, because the paper provides no guidance on what speedup to expect in practice.

Limitation 3: The Evaluation Is Confined Almost Entirely to Math Reasoning, with Only a Preliminary Probe of General-Domain Tasks That Underperforms the Target Model

The assumption or constraint. The paper's main evaluation spans six benchmarks (Table 2): MATH500, GSM8K, GaoKao-2023-En, Olympiad Bench, GPQA Diamond, and MMLU STEM. Four are explicitly math benchmarks; GPQA tests graduate-level science (physics, chemistry, biology) reasoning that is structurally similar to math problem-solving; and MMLU STEM includes math, physics, and engineering questions. All six benchmarks share a common structure: they have objectively correct final answers (numeric, multiple choice), they require multi-step deductive reasoning, and they have natural step boundaries (the intermediate reasoning can be segmented by logical sub-conclusions, typically delimited by line breaks in model outputs). The paper acknowledges in Section 4 that "to the best of our knowledge, there is not yet a PRM for general-domain generation," and the AlpacaEval experiment is explicitly described as an exploration of "whether we could use an ORM instead of PRM in RSD?"

The consequence. RSD's core mechanism—step-level evaluation by a PRM, binary thresholding, target model fallback—is deeply coupled to the structure of reasoning tasks. Three assumptions are tested only within the math/science reasoning domain:

  1. Step boundaries exist and are detectable. The paper uses \n\n as the step delimiter, which works for math models trained to output step-by-step solutions but is unnatural for tasks like summarization, translation, dialogue, or creative writing. Without meaningful step boundaries, the PRM cannot evaluate coherent sub-units of reasoning, and the accept/reject decision becomes arbitrary.

  2. Intermediate quality is correlated with final correctness. The PRM is trained to predict whether a partial solution will lead to the correct final answer. This is well-defined for math (the reasoning is deductive) but not for open-ended generation where there is no ground-truth "correct" output—a step might be stylistically elegant, factually accurate, or engaging without being "correct" in any objective sense.

  3. An outcome or process reward model exists that can evaluate steps. For math, the paper uses Skywork-o1-Open-PRM, an open-source PRM trained on math reasoning. The AlpacaEval experiment substitutes an ORM (Skywork-Reward-Llama-3.1-8B-v0.2), but the results are substantially weaker: RSD achieves 18.85% win rate vs. GPT-4-Turbo, far below the target model alone at 24.47%. This is not a regime where RSD improves over the target model—it is a regime where RSD degrades performance compared to just using the large model, achieving only a partial recovery over the draft model (7.09%). The paper notes that "δ was not extensively tuned" and that "a general-domain PRM and dedicated tuning of δ could further boost the performance," but the gap is large enough (5.62 percentage points below the target model) that tuning alone seems unlikely to close it.

What evidence exists in the paper. Table 6 provides the only non-reasoning evaluation, and it undermines the generality claim. The win rate hierarchy is: target-only (24.47%) > RSD (18.85%) > draft-only (7.09%). RSD helps over the draft model, but it does not approach the target model's performance. The paper does not analyze why—possible explanations include: the ORM's scores are not calibrated for thresholding (the ORM output range is −∞ to ∞, and δ = 0 was chosen heuristically), the \n\n delimiter does not create meaningful step boundaries in open-ended text, or the reward signal is simply too noisy for routing decisions. The paper reports that "65% tokens are generated by the draft model only," confirming that the routing mechanism is active (the draft model is used for most tokens), but the routing decisions appear to be degrading quality rather than preserving or improving it.

Mitigation status. The paper is transparent that the AlpacaEval experiment is preliminary and that general-domain PRMs do not yet exist. Section 4 states that "a general-domain PRM and dedicated tuning of δ could further boost the performance," but this is entirely speculative. For a practitioner considering RSD for non-reasoning workloads, the paper provides no evidence that RSD can match or exceed the target model's quality—the only data point shows a substantial degradation. This limitation is fundamental: RSD in its current form is a method for reasoning tasks with available PRMs, and its extension to general-domain generation requires advances in reward modeling (general-domain PRMs, better ORMs for routing) that the paper does not provide.

Limitation 4: The Paper Evaluates Only a Single Threshold per Question; There Is No Adaptive or Per-Step Tuning

The assumption or constraint. The paper uses a global, static threshold δ (typically 0.7) for all steps in all questions. The acceptance criterion is uniform: every draft step with reward ≥ δ is accepted; every step with reward < δ is rejected and triggers the target model. This is the simplest possible routing policy, and the paper's theoretical analysis (Proposition 2.3) justifies it as optimal under a fixed budget constraint on expected target model invocations. However, the constraint is on average target model usage across all steps and questions, not on per-step or per-question usage. The paper does not explore whether adjusting δ dynamically—per-step, per-question, or adaptively based on the trajectory so far—could improve efficiency or accuracy.

The consequence. A global δ has two weaknesses:

  1. It cannot adapt to question difficulty. Figure B.1 shows that at δ = 0.7, the draft model alone solves 84% of Level 1 questions but only 19% of Level 5 questions. For a Level 1 question, a higher δ (e.g., 0.9) might be better—the draft is already accurate, and invoking the target model on the few low-reward steps could catch rare errors without much additional cost. For a Level 5 question, a lower δ might be better—the draft model is unreliable, and wasting time on PRM evaluations for steps that will mostly be rejected anyway might be less efficient than just using the target model for the entire question. The paper's analysis of difficulty-dependent behavior (Appendix B.2) demonstrates that the optimal threshold likely varies with difficulty, but the main experiments use a single δ for all questions.

  2. It cannot adapt within a trajectory. Some reasoning steps are more consequential than others. The first step of a solution (parsing the problem, setting up the approach) is often critical—an error there propagates to all subsequent steps. The final arithmetic step is usually simple and the draft model likely gets it right. A policy that uses a higher δ for early steps (conservative, prefer target model for setup) and a lower δ for later steps (aggressive, trust the draft for routine computation) could outperform the uniform policy. The paper does not explore this.

What evidence exists in the paper. Table B.2 provides per-task δ sweeps showing that the optimal threshold varies modestly across benchmarks—e.g., GPQA peaks at δ = 0.7 or 0.8 depending on configuration, GSM8K peaks at δ = 0.8 or 0.9, and MATH500 peaks at δ = 0.7. The "optimized threshold" δ* column in Table 2 tunes δ per-benchmark, yielding accuracy improvements of 0.2–1.0 points over the fixed δ = 0.7 setting. This demonstrates that per-benchmark tuning helps, but the paper does not test per-question or per-step adaptation. Figure B.1 shows accuracy curves for each difficulty level as a function of δ, revealing that the optimal δ for Level 1–3 questions is around 0.6–0.7, while for Level 4–5 questions, higher δ values (more target model) continue to improve accuracy—but the paper does not exploit this insight for adaptive thresholding.

Mitigation status. The paper does not address per-step or per-question threshold adaptation. It frames RSD as providing "automatic compute allocation" through the implicit mechanism that harder questions naturally have lower draft step rewards and thus trigger more target model invocations at a fixed δ (Section 3.4, Appendix B.2). This is a form of implicit adaptation, but it is coarse: a bad draft step on an easy question and a bad draft step on a hard question both get rejected at δ = 0.7, even though the target model's marginal value may differ. The paper's theoretical framework (Proposition 2.3) assumes a budget constraint γ on expected target model usage; a fixed δ is one way to satisfy this constraint, but an adaptive policy could potentially achieve higher reward under the same budget. Future work on dynamic thresholding—starting with a default δ and adjusting based on the trajectory's reward history, the question's estimated difficulty, or the step's position in the solution—could improve upon the static policy.

Limitation 5: The Paper Provides No Statistical Replication, Confidence Intervals, or Variance Estimates for Any Result, and the Per-Task "Optimized Threshold" Introduces Test-Set Overfitting

The assumption or constraint. All accuracy numbers in Tables 2, 3, 5, and B.1–B.3 are single-point estimates from one evaluation run per benchmark configuration. The paper does not report: standard deviations across multiple random seeds, confidence intervals, results from cross-validation splits, or any statistical tests comparing methods. For the FLOPs analysis (Figure 4), each point is a single run. The MATH500 test set has 500 questions; GSM8K has 1,319; the paper does not report exact sizes for GPQA Diamond, Olympiad Bench, GaoKao-2023-En, or MMLU STEM, but these are typically in the hundreds to low thousands of questions. On a 500-question test set, a 1-point accuracy difference corresponds to 5 questions—well within the range of sampling variability from a single run, particularly for methods with non-deterministic components (RSD's draft model uses temperature = 0 and is deterministic, but standard SD with temperature = 0 can still be affected by floating-point nondeterminism, as the paper notes citing Chen et al., 2023a).

The consequence. The paper's headline claims about accuracy improvements—RSD outperforms the target model by +0.8 to +1.2 points on average, and outperforms SD by +1.5 to +2.7 points—cannot be assessed for statistical reliability. A practitioner cannot determine whether the observed gains are genuine or could be reversed on a different set of questions, with a different random seed, or on a different evaluation run. This is particularly concerning for benchmarks where the margin is small: RSD (7B/72B/1.5B, δ = 0.7) achieves 72.2 average accuracy vs. 71.8 for the 72B target (+0.4). On six benchmarks with unknown total questions, a +0.4 average could be driven by one benchmark where RSD happened to get lucky on a few questions and another where it did slightly worse.

The per-task "optimized threshold" δ* column in Table 2 introduces an additional overfitting concern. The paper sweeps δ values (Table B.2) and selects the δ that gives the best accuracy on each benchmark's test set. This is explicitly test-set optimization: the threshold is chosen to maximize performance on the same data used for evaluation. The improvement from δ* over δ = 0.7 (+0.4 points on average for the math model family) cannot be interpreted as a genuine accuracy gain—it is at least partially the expected gain from optimizing a hyperparameter on the test set, which overfits to the specific questions. The paper mitigates this by also reporting the fixed δ = 0.7 results as the primary numbers, with δ* as a supplementary "what's possible with tuning" column, but the δ* numbers appear prominently in the main table and contribute to the paper's narrative of improvement.

What evidence exists in the paper. The paper provides no evidence regarding statistical reliability. There are no error bars on any figure, no standard deviations in any table, no mention of multiple runs or random seeds, and no statistical tests. For the threshold sweep (Figure 5), it's unclear whether the smooth curve is from a single run (where the apparent peak at δ = 0.7 could be noise) or averaged over multiple runs. The difficulty-level analysis (Figure B.1) shows accuracy curves with what appears to be substantial noise at the harder levels (Level 5 accuracy oscillates between 50% and 66% as δ varies), suggesting that per-bin sample sizes are small and single-run variance is large.

Mitigation status. The paper makes no attempt to address this limitation—no replication, no variance reporting, no mention of statistical methodology. This is consistent with common practice in the LLM benchmarking literature (where large-scale evaluations are expensive and single-run reporting is standard), but it is particularly consequential for a paper whose core claims involve small-magnitude accuracy improvements over strong baselines. The consistency of the direction of improvement across 16 configurations in Table 2 provides qualitative confidence, but a practitioner wanting to estimate the likely range of RSD's performance on their own task distribution would need to conduct their own repeated evaluations—the paper provides no guidance on expected variance.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual shift in how the field thinks about speculative decoding: it relabels the unbiasedness guarantee—long treated as the defining virtue of speculative decoding—from a safety net to a straitjacket on reasoning tasks, and proposes that a controlled, reward-guided bias can simultaneously improve both efficiency and output quality in ways that strict distributional fidelity cannot. This is not an incremental refinement of speculative decoding (like improved acceptance criteria or tree-based speculation within the unbiasedness framework); it is a category expansion that opens a new class of decoding methods—biased, quality-driven model routing—that the field had not previously explored because the unbiasedness guarantee was treated as definitional to speculative decoding.

The magnitude of this shift is reframing rather than paradigm-shifting. The paper does not claim that standard speculative decoding is obsolete or that unbiasedness is universally harmful. Rather, it identifies a specific regime—multi-step reasoning tasks where correctness and model-internal probability are poorly correlated—where the guarantee that previously ensured safety instead guarantees mediocrity: it forces the system to reject draft outputs that are correct but distributionally mismatched, and to accept draft outputs that are plausible but wrong. The paper's diagnosis is that "high-quality tokens (e.g., those favored by a process reward) may still be rejected if their probabilities under the large model are too low" (Section 1), and this is not an edge case but the dominant failure mode on reasoning workloads. By demonstrating that abandoning unbiasedness and routing based on an external quality signal yields better accuracy and better efficiency, the paper opens a design space that was previously closed by assumption.

This reframing resolves a latent tension in the literature. Prior work on speculative decoding consistently framed distributional fidelity as both necessary (users won't accept quality degradation) and sufficient (it preserves everything the large model can do). Yet practitioners observed that speculative decoding's efficiency gains collapse on reasoning tasks, and the standard diagnosis was that draft models need to be better at mimicking the target. The paper shows this diagnosis is partially wrong: the problem is not just that the draft model is a poor mimic, but that mimicking the target is the wrong objective. On CN Middle School 24 (Appendix B.1), the draft model (1.5B Qwen2.5-Math) achieves 75.2% while the target (7B) achieves only 72.3%—the draft outperforms the target, so forcing the system to match the target's distribution is actively harmful. Standard SD achieves 73.3%, worse than the draft alone; RSD achieves 78.2%. This single result crystallizes the paper's thesis: when correctness and model-internal probability diverge, distributional fidelity becomes a liability rather than a safeguard.

The work also connects two previously separate research communities. The speculative decoding literature (Leviathan et al., 2023; Chen et al., 2023a; Miao et al., 2024) has focused on efficient, lossless acceleration through draft-then-verify mechanisms, while the process reward model literature (Lightman et al., 2023; Chen et al., 2024a; Snell et al., 2024) has used PRMs for search, verification, and test-time compute scaling within a single model. RSD sits at the intersection, repurposing the PRM from a search heuristic into a model router. This synthesis suggests that reward models—which the reasoning community has been developing for capability amplification—can also serve as efficiency multipliers when used to dynamically allocate compute between models of different scales.

Several research directions become more attractive as a result of this work:

  • Biased decoding with external quality signals becomes a legitimate design paradigm rather than a theoretical compromise. The paper provides both a theoretical framework (the mixture distribution, Proposition 2.2's guarantee that RSD cannot be worse than the draft model, Proposition 2.3's optimality of threshold-based routing) and empirical validation that controlled bias can improve over unbiased methods. Future work on decoding acceleration no longer needs to accept the unbiasedness constraint as a hard requirement.

  • Process reward model training for routing becomes a distinct objective from PRM training for search. The paper observes that "training or fine-tuning a specialized PRM that is closely aligned with the draft model could further enhance performance" (Section 4) and that the GPQA anomaly (1.5B PRM outperforms 7B PRM by 4.6 points, Table 2) suggests off-the-shelf PRMs are not optimal for routing. A PRM trained specifically to maximize routing accuracy—distinguishing when the draft model is correct from when it is wrong—rather than to predict final-answer correctness, could substantially improve RSD's efficiency-quality frontier.

  • Architectures that combine models of different scales become more interesting. The paper shows that a system combining a 1.5B draft and a 7B target outperforms the 7B alone (66.1 vs. 65.3 average for Qwen2.5-Math with δ = 0.7, 7B PRM), and a system combining a 7B draft and a 72B target outperforms the 72B alone (72.7 vs. 71.8, Table 2). This suggests that heterogeneous model ensembles with quality-based routing can exceed the performance of any single model in the ensemble—not by averaging or voting, but by selectively amplifying each model's strengths.

Conversely, some directions become less attractive:

  • Improving standard speculative decoding's acceptance rate through better draft models (tree-based speculation, self-speculation, optimal transport acceptance criteria) remains valuable for non-reasoning tasks, but the paper's results suggest that on reasoning workloads, even a perfect draft model (one that matches the target's token distribution exactly) would be limited by the target model's own errors. RSD's advantage—potentially exceeding the target model's accuracy—cannot be achieved within the unbiasedness framework regardless of draft model quality.

  • Scaling test-time compute with search over a small model's outputs (beam search, process Best-of-N, majority voting) is shown to hit capability ceilings that no amount of search can overcome. Table 3 demonstrates that beam search with a 1.5B model peaks at 78.2% on MATH500 while RSD (same draft, same PRM, plus a 7B target) reaches 82.6%. For practitioners with access to models of different scales, routing to a larger model for difficult steps is more effective than exhaustively searching the small model's limited proposal space.

Follow-Up Research This Work Enables

1. Training a process reward model specifically for the routing objective rather than for final-answer correctness prediction.

The paper uses off-the-shelf PRMs (Skywork-o1-Open-PRM, Qwen2.5-Math-PRM) trained to predict whether a partial solution will lead to the correct final answer. This is not the same objective as the routing decision RSD needs: distinguishing whether the draft model's current step is correct versus incorrect, regardless of whether a different valid step could also be correct. A PRM trained for routing would need to be calibrated specifically on the draft model's output distribution—scoring the draft's actual proposed steps rather than generic reasoning quality. The GPQA anomaly (Table 2: 1.5B PRM outperforms 7B PRM by 4.6 points with the math model family) hints that PRM size and training distribution interact with task type in ways that affect routing quality. A concrete follow-up would train a PRM on (context, draft step, correctness) triples generated by the specific draft model being used, using the same Monte Carlo rollout supervision approach that Snell et al. (2024) used for their PRM training, but with the binary classification target being "does this draft step lead to a correct final answer?" rather than "is this step generally correct?" The hypothesis is that a draft-aligned PRM would improve both the acceptance rate of correct draft steps and the rejection rate of incorrect ones, shifting the efficiency-quality Pareto frontier upward across all δ values.

2. Evaluating RSD on code generation benchmarks with unit-test-based process rewards, measuring both efficiency and pass@k.

Code generation shares the structure that makes RSD effective on math reasoning: multi-step deductive production with objectively verifiable correctness (unit tests). Unlike math, code generation has a natural, cheap process reward signal: whether each line or block of code executes correctly given the preceding context, verifiable through incremental execution or static analysis. A concrete experiment would replicate RSD on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) using a small draft model (e.g., CodeQwen-1.5B), a large target model (e.g., CodeQwen-7B or DeepSeek-Coder-33B), and a PRM trained by executing partial code snippets in a sandbox and scoring them based on whether they produce the expected intermediate outputs (or compile without errors, or pass a subset of tests). The key metric would be whether RSD can achieve the same pass@1 as the large model at a fraction of the FLOPs, or whether—as on math—it can exceed the large model's accuracy by preserving correct draft-generated code blocks that the large model would not have produced. A negative result (RSD fails to improve over the target model because code correctness is more path-dependent than math reasoning) would refine our understanding of when reward-guided routing is beneficial.

3. Measuring the wall-clock latency and throughput of RSD under realistic serving conditions with continuous batching, and comparing to the FLOPs-based efficiency claims.

The paper reports up to 4.4× FLOPs reduction with RSD (7B/72B/7B) compared to the 72B target model alone (Figure 4), but provides no latency or throughput measurements. The serial dependencies in RSD's per-step loop (draft → PRM → decision → potentially target → next draft) mean the 4.4× FLOPs reduction is an upper bound on practical speedup. A concrete follow-up would deploy RSD using vLLM or TensorRT-LLM with a workload of 1000–5000 math reasoning queries, sweeping batch sizes from 1 to 64, and measuring: (1) median and P99 time-to-first-token and time-per-output-token for RSD vs. the target model alone vs. standard SD; (2) throughput in queries per second at a fixed accuracy target; (3) GPU utilization and memory bandwidth saturation for each model in the RSD pipeline. The hypothesis is that at low batch sizes (1–4), RSD's latency will be dominated by the serial draft-PRM-target handoffs, yielding a speedup closer to 1.5–2× rather than 4.4×; at high batch sizes (32–64), the target model's continuous batching efficiency may actually make the single target model more competitive. A negative result—RSD's wall-clock speedup is below 1× at any batch size—would be practically important, indicating that the method's FLOPs savings are consumed by scheduling overhead and GPU underutilization.

4. Combining RSD's reward-guided step-level routing with standard SD's token-level speculative verification during target model invocations.

The paper notes in Section 4 that "RSD is not inherently opposed to SD; in fact, they can be seamlessly combined." The concrete proposal: when the draft step is rejected (reward < δ) and the target model must generate a replacement step, instead of having the target model generate from scratch, use standard speculative decoding to generate the replacement step more efficiently—the draft model proposes tokens, and the target model verifies them in parallel using the standard likelihood-ratio acceptance criterion. This hybrid approach would reduce the cost of target model invocations (the most expensive part of RSD) while maintaining RSD's step-level routing advantages. A concrete experiment would measure the FLOPs and accuracy of RSD+SD hybrid against pure RSD and pure SD on MATH500, sweeping the number of speculative tokens from 1 to 7, with the hypothesis that the hybrid achieves RSD's accuracy at lower FLOPs by making each target model invocation cheaper. This experiment would also test whether the reward-guided routing (which decides when to invoke the target model) and the token-level speculative verification (which makes each invocation cheaper) are complementary rather than redundant.

5. Investigating whether adaptive, per-step thresholding outperforms the static global δ by conditioning the threshold on step position, trajectory reward history, or estimated question difficulty.

The paper uses a global δ (typically 0.7) applied uniformly to all steps and all questions. However, Figure B.1 shows that the optimal δ varies with difficulty level—harder questions benefit from more target model invocation (higher effective δ), and the paper's own analysis of difficulty-dependent behavior demonstrates that the same δ results in vastly different draft-only solve rates across difficulty tiers (84% for Level 1 vs. 19% for Level 5 at δ = 0.7). A concrete follow-up would test adaptive threshold policies: (1) position-dependent: use a higher δ for the first few steps (conservative, prefer target model for problem setup) and a lower δ for later steps (aggressive, trust the draft for routine computation); (2) trajectory-dependent: adjust δ downward if the previous N steps all had high reward (the trajectory is on track, trust the draft more) and upward if recent steps had borderline rewards (the trajectory may be veering off, bring in the target model); (3) difficulty-estimated: use a lightweight classifier or the PRM's average score on the first 2–3 draft steps to estimate question difficulty, then select a difficulty-appropriate δ for the remaining steps. The hypothesis is that adaptive policies can achieve the same or higher accuracy as the best fixed δ while using fewer target model invocations, because they apply the expensive model only where it has the highest expected marginal value.

6. A systematic study of RSD with deliberately degraded PRMs to characterize the relationship between PRM quality and RSD performance, and to identify the minimum PRM capability needed for RSD to outperform SD and the target model.

The paper shows that RSD works with three different PRM architectures (Table 5), but all three are strong, math-trained PRMs. The GPQA anomaly—where the 1.5B PRM outperforms the 7B PRM—hints at complex interactions between PRM quality, PRM size, and task alignment, but the paper provides no systematic characterization. A concrete follow-up would progressively degrade a PRM (by reducing training data, adding noise to labels, reducing model size, or introducing distribution shift between PRM training data and the evaluation task) and measure RSD's accuracy and efficiency at each degradation level. The key question: at what point does a noisy PRM cause RSD to underperform standard SD (which uses no PRM at all)? This would establish the "PRM quality floor" below which reward-guided routing is harmful, providing practical guidance for practitioners considering RSD in domains where strong PRMs don't exist. A particularly informative negative result would be: with a PRM achieving only 60% accuracy at classifying correct vs. incorrect reasoning steps, does RSD still outperform SD? If yes, RSD is robust to very weak reward signals; if no, PRM quality is a hard requirement that limits RSD's applicability to domains with mature reward models.

Practical Applications and Downstream Use Cases

1. Cost-efficient serving of math reasoning APIs for education technology platforms.

A platform providing step-by-step math tutoring (e.g., photomath-style apps, automated grading systems, or AI tutors) faces a common cost structure: most student queries are on routine problems where a small model suffices, but a minority are on competition-level or advanced problems that genuinely require large-model reasoning. RSD with δ = 0.7 automatically allocates compute across this difficulty spectrum: on MATH500, the draft model alone solves 84% of Level 1 questions and 67% of Level 2 questions without ever invoking the target model, while the target model is invoked primarily for Level 4–5 questions where the draft's accuracy is low (Appendix B.2). The paper's FLOPs analysis (Figure 4) shows that RSD (7B/72B/7B) achieves 88.0% accuracy on MATH500 at ~4.4× fewer FLOPs than the 72B model alone. For a platform serving 10 million queries per month, a 4× FLOPs reduction on the large model translates directly to a ~4× reduction in GPU-hours and associated energy costs, while actually improving accuracy on the hardest problems. The auto-adaptive nature of the routing—harder problems naturally trigger more target model invocations without requiring an explicit difficulty classifier—makes this particularly suitable for deployment where the query difficulty distribution is unknown or shifts over time.

2. On-device or edge deployment with cloud fallback for reasoning workloads.

A mobile coding assistant or math help application running on-device with a quantized 1.5B-parameter model faces a fundamental limitation: the on-device model can handle most routine queries but fails on complex multi-step reasoning. RSD provides a natural architecture for this setting: the draft model runs locally with low latency and zero network cost, a lightweight PRM (also running locally, potentially merged with the draft model as demonstrated in Table 4) scores each reasoning step, and when the score falls below δ, the step is sent to a cloud-hosted 7B or 72B target model for regeneration. The paper's difficulty-level analysis (Figure B.1) quantifies the cloud offload rate: at δ = 0.7, only 16% of Level 1 questions require any cloud intervention, while 81% of Level 5 questions do. This means the system can guarantee that the majority of queries (which are on easier problems) remain fully on-device with low latency and no data transmission, while still achieving near-cloud-model accuracy on hard problems by selectively invoking the cloud model. The model merging results (Table 4) further simplify on-device deployment: merging the draft model with the PRM reduces the on-device footprint from two models to one, making this architecture practical for devices with limited memory.

3. Batch inference for synthetic data generation in LLM self-improvement pipelines.

When generating synthetic training data for math reasoning (e.g., using an LLM to produce solution trajectories for fine-tuning a smaller model, as in STaR or ReST^EM), the cost-quality tradeoff is severe: using a 72B model to generate 100,000 solutions is accurate but expensive; using a 7B model is cheaper but produces lower-quality training data that may not improve the student model. RSD offers a middle ground: run RSD (7B/72B/7B) in batch mode to generate solutions where most steps are produced by the cheap 7B draft, but hard steps are handled by the 72B target. The paper's results (Table 2) show that RSD (7B/72B/7B) achieves 88.0% on MATH500—higher than either the 7B alone (83.2%) or the 72B alone (85.6%)—suggesting that the generated solutions would be higher quality than using either model independently, while costing substantially less than generating all solutions with the 72B model. The ~4.4× FLOPs reduction (Figure 4) directly translates to a 4.4× cost reduction for the generation phase. Moreover, RSD generates reasoning trajectories that naturally include both draft-generated and target-generated steps, providing diverse training examples that could help student models learn both efficient heuristics (from the draft's correct steps) and robust reasoning (from the target's corrections of draft errors).

4. Model-agnostic inference acceleration for organizations with existing heterogeneous model deployments.

Many organizations already deploy multiple model sizes (e.g., a fast 7B model for high-throughput tasks and a slow 70B model for high-quality tasks) behind a routing layer. RSD provides a principled, per-step routing mechanism that can be deployed on top of existing model infrastructure without retraining either the draft or target models. The paper's evidence that RSD works across three model families (Qwen2.5-Math, Qwen2.5-Instruct, Llama-3.1) with an off-the-shelf PRM (Table 2) suggests that practitioners can adopt RSD by simply adding a PRM evaluation step between draft and target model invocations, without fine-tuning. The δ = 0.7 default is robust across benchmarks (Table B.2), meaning minimal per-task tuning is needed. For organizations currently routing queries at the whole-request level (easy queries → 7B, hard queries → 70B), RSD's step-level routing provides finer-grained control: a single query can mix 7B-generated and 70B-generated steps, improving accuracy on medium-difficulty problems that are too easy to fully route to the 70B but too hard for the 7B to handle entirely on its own. The paper's finding that RSD (7B/72B/7B) improves over the 72B alone by +2.4 points on MATH500 (Table 2) demonstrates that step-level routing can extract more value from existing heterogeneous deployments than query-level routing alone.

When to Prefer This Method

The paper positions RSD as a replacement for standard speculative decoding specifically on multi-step reasoning tasks, while acknowledging that RSD requires a quality reward model (PRM or ORM) that standard SD does not. The decision boundary emerges clearly from the paper's evidence:

Prefer RSD over standard speculative decoding when:

  • The task involves multi-step reasoning where correctness and target-model-internal token probability are poorly correlated (math, code, scientific reasoning), as evidenced by RSD's consistent accuracy improvements over SD across all configurations in Table 2 (average +1.5 to +2.7 points depending on model family).
  • A process or outcome reward model is available that can evaluate intermediate outputs with reasonable accuracy—the PRM does not need to be perfect (RSD works with a 1.5B PRM, Table 2), but must be better than random at distinguishing correct from incorrect steps.
  • The deployment scenario can tolerate a small amount of additional memory (loading the PRM) in exchange for significant FLOPs reduction—the paper's 4.4× FLOPs reduction on MATH500 (Figure 4) establishes the upper bound on efficiency gains.
  • The draft model sometimes outperforms the target model on specific problem types—in this case, SD's unbiasedness forces worse quality than the draft alone, while RSD can preserve the draft's superior outputs (Appendix B.1, Table B.1).

Prefer standard speculative decoding over RSD when:

  • The task does not involve multi-step reasoning and a reward model is unavailable or costly—standard SD requires no external verifier and maintains the target model's quality guarantee, which is sufficient for open-ended generation where correctness is subjective.
  • The PRM's quality is unknown or likely poor on the target task—the paper provides no evidence for RSD with weak PRMs, and the AlpacaEval result (Table 6: 18.85% win rate for RSD vs. 24.47% for the target model alone) demonstrates that RSD can degrade below the target model when the reward signal is insufficiently reliable.
  • Latency is the dominant constraint and the serial draft-PRM-target handoff cannot be overlapped—the paper provides no latency data, but the step-level grain implies longer idle periods between model invocations.

Prefer scaling test-time compute with the target model (Best-of-N, majority voting) over RSD when:

  • The target model's accuracy ceiling is substantially above the draft model's and the compute budget allows multiple target samples—BoN (7B/7B, N = 64) achieves 86.2% on MATH500 (Figure 4), within 1.8 points of RSD (7B/72B/7B) at 88.0%, suggesting that multiple samples from a strong model can approach RSD's accuracy at the cost of much higher FLOPs. If FLOPs are plentiful but a second model size is unavailable, target-model sampling is a viable alternative.