Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding

🎯 Pitch

Speculative decoding can accelerate RL rollout generation by up to 2.5× at scale without altering the training distribution, but its gains critically depend on a new issue: draft models degrade aggressively the moment they lag behind a rapidly updating policy. The paper demonstrates that continuously reinitializing the draft from the latest learner weights is both necessary and sufficient to sustain speedups, while long-speculated off-the-shelf drafts collapse to near-zero acceptance rates within a handful of training steps.

1. Executive Summary

This paper integrates speculative decoding as a lossless acceleration primitive into an RL post-training framework, studying how draft-model-based generation acceleration can reduce the dominant cost of autoregressive rollout generation without altering the policy's sampling distribution. The work evaluates the system on 8B-scale mathematical reasoning workloads (Qwen3-8B under GRPO, validated on AIME-2024) using EAGLE-3 drafting—a general mechanism that works without native multi-token prediction heads—and demonstrates 1.8× rollout generation speedup on RL-Zero, translating to 1.35–1.41× end-to-end RL step speedup under synchronous execution, with indistinguishable validation accuracy trajectories. Simulator projections for a 235B model at deployment scale show rollout speedups exceeding 3× and end-to-end training speedup reaching approximately 2.5× at favorable operating points, establishing that speculative decoding composes with asynchronous execution as a complementary mechanism but yields gains that depend critically on draft initialization quality, draft length, and the generation share of the RL step.

2. Context and Motivation

The Core Problem: RL Rollout Generation Is the Bottleneck, and Existing Accelerations Change the Training Semantics

This paper addresses a specific, concrete systems problem: in reinforcement learning (RL) post-training of large language models, autoregressive rollout generation dominates wall-clock time, and most existing methods for reducing this cost alter the training distribution in ways that can degrade the learning signal. The paper does not propose a new RL algorithm or a new speculative decoding technique. Instead, it studies how to deploy speculative decoding—a well-established lossless inference acceleration method—as a systems primitive inside an RL training loop, and characterizes the operational decisions and deployment-scale regimes where it yields meaningful speedups.

To understand why this matters, one must first understand the anatomy of an RL training step for language models. In a typical GRPO-style pipeline (Shao et al., 2024), each training step involves: (1) generating rollout trajectories from the current policy (the model samples complete responses to training prompts), (2) computing log-probabilities of those trajectories under the current policy, (3) computing rewards (e.g., correctness on math problems) and advantages, and (4) updating the policy parameters via gradient descent. The paper reports in Table 1 that for their 8B-scale mathematical reasoning workloads, generation alone consumes 65–72% of total step time—133.6 seconds out of 185.3 seconds on RL-Think, and 100.0 seconds out of 151.2 seconds on RL-Zero. This imbalance is not an artifact of their particular setup; Section 1 cites multiple recent large-scale RL systems (Shen et al., 2024; Hu et al., 2025; Noukhovitch et al., 2025; Piché et al., 2025; Meta GenAI, 2025) where rollout generation is "routinely the single largest wall-clock component." The same trend is emerging in agentic RL, where multi-turn tool-use and web-interaction trajectories amplify the per-token generation cost (Jin et al., 2025; Qian et al., 2025; Wang et al., 2025b; Qi et al., 2025).

The problem, then, is clear: if you want to train frontier reasoning models faster, you must accelerate rollout generation. But the way you accelerate it matters enormously, because the training signal in on-policy RL depends on the distribution from which rollouts are sampled.

The Distributional Integrity Constraint: Why You Can't Just Generate Faster by Any Means

This brings us to the central tension that motivates the paper's approach. In on-policy RL for language models, the policy loss (GRPO, PPO, or similar) is computed using trajectories drawn from the current policy itself. The gradient estimate is unbiased only when the sampling distribution matches the policy being optimized. If you accelerate generation by changing that distribution—for example, by using a smaller quantized model to generate rollouts, or by reusing stale trajectories from an older policy—you introduce a distribution mismatch that can bias the gradient and degrade the final policy quality.

The paper explicitly frames this through a decomposition borrowed from Piché et al. (2025):

$\text{learning speed} = \text{effectiveness} \times \text{throughput}$

Here, throughput is how much rollout and training work the system completes per unit wall-clock time, and effectiveness is how much useful learning signal is extracted from that work. The tension is that many throughput-improving methods trade away effectiveness:

Asynchronous execution (Noukhovitch et al., 2025; Piché et al., 2025; Meta GenAI, 2025) overlaps generation with training, but the policy that generated the rollouts is stale by the time gradients are computed—this is policy lag, and it can slow or destabilize learning.
Off-policy replay and importance sampling (Li et al., 2025a; Zheng et al., 2025a; Wang et al., 2025a; Sheng et al., 2026) reuse trajectories from older policies, applying correction weights to account for the distribution shift. But importance sampling ratios can have high variance, especially when the policy has moved significantly, and the corrections are approximate.
Lower-precision rollouts (Xi et al., 2026; Qiu et al., 2026; Li et al., 2026) reduce per-token compute cost by quantizing the model, but the quantized model's output distribution differs from the full-precision policy's distribution—another form of mismatch.
Selective prompt filtering (Zheng et al., 2025b) skips rollouts on prompts deemed uninformative, which changes the effective training distribution.

Each of these is a valid engineering trade-off, and frontier model reports confirm that practitioners routinely combine several of them (DeepSeek-AI, 2024; Qwen Team, 2025; MiniMax Team, 2025a,b; Z.ai Team, 2025; DeepSeek-AI, 2025b; Kimi Team, 2026; Z.ai Team, 2026; NVIDIA, 2026). But each also changes the optimization semantics—the RL algorithm is no longer optimizing exactly the objective it was designed for. In some regimes, the degradation in effectiveness may be small enough to ignore; in others, it may matter substantially. The paper does not argue that these methods are wrong, only that they occupy a different point on the effectiveness-throughput Pareto frontier.

Speculative Decoding as a Lossless Alternative: The Gap in Prior RL Systems Work

Speculative decoding offers a fundamentally different trade-off. First introduced by Leviathan et al. (2023) and Chen et al. (2023), the core idea is: instead of generating tokens one at a time autoregressively, use a smaller, faster draft model to propose several tokens at once, then have the full target model verify them in a single forward pass. A rejection sampling procedure ensures that the accepted tokens follow exactly the target model's distribution—the draft model's predictions are only accepted when they match what the target model would have produced. This means speculative decoding provides lossless acceleration: the generated text is statistically indistinguishable from what the target model would have produced autoregressively, but it is generated faster.

Applied to RL rollouts, speculative decoding targets throughput without touching effectiveness. The rejection procedure guarantees that rollouts are drawn from the verifier (target) policy's distribution, so the RL training signal is—by construction—unchanged. The paper calls this property "verifier-exact training semantics": the optimization problem being solved is identical to the autoregressive baseline, just with lower latency.

Why wasn't this done before? The paper identifies a gap that is more about systems integration than algorithmic novelty. Prior work on speculative decoding focused overwhelmingly on inference serving—deploying a fixed, frozen model for user-facing applications. In that setting, the draft model is trained once and never updated, weight synchronization is irrelevant, and there is no concept of a moving policy that changes every training step. Integrating speculative decoding into an RL training loop requires solving several engineering challenges that don't arise in inference serving:

Weight synchronization: The policy model updates every RL step. The rollout engine (vLLM, in this paper's case) must receive new weights for both the target policy and (if applicable) the draft model. In a distributed training setup with separate learner and rollout nodes, this requires a coordinated weight transfer pipeline.
Draft-policy alignment: The draft model must remain aligned with the moving target policy to maintain high acceptance rates. If the policy drifts and the draft becomes stale, acceptance length drops, and the speedup evaporates or even reverses (as the paper shows with the weaker chat-domain draft initialization that achieves only 1.19× speedup on RL-Think before online adaptation; Table 3).
Log-probability correctness: KL penalties, importance weights, and the policy loss must all be computed against the target policy's log-probabilities, not the draft's. The system must ensure that the log-probability recomputation pass uses the verifier model, not the draft, even though the tokens were generated speculatively.
Telemetry and debugging: Operators need to monitor generation latency, acceptance length, and draft alignment throughout training, not just at deployment time. These metrics must be integrated into the RL training loop's observability infrastructure.

Two concurrent papers, FastGRPO (Zhang et al., 2025) and ReSpec (Chen et al., 2025), applied speculative decoding to RL systems, but with different emphases. FastGRPO focuses on concurrency-aware scheduling and online draft learning under high-concurrency group sampling—a scheduling optimization problem. ReSpec studies adaptive draft configurations and reward-weighted drafter adaptation—an algorithmic optimization of the draft itself. This paper's contribution is complementary: it studies end-to-end systems integration inside a production-grade RL stack (NeMo RL with vLLM backend), including the practical engineering requirements listed above, and provides a characterization of how speculative decoding composes with both synchronous and asynchronous execution across deployment scales. The paper positions itself not as competing with FastGRPO or ReSpec, but as filling a different level of the stack: the integration layer between speculative decoding algorithms and RL training orchestration.

Where Prior Efficiency Methods Fall Short, Specifically

To sharpen the motivation, it's worth examining the specific limitations of the main alternative approaches, as the paper implicitly contrasts against them:

Asynchronous RL has a fundamental tension between speedup and policy quality. The more you overlap generation with training (higher policy lag), the staler the rollout-generating policy becomes. At some lag threshold, the off-policyness degrades the gradient signal enough that the optimizer makes less progress per step, partially or fully negating the throughput gain. The paper's simulation results in Figure 4 show that for large models at small GPU counts, even modest policy lag can substantially reduce the benefit of all acceleration methods—speculative decoding included—because the generation share shrinks on the critical path. But the key point is that asynchronous execution and speculative decoding are orthogonal mechanisms: async overlap hides generation behind training, while speculation makes each generation cheaper. The paper shows they compose (Section 3.3, interaction with asynchronous execution), with speculation providing 1.24× end-to-end speedup in an async setting where much of the rollout cost is already hidden.

Off-policy replay with importance sampling faces a variance wall. Importance sampling ratios $\frac{\pi_{\text{new}}(a|s)}{\pi_{\text{old}}(a|s)}$ can explode when the new and old policies diverge, especially in the high-dimensional action space of token-by-token generation. Practitioners typically clip or truncate these ratios, which introduces bias. The paper does not empirically compare against replay methods, but the conceptual argument is clear: speculative decoding avoids this entire problem by generating on-policy rollouts in the first place.

Low-precision rollouts introduce a distribution mismatch that is hard to characterize. A quantized model may have subtly different token probabilities than the full-precision policy. Whether this matters for the final policy quality depends on the task, the quantization scheme, and the RL algorithm's sensitivity to distribution shift. The paper's approach sidesteps this uncertainty: speculative rollouts are verified against the full-precision target model, so there is no mismatch to worry about.

Selective prompt filtering changes the data distribution the policy trains on. If you filter out prompts where the model's current answer is already correct or where the reward signal is uninformative, you bias the training distribution toward harder or more uncertain prompts. This may be desirable or undesirable depending on the goal, but it is unquestionably a change to the optimization problem, not a pure acceleration primitive.

The Scale-Dependence of the Problem

An important motivator that the paper emphasizes is that the generation bottleneck worsens with model scale. Section 1 notes that at frontier model sizes, rollout generation is the dominant cost—a claim supported by multiple recent technical reports. This matters because it means the potential upside of speculative decoding grows with model size, even if the acceptance length remains constant. The reason is mechanical: larger models have longer per-token latencies, making each skipped autoregressive step more valuable, and larger models are typically deployed at larger GPU counts where the batch-per-instance shrinks, increasing the relative benefit of reducing per-token compute. The paper's simulator projections in Section 4 directly quantify this scale dependence, showing that for a 235B model, rollout speedups of 3.5× are achievable at favorable operating points, compared to 2.8–3.2× for the 8B model (Figure 4).

At the same time, larger models introduce new challenges: they occupy more GPUs per instance, making weight synchronization heavier; they produce longer reasoning traces in RL post-training (the paper observes the RL-Zero baseline latency rising sharply as the policy learns to produce longer chain-of-thought; Figure 2a), which increases the decode-to-prefill ratio—good for speculative decoding, since it only accelerates the decode phase; and they are more sensitive to deployment configuration, as the simulation results in Figure 4a demonstrate.

How the Paper Positions Itself

The paper positions its contribution at the systems integration layer, not the algorithmic layer. It does not claim to invent a new speculative decoding method—EAGLE-3 (Li et al., 2024, 2025b) and MTP heads (Gloeckle et al., 2024) are prior work. It does not claim to invent a new RL algorithm or training framework—GRPO (Shao et al., 2024) and NeMo RL (Shen et al., 2024) are prior work. What it contributes is:

A working integration of speculative decoding into a production RL stack, with specific engineering solutions for weight synchronization, draft alignment, log-probability correctness, and telemetry (Section 2.3 and Figure 1).
An empirical characterization of the operational decisions that determine realized speedup: draft initialization quality (Table 3), draft length (Table 4), online vs. offline draft adaptation (Table 5), and interaction with asynchronous execution (Section 3.3).
Simulator-based projections that map these findings to deployment-scale regimes, identifying where speculation yields meaningful gains (2.5× end-to-end at 235B scale under favorable conditions) and where the gains are bounded by non-generation stages or policy lag (Section 4).

The paper's framing in Section 2.2—the Amdahl's law bound $S_{\text{step}} \leq \frac{1}{R_{\text{gen}}/\alpha + (1 - R_{\text{gen}})}$ —captures the central insight: speculative decoding helps only to the extent that generation dominates step time ( $R_{\text{gen}}$ is high) and the draft is aligned enough to achieve high acceptance ( $\alpha$ is high). This bound, while simple, explains essentially all of the paper's empirical findings: why draft initialization matters (it affects $\alpha$ ), why longer draft lengths can be counterproductive (they increase per-step cost without proportionally increasing $\alpha$ ), why async execution reduces speculation's benefit (it lowers the effective $R_{\text{gen}}$ on the critical path), and why larger models benefit more (they have higher $R_{\text{gen}}$ due to longer per-token latencies and longer reasoning traces).

3. Technical Approach

3.1 Reader Orientation

The paper builds a system integration layer that adds speculative decoding—a lossless inference acceleration technique—as a first-class rollout primitive inside NeMo RL, a production-grade reinforcement learning post-training framework. The problem it solves is concrete: in RL training of language models, autoregressive rollout generation consumes 65–72% of wall-clock time per step, and existing acceleration methods (asynchronous execution, off-policy replay, low-precision rollouts) all trade away distributional fidelity for speed, potentially degrading the RL training signal. The solution takes the shape of a coordinated pipeline where a draft model proposes multiple tokens at once, the full policy model verifies them through a rejection sampling procedure that exactly preserves the policy's output distribution, and the system handles the moving-target complexity of RL (weight synchronization every step, draft-policy alignment, log-probability correctness) so that the optimization sees the same trajectories it would have seen under autoregressive generation—just generated faster.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, connected in a loop:

vLLM Rollout Engine (with speculative decoding) — a serving backend that generates complete response trajectories from training prompts. It runs the target policy model for verification and the draft model for token proposal, coordinating them through EAGLE-3 speculative decoding to produce tokens faster than autoregressive generation, while guaranteeing that all accepted tokens follow the target policy's exact distribution.
MegatronLM Policy Model (Learner) — the language model being trained via RL. After rollouts are generated, this component runs the forward pass to recompute log-probabilities of the generated trajectories under the current policy weights (needed because the rollout engine may have used slightly stale weights due to asynchronous weight transfer), computes the GRPO policy loss, and produces gradient updates.
Draft Model (EAGLE-3 head) — an auxiliary neural network, separate from the policy model, that predicts multiple future tokens given the current policy's hidden states. It is trained on the policy's own outputs to maximize acceptance rate during speculative decoding. It can remain frozen (offline drafting) or be updated during training (online adaptation) using hidden-state and log-probability caches from the policy forward pass.
Weight Synchronization Pipeline — a coordination mechanism that moves updated policy weights from the MegatronLM learner to the vLLM rollout engine after each RL step. When online draft adaptation is enabled, draft weights are also synchronized. This pipeline must be fast enough to not become a new bottleneck.
Caching Pathway for Online Draft Training — a gradient-detached data pathway that routes hidden states and log-probabilities from the MegatronLM forward pass (which computes the GRPO loss) to a separate speculative-decoding loss function that trains the draft head. The .detach() operation ensures that draft training gradients do not flow back into the policy model, preserving the integrity of the RL optimization signal.

Information flow per RL step:

The learner (MegatronLM) synchronizes its latest policy weights to the vLLM rollout engine.
vLLM generates rollout trajectories on training prompts using speculative decoding: the draft model proposes candidate token sequences, the target policy verifies them via rejection sampling, and only distribution-matching tokens are accepted.
The generated trajectories (token sequences and their associated draft metadata) are sent back to the learner.
MegatronLM recomputes log-probabilities of these trajectories under the current policy weights (the verifier forward pass).
Rewards are computed (correctness on math problems), advantages are estimated, and the GRPO policy loss is computed.
Policy gradients are computed and applied, updating the policy model weights.
If online draft adaptation is enabled, the hidden states and log-probabilities from step 4 are routed (via .detach()) to a separate draft training loss, which updates the EAGLE-3 draft head to stay aligned with the evolving policy.
The cycle repeats from step 1.

3.3 Roadmap for the Deep Dive

First, the learning speed decomposition and Amdahl's law bound (Section 2.1–2.2 in the paper), which defines the mathematical framework for understanding when speculative decoding helps and by how much. This establishes the quantitative relationship between generation share, acceptance length, and achievable speedup—the analytical backbone that explains all subsequent empirical findings.
Second, the speculative decoding mechanism itself (the verification procedure), since it is the core guarantee of lossless acceleration. Understanding the rejection sampling step is necessary to see why "verifier-exact training semantics" holds and what the draft model must achieve.
Third, the EAGLE-3 draft architecture and training, because the draft model is the active component that determines acceptance length—the key variable in the speedup equation. The distinction between offline and online draft maintenance, and between general (EAGLE-3) and native (MTP) drafting paths, shapes the system's deployment flexibility.
Fourth, the system integration architecture (Figure 1), covering weight synchronization, the gradient-detached caching pathway for online draft training, and the coordination between vLLM and MegatronLM. This is where the paper's engineering contribution lives: the specific mechanisms that make speculative decoding work inside a moving-policy RL loop rather than a frozen inference server.
Fifth, the composition with synchronous and asynchronous RL, explaining how speculation interacts with policy lag to determine end-to-end speedup. Understanding this interaction requires the Amdahl's law framework established at the beginning.
Sixth, the simulator methodology (Section 4.1), which extends the analysis to deployment-scale regimes not directly tested. This section explains how the simulator models GPU compute, memory, interconnect, sharding strategies, and long-tailed response length distributions to project speedups at 235B scale and up to 2048 GPUs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems integration paper whose core idea is that speculative decoding can be deployed as a rollout acceleration primitive inside an RL training loop, preserving the policy's sampling distribution while reducing generation latency, provided the system handles weight synchronization, draft-policy alignment, and log-probability correctness—and that the realized speedup is governed by an Amdahl's law bound that depends on the generation share of the RL step and the draft model's acceptance length.

The Learning Speed Decomposition and Amdahl's Law Bound

The paper begins its technical analysis by framing the RL training problem through a decomposition that separates the two levers any acceleration method can pull (Section 2.1):

$\text{learning speed} = \text{effectiveness} \times \text{throughput}$

where effectiveness is how much useful learning signal is extracted from each unit of computation (measured in policy improvement per RL step), and throughput is how much computation the system completes per unit wall-clock time (measured in RL steps per second).

This decomposition matters because it makes explicit a trade-off that many acceleration methods exploit: they increase throughput at the cost of reduced effectiveness. Asynchronous execution increases throughput by overlapping generation with training but reduces effectiveness because the policy that generated the rollouts is stale. Off-policy replay increases throughput by reusing old trajectories but reduces effectiveness because importance sampling corrections are approximate and high-variance. Low-precision rollouts increase throughput by reducing per-token compute but reduce effectiveness because the quantized model's output distribution differs from the full-precision policy.

Speculative decoding operates differently: it targets throughput without changing effectiveness at all. The rejection sampling procedure (described in detail below) guarantees that every accepted token is drawn from exactly the target policy's distribution, meaning the rollouts are statistically indistinguishable from autoregressive generation. The RL training signal—the gradient of the policy loss with respect to the policy parameters—is therefore unchanged. The paper calls this property "verifier-exact training semantics" because the verifier model (the target policy) sees the same distribution of trajectories regardless of whether speculation is used.

However, speculative decoding does not accelerate all parts of the RL step equally. To understand where the speedup ceiling comes from, the paper decomposes a synchronous RL step into its constituent stages (Section 2.2):

$T_{\text{step}} = T_{\text{data}} + T_{\text{prepare}} + T_{\text{gen}} + T_{\text{logprob}} + T_{\text{train}}$

where $T_{\text{data}}$ is the time to load and preprocess training prompts, $T_{\text{prepare}}$ is the time for weight synchronization and rollout backend preparation (moving updated policy weights from the learner to the vLLM engine), $T_{\text{gen}}$ is the time for rollout generation (the autoregressive decode phase where the model samples response tokens one by one), $T_{\text{logprob}}$ is the time for log-probability recomputation (running the policy forward pass on the generated trajectories to get token-level probabilities under the current weights), and $T_{\text{train}}$ is the time for advantage computation and policy optimization (the gradient computation and parameter update).

Speculative decoding targets only $T_{\text{gen}}$ , and within generation, only the autoregressive decode phase (not the prefill phase where the prompt is processed). The other stages— $T_{\text{data}}$ , $T_{\text{prepare}}$ , $T_{\text{logprob}}$ , $T_{\text{train}}$ —are completely unaffected.

Given this decomposition, the paper derives an upper bound on the end-to-end step speedup using Amdahl's law. The bound assumes that each speculation step costs exactly the same as one autoregressive forward pass (a simplifying assumption that ignores draft model overhead, prefill time, and batching effects), and that speculative decoding accelerates the generation stage by a factor equal to the mean acceptance length $\alpha$ (the average number of tokens produced per speculation step):

$S_{\text{step}} \leq \frac{1}{R_{\text{gen}}/\alpha + (1 - R_{\text{gen}})}$

where $S_{\text{step}} = T_{\text{step}}^{\text{AR}} / T_{\text{step}}^{\text{Spec}}$ is the ratio of autoregressive step time to speculative step time (the end-to-end speedup), $R_{\text{gen}} = T_{\text{gen}} / T_{\text{step}}$ is the fraction of total step time spent on generation in the autoregressive baseline (the generation share), and $\alpha$ is the mean acceptance length (the average number of tokens accepted per speculative forward pass).

What this equation computes in operational terms: given the generation share $R_{\text{gen}}$ (measured from profiling the autoregressive baseline) and the acceptance length $\alpha$ (measured from running speculative decoding on the same workload), the bound tells you the maximum possible end-to-end speedup, assuming perfect acceleration of the generation stage and zero overhead from the draft model. The denominator $R_{\text{gen}}/\alpha + (1 - R_{\text{gen}})$ is the fraction of original step time remaining after acceleration: the $R_{\text{gen}}/\alpha$ term is the new generation time (the original generation time divided by the acceptance length), and the $(1 - R_{\text{gen}})$ term is the unchanged non-generation time. The reciprocal converts this remaining fraction into a speedup factor.

Why this form: Amdahl's law is the standard way to bound the speedup from accelerating only a fraction of a workload. The specific form $1/(f/s + (1-f))$ captures that even infinite acceleration of the optimizable fraction $f$ can at best yield speedup $1/(1-f)$ , determined entirely by the unoptimizable remainder. The paper uses this form because it explains, in a single equation, essentially all of their empirical findings:

Why draft initialization matters so much: poor initialization reduces $\alpha$ , directly shrinking the achievable speedup. If $\alpha$ is low enough, the denominator $R_{\text{gen}}/\alpha + (1 - R_{\text{gen}})$ can exceed 1, meaning speculative decoding is slower than autoregressive (as observed with $n$ -gram drafting in Table 2 and with $k=5,7$ drafts on RL-Think in Table 4).
Why longer draft lengths can be counterproductive: increasing $k$ increases the per-speculation-step cost (more draft tokens to verify), which effectively reduces the net $\alpha$ per unit of draft overhead. The bound assumes constant per-step cost, but the empirical finding that $k=3$ is optimal reflects that larger $k$ increases verification cost faster than it increases acceptance.
Why asynchronous execution reduces speculation's benefit: async overlap hides generation behind training, reducing the effective $R_{\text{gen}}$ on the critical path (the portion of generation time that is not overlapped and therefore still contributes to wall-clock step time). A smaller $R_{\text{gen}}$ means a smaller fraction of step time is optimizable, tightening the Amdahl ceiling.
Why larger models benefit more: they have longer per-token latencies and longer reasoning traces, both of which increase $R_{\text{gen}}$ by making generation a larger fraction of total step time.

The paper is careful to note that this bound is an upper bound that does not account for draft model overhead, prefill time, or batching effects. Realized speedup will always be below this bound. Its purpose is analytical, not predictive: it provides a framework for reasoning about why certain configurations yield better speedups than others.

The Speculative Decoding Verification Procedure

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) is the core mechanism that guarantees lossless acceleration. The paper uses it as a black-box primitive, but understanding its rejection sampling procedure is essential to understanding the "verifier-exact" guarantee and the system requirements that follow.

In standard autoregressive decoding, the model generates one token at a time: given the current context (the prompt plus all previously generated tokens), the model computes a probability distribution over the vocabulary, samples a token from this distribution, appends it to the context, and repeats. Each step requires one forward pass through the model. If the model needs to generate $L$ tokens, this takes $L$ sequential forward passes, each dependent on the output of the previous one.

Speculative decoding breaks this sequential dependency by interposing a draft model $M_{\text{draft}}$ that is smaller and faster than the target model $M_{\text{target}}$ . The procedure for one speculation step with draft length $k$ works as follows:

Step 1: Draft proposal. The draft model $M_{\text{draft}}$ takes the current context and generates $k$ draft tokens $x_1, x_2, ..., x_k$ autoregressively. Because the draft model is smaller, these $k$ forward passes are much cheaper than $k$ forward passes through the target model. The draft tokens are not necessarily what the target model would have produced; they are the draft model's best guess.

Step 2: Target verification. The target model $M_{\text{target}}$ takes the original context plus all $k$ draft tokens and runs one single forward pass. From this single forward pass, it computes the probability distribution at each of the $k+1$ positions: the probability of the first draft token given the original context, the probability of the second draft token given the original context plus the first draft token, and so on through the probability of what would come after the $k$ -th draft token.

Step 3: Rejection sampling. For each position $i$ from 1 to $k$ , the system performs a sequential rejection test. At position $i$ , it samples a random number $r_i$ uniformly from $[0, 1]$ and compares it to the ratio:

$r_i < \frac{p_{\text{target}}(x_i \mid \text{context}_{i-1})}{p_{\text{draft}}(x_i \mid \text{context}_{i-1})}$

where $p_{\text{target}}(x_i \mid \text{context}_{i-1})$ is the probability the target model assigns to draft token $x_i$ given the context up to position $i-1$ , and $p_{\text{draft}}(x_i \mid \text{context}_{i-1})$ is the probability the draft model assigned to the same token when it was originally proposed. If the inequality holds, token $x_i$ is accepted and the procedure moves to position $i+1$ . If the inequality fails, token $x_i$ is rejected, all remaining draft tokens $x_{i+1}, ..., x_k$ are discarded, and the target model samples a new token from a modified distribution at position $i$ :

$p_{\text{residual}}(x) = \text{norm}\left(\max\left(0, p_{\text{target}}(x \mid \text{context}_{i-1}) - p_{\text{draft}}(x \mid \text{context}_{i-1})\right)\right)$

where the normalization ensures this is a valid probability distribution. The sampled token replaces $x_i$ and the speculation step ends, having accepted $i-1 tokens and generated one new token.

Why this procedure preserves the target distribution: The rejection sampling step is designed so that, regardless of what the draft model proposes, the set of accepted tokens is drawn exactly from the distribution $p_{\text{target}}$ would have produced autoregressively. The key insight is that when $p_{\text{draft}}(x_i) \leq p_{\text{target}}(x_i)$ , the token is accepted with probability $p_{\text{draft}}(x_i) / p_{\text{target}}(x_i)$ times the draft's probability of proposing it, which yields exactly $p_{\text{target}}(x_i)$ marginal probability. When $p_{\text{draft}}(x_i) > p_{\text{target}}(x_i)$ , the excess probability mass $p_{\text{draft}}(x_i) - p_{\text{target}}(x_i)$ is redistributed via the residual sampling step to other tokens, again matching $p_{\text{target}}$ exactly. This is the lossless guarantee: the marginal distribution of generated text is identical to what the target model would have produced autoregressively, even though generation is faster.

What acceptance length $\alpha$ means: The mean acceptance length is the average number of draft tokens accepted per speculation step. If the draft model perfectly predicts the target model's output (i.e., $p_{\text{draft}} = p_{\text{target}}$ for all proposed tokens), then $\alpha$ approaches $k+1$ (all $k$ draft tokens are accepted, plus the target model generates one additional token to complete the step). In practice, $\alpha$ is always less than $k+1$ because the draft model is imperfect. The acceptance length determines the effective speedup of the generation stage: if $\alpha$ tokens are produced per target forward pass (which costs roughly the same as one autoregressive forward pass), then generation is accelerated by approximately a factor of $\alpha$ .

The cost model: The paper assumes that one target model forward pass (verification) costs approximately the same as one autoregressive forward pass. This is a reasonable approximation because the verification pass processes $k+1$ positions in parallel, and modern transformer implementations are optimized for processing multiple positions simultaneously (the compute cost scales sub-linearly with sequence length due to the quadratic attention being memory-bandwidth-bound for short sequences). The draft model's forward passes are assumed to be much cheaper than the target model's (since the draft is smaller), making their cost negligible in the overall accounting. This cost model is the basis for the $\alpha$-times speedup claim.

Implications for RL integration: The rejection sampling procedure has two critical implications for RL systems integration. First, the draft model never appears in the training signal. The policy loss, KL penalties, and advantage estimates are all computed using the target policy's log-probabilities, which are recomputed from scratch during the $T_{\text{logprob}}$ stage using MegatronLM with the current policy weights. The draft model's probabilities are used only during the rejection step in vLLM and are discarded afterward. This means the draft model can be trained independently, using a separate loss function, without interacting with the RL optimization objective.

Second, the acceptance length $\alpha$ depends on how well the draft model matches the target policy's distribution. As the policy updates during RL training, its output distribution shifts. If the draft model is fixed (offline drafting), the acceptance length will drift over time as the policy moves away from the distribution the draft was trained on. If the acceptance length drops too low, the speedup evaporates or even reverses (as shown in Table 4, where $k=5$ and $k=7$ drafts on RL-Think are slower than autoregressive, and in Table 3, where a chat-domain draft achieves only 1.19× speedup on RL-Think). This motivates the online draft adaptation mechanism (Section 2.3 and Table 5), which periodically retrains the draft to track the moving policy.

The EAGLE-3 Draft Architecture and Training

The paper uses EAGLE-3 (Li et al., 2024, 2025b) as its primary drafting mechanism, deliberately choosing it as "the harder case" (Section 3.1) because it requires training and maintaining an external draft model that must be kept aligned with a moving policy. The authors note that a native multi-token prediction (MTP) path (Gloeckle et al., 2024) is also supported, where the target model's own built-in auxiliary heads serve as the draft, but they focus their empirical study on EAGLE-3 because the system-level findings (draft initialization, draft length, online adaptation, composition with asynchronous execution) carry over directly to the MTP case.

What EAGLE-3 does: EAGLE-3 is a speculative decoding architecture where the draft model is an auxiliary head attached to the target model's hidden states. Unlike older approaches that use a completely separate smaller language model as the draft (e.g., a distilled student model), EAGLE-3 reuses the target model's own computations: it takes the hidden states from one or more layers of the target model during the verification forward pass and processes them through a lightweight draft network to predict multiple future tokens. This design has two advantages. First, the draft model is very small (it is just a few transformer layers operating on already-computed hidden states, not a full language model with its own embedding and attention layers), so its forward passes are extremely cheap. Second, because it operates on the target model's representations, it naturally tracks the target model's behavior—the hidden states encode the target model's understanding of the context, so the draft head can learn to predict what the target model would output.

EAGLE-3 draft training: The draft model is trained using standard supervised learning on data generated by the target policy. For each training prompt, the target model generates a complete response autoregressively, and the hidden states at each position are recorded along with the ground-truth next tokens. The draft model is then trained to predict the next $k$ tokens given the hidden states at the current position. The training objective is the standard cross-entropy loss between the draft's predicted token probabilities and the ground-truth tokens, summed over all $k$ prediction positions:

$\mathcal{L}_{\text{draft}} = -\sum_{i=1}^{k} \log p_{\text{draft}}(x_{t+i} \mid h_t, x_{t+1}, ..., x_{t+i-1})$

where $h_t$ is the target model's hidden state at position $t$ , and $x_{t+1}, ..., x_{t+k}$ are the ground-truth next $k$ tokens generated by the target model. The draft model is auto-regressive in its predictions: it first predicts $x_{t+1}$ from $h_t$ , then conditions on $x_{t+1}$ to predict $x_{t+2}$ , and so on up to $x_{t+k}. The loss sums the negative log-probability of each ground-truth token under the draft's predicted distribution at that step.

Draft initialization (Table 3): The quality of the draft model's initialization has a first-order effect on realized speedup because it directly determines the initial acceptance length $\alpha$ . The paper experiments with two initialization strategies:

UltraChat initialization: Train the EAGLE-3 draft on general chat data (UltraChat and Magpie; Ding et al., 2023; Xu et al., 2025). This produces a draft model that is good at predicting tokens in generic conversational contexts but has not been exposed to the mathematical reasoning traces that the RL training will produce. At $k=3$ , this draft achieves acceptance length 2.88 on RL-Zero and 2.40 on RL-Think, yielding speedups of 1.51× and 1.19× respectively.
DAPO initialization: Train the EAGLE-3 draft on responses generated by the policy model on the DAPO-Math-17K training prompts (Yu et al., 2025)—the same prompts and policy used for RL training. This produces a draft model that is aligned to the exact rollout distribution the RL training will encounter. At $k=3$ , acceptance length rises to 3.32 on RL-Zero and 2.77 on RL-Think, yielding speedups of 1.77× and 1.53× respectively—substantial improvements from 1.51× to 1.77× (+17%) and from 1.19× to 1.53× (+29%).

The paper's key insight about initialization is not just that in-domain data is better (that is expected), but that the gap between generic and in-domain drafting widens in harder RL regimes. RL-Think continues training from an already-reasoning-capable checkpoint, so its outputs already contain structured chain-of-thought reasoning—a distribution that is somewhat closer to general chat than the RL-Zero outputs, which start from a base model with no reasoning training. The UltraChat draft therefore performs relatively better on RL-Think (acceptance 2.40, speedup 1.19×) than it would if the policy were completely out-of-domain, but still substantially worse than the DAPO-initialized draft (acceptance 2.77, speedup 1.53×).

Draft length (Table 4): The draft length $k$ —the number of tokens the draft model proposes per speculation step—has a non-monotonic relationship with realized speedup. The paper sweeps $k \in \{3, 5, 7\}$ and finds that $k=3$ is consistently optimal. On RL-Zero, acceptance length increases from 3.32 to 4.35 to 5.06 as $k$ increases, but speedup falls from 1.77× to 1.44× to 1.21×. On RL-Think, the effect is more dramatic: speedup drops from 1.53× at $k=3$ to 0.84× at $k=5$ and 0.71× at $k=7$ , meaning the longer drafts are slower than autoregressive decoding.

The explanation lies in the cost model. Each speculation step involves two costs: (1) the draft model's $k$ autoregressive forward passes to propose tokens, and (2) the target model's one forward pass to verify all $k+1$ positions. As $k$ increases, the verification pass becomes more expensive because it processes more positions (though the marginal cost per additional position is small due to parallel processing). Meanwhile, the acceptance length $\alpha$ grows sub-linearly with $k$ —each additional draft token is less likely to be accepted than the previous one, because the draft model's predictions become less accurate at longer horizons. The net effect is that the cost per accepted token (total speculation cost divided by $\alpha$ ) reaches a minimum at small $k$ and then rises.

The paper's phrasing captures this precisely: "Larger drafts increase speculative work enough to erase the benefit of higher acceptance." The Amdahl's law bound (Section 2.2) assumed constant per-speculation-step cost, but in reality the cost grows with $k$ , making the effective $\alpha$ (acceptance per unit cost) lower than the raw acceptance length would suggest.

Online draft adaptation (Table 5): The paper studies whether periodically retraining the draft model during RL training—using trajectories generated by the current policy as supervision—improves speedup. When online adaptation is enabled, the system reuses the hidden-state and log-probability caches from the same MegatronLM forward pass that computes the GRPO policy loss. This avoids an additional policy recomputation for draft training: the draft head is trained using a separate loss function on the same hidden states that were already computed for the policy forward pass.

The results show that online adaptation provides limited additional gains when the draft is already well-initialized. For the DAPO-initialized draft on RL-Zero, online adaptation changes the acceptance length from 3.32 to 3.29 and the speedup from 1.77× to 1.78×—a negligible difference. For RL-Think, it changes acceptance from 2.77 to 2.74 and speedup from 1.53× to 1.52×—also negligible.

The larger benefit appears for the weaker UltraChat initialization. On RL-Zero, online adaptation improves acceptance from 2.88 to 3.04 and speedup from 1.51× to 1.63× (+8%). On RL-Think, it improves acceptance from 2.40 to 2.55 and speedup from 1.19× to 1.26× (+6%). These are modest gains, but they demonstrate that online adaptation can partially compensate for a poor initialization by incrementally aligning the draft to the policy's evolving distribution.

The paper characterizes online draft adaptation as "insurance against distribution mismatch rather than a general improvement strategy." If the draft is already well-aligned at initialization, the policy's distribution shift during training is small enough that the draft remains effective without updates. The computational cost of online adaptation (running the draft training loss and updating draft weights) is therefore not justified in this regime. But if the draft must be initialized from generic data (e.g., because in-domain policy outputs are not available before training starts), online adaptation can recover some of the lost speedup.

The gradient-detached pathway (Figure 1): A critical design choice for online adaptation is that the draft training loss must not interfere with the policy gradient signal. The system implements this by routing the hidden-state cache through a .detach() operation before it reaches the draft head. This means that gradients from the draft loss flow only into the draft head parameters, not back into the policy model's transformer layers. If this detachment were not present, the draft training loss would contribute to the policy's gradient, effectively adding a multi-token prediction auxiliary objective to the RL training—which would change the optimization semantics and violate the "verifier-exact" guarantee.

The paper describes this as: "The hidden-state cache is routed through a gradient-detached pathway to the draft head, so that draft training does not interfere with the policy gradient signal." In PyTorch terms, the hidden states are detached from the computation graph before being passed to the draft loss function, creating a branch in the graph where gradients flow only to the draft parameters.

System Integration Architecture

The system integration is the paper's primary engineering contribution: making speculative decoding work inside an RL training loop where the policy changes every step. This section details the four components from Figure 1 and their coordination.

vLLM Rollout Engine: The vLLM backend is responsible for generating rollout trajectories from training prompts using speculative decoding. It runs two models: the target policy model (the language model being trained, loaded with the latest weights synchronized from the learner) and the draft model (EAGLE-3 head, either frozen or online-updated). For each training prompt in the batch, vLLM executes the speculative decoding procedure described above: the draft proposes token sequences, the target verifies them, and accepted tokens form the rollout trajectory.

The vLLM engine is configured to generate complete responses—not just single tokens—because RL training requires full trajectories to compute rewards (e.g., whether the final answer to a math problem is correct). The generation continues until the model produces an end-of-sequence token or reaches a maximum length limit. The engine returns the complete token sequence for each prompt, along with metadata needed for downstream processing (e.g., which tokens were accepted vs. rejected, though this metadata is not used in the current implementation since log-probabilities are recomputed from scratch).

The choice of vLLM as the serving backend is motivated by its support for continuous batching and PagedAttention, which enable efficient handling of the variable-length responses typical in RL rollouts. Reasoning traces in mathematical RL can vary dramatically in length—the paper observes RL-Zero baseline latency rising sharply in the first ~100 steps as the policy learns to produce longer chain-of-thought (Figure 2a). vLLM's dynamic memory management handles this variability efficiently.

MegatronLM Policy Model (Learner): MegatronLM is the training engine that runs the policy model's forward and backward passes. After vLLM generates rollout trajectories, MegatronLM takes over for two computations:

Log-probability recomputation ( $T_{\text{logprob}}$ ): The rollout trajectories were generated by the vLLM engine, which may have used slightly stale policy weights (due to the time lag between the last weight synchronization and the current training step). To ensure the RL loss is computed correctly, MegatronLM runs a forward pass of the current policy on the generated trajectories, computing the log-probability of each token position under the current weights. These log-probabilities are used to compute importance sampling ratios (for KL penalties or advantage estimation) and the policy gradient.
GRPO policy loss and gradient computation ( $T_{\text{train}}$ ): Using the recomputed log-probabilities and the rewards (computed from the generated trajectories, e.g., via string matching on math answers), MegatronLM computes the GRPO policy loss (Shao et al., 2024) and its gradient with respect to the policy parameters. The optimizer then updates the policy weights.

The separation between vLLM (generation) and MegatronLM (training) is a standard design in RL training systems: generation benefits from inference-optimized serving infrastructure (continuous batching, KV-cache management), while training benefits from training-optimized infrastructure (distributed data parallelism, mixed-precision training, gradient accumulation). The weight synchronization pipeline bridges these two worlds.

Weight Synchronization Pipeline: After each RL step, the updated policy weights must be transferred from MegatronLM to the vLLM rollout engine so that the next batch of rollouts is generated from the latest policy. The paper does not provide detailed profiling of the weight synchronization time, but Table 1 includes $T_{\text{prepare}}$ as a separate stage (1.6–2.1 seconds per step), which covers both weight synchronization and rollout backend preparation. This is a small fraction of total step time (approximately 1–2% in the 8B experiments), indicating that weight synchronization is not a bottleneck at this scale.

The weight synchronization procedure must handle two model types when online draft adaptation is enabled: the policy model weights and the draft head weights. Both are transferred from the learner to the rollout engine. The transfer is likely implemented via shared memory or RDMA (remote direct memory access) over NVLink/NVSwitch, given that the experiments run on GB200 NVL72 nodes with fifth-generation NVLink interconnect.

A subtle requirement is that the weight transfer must be atomic with respect to the rollout generation. If the vLLM engine receives partial weight updates (e.g., some layers updated, others not), the generated trajectories would be drawn from an inconsistent policy—a mixture of old and new weights—which would break the on-policy guarantee. The system must ensure that all weights are updated before any new rollouts begin.

Caching Pathway for Online Draft Training: When online draft adaptation is enabled (Section 3.3), the system reuses computations from the MegatronLM forward pass to train the draft head. The mechanism works as follows:

During the log-probability recomputation forward pass ( $T_{\text{logprob}}$ ), MegatronLM computes hidden states at each transformer layer for every token position in the rollout trajectories. These hidden states are the internal representations that encode the policy model's understanding of the context at each position.
These hidden states, along with the policy's log-probabilities at each position, are cached and routed to the draft training module. The routing goes through a .detach() operation (PyTorch's mechanism for severing gradient flow), which creates a copy of the hidden states that does not propagate gradients back to the policy model's parameters.
The draft training module runs the EAGLE-3 draft head on the cached hidden states, computes the draft training loss $\mathcal{L}_{\text{draft}}$ (cross-entropy between draft predictions and the ground-truth next tokens from the rollout trajectories), and updates the draft head parameters via gradient descent.
The updated draft head weights are then synchronized to the vLLM engine alongside the updated policy weights in the next weight synchronization step.

The .detach() operation is crucial: without it, the draft loss gradient would flow back through the hidden states into the policy model, effectively turning the draft training loss into an auxiliary objective for the policy. This would change the optimization semantics—the policy would be trained not only to maximize the GRPO objective but also to produce hidden states that make the draft's job easier. While this might improve acceptance length, it would violate the "verifier-exact" guarantee because the policy's parameters would be influenced by a non-RL objective. The detached pathway ensures that the draft head and the policy model are trained independently, even though they share the same hidden states as input.

Telemetry and Observability: The paper mentions "stage-level telemetry" as a system requirement (Section 1, contributions bullet 1). Throughout the experiments, the system tracks per-stage timings (Table 1), generation latency per step (Figure 2a), acceptance length (Tables 2–5), and validation accuracy (Figure 2b). These metrics are integrated into the RL training loop's logging infrastructure, allowing operators to monitor the health of the speculative decoding pipeline alongside standard RL metrics. If acceptance length drops unexpectedly (indicating draft-policy misalignment), or if weight synchronization time spikes (indicating a networking bottleneck), operators can detect and diagnose the issue without stopping training.

Composition with Synchronous and Asynchronous RL

The paper studies speculative decoding under both synchronous and asynchronous RL execution to characterize how speculation interacts with policy lag.

Synchronous RL: In synchronous mode, each RL step is executed sequentially: generate rollouts → recompute log-probabilities → compute advantages → update policy → synchronize weights → repeat. There is no overlap between stages. This mode makes the generation bottleneck fully exposed: $T_{\text{gen}}$ contributes directly to $T_{\text{step}}$ , and any reduction in $T_{\text{gen}}$ translates directly to a reduction in $T_{\text{step}}$ (subject to Amdahl's law, since the other stages are unchanged). The main experiments (Tables 1–5, Figure 2) use synchronous RL to isolate the effect of speculative decoding on generation latency without the confounding factor of pipeline overlap.

In synchronous mode, the speedup from speculative decoding is bounded by the Amdahl's law expression with $R_{\text{gen}}$ measured from the autoregressive baseline (approximately 0.65–0.72 from Table 1). The observed end-to-end speedups of 1.35× (RL-Think) and 1.41× (RL-Zero) reflect this bound: even with generation speedups of 1.5× and 1.8× respectively, the non-generation stages (30–35% of step time) dilute the overall gain.

Asynchronous RL: In asynchronous mode, generation and training are overlapped across multiple nodes. The paper evaluates "policy lag 1" on RL-Think in a 16-node non-colocated configuration, with 12 nodes dedicated to generation and 4 nodes to training. In this configuration, the rollout engine generates trajectories continuously on the 12 generation nodes, while the training nodes consume previously generated trajectories, compute log-probabilities, and update the policy. The policy used for generation at time $t$ is the policy from time $t - \text{lag}$ —in this case, lag 1 means the generation nodes use the policy from one training step ago.

The key effect of asynchronous execution is that much of generation time is hidden behind training time. If generation and training are perfectly overlapped (training takes at least as long as generation), the effective step time is determined by the slower of the two stages, and generation contributes zero to the critical path. In practice, overlap is imperfect: some generation time is exposed (the training nodes must wait for rollouts to be ready before they can compute log-probabilities), and this exposed portion is what speculative decoding can accelerate.

The paper reports that in the async RL-Think configuration, speculative decoding reduces the exposed generation time (the portion on the critical path) from 10.4 seconds to 0.6 seconds per step, and lowers effective step time from 75.0 seconds to 60.5 seconds—an end-to-end speedup of 1.24×. The generation-side speedup is larger (the 10.4 → 0.6 reduction represents a ~17× speedup of the exposed portion), but because the exposed portion was already small relative to total step time (10.4 out of 75.0 seconds, or 14%), the end-to-end gain is modest.

This result illustrates a general principle: speculative decoding and asynchronous execution are complementary but partially overlapping mechanisms. Asynchronous execution reduces $R_{\text{gen}}$ on the critical path by hiding generation behind training, which shrinks the fraction of step time that speculation can accelerate. But they are not redundant: even in a well-overlapped async pipeline, some generation time remains exposed (due to load imbalance, startup transients, or generation nodes falling behind training nodes), and speculation can eliminate this residual exposure. The paper's characterization is that "the two mechanisms are complementary: speculation makes each rollout cheaper, while async overlap hides remaining generation cost."

Importantly, the paper confirms that the learning trajectory under async execution with speculative decoding remains similar to the autoregressive async baseline—validation accuracy curves overlap, just as they do in the synchronous case. This confirms that the verifier-exact guarantee holds regardless of whether the RL loop is synchronous or asynchronous.

Simulator Methodology for Deployment-Scale Projections

The experiments in Section 3 are conducted at 8B scale on 32 GPUs (8 GB200 NVL72 nodes with 4 GPUs each). Section 4 uses a "high-fidelity performance simulator" to project how speculative decoding speedups extrapolate to deployment-scale regimes: models up to 235B parameters (Qwen3-235B-A22B), GPU counts up to 2048 GB200s, policy lags up to 8, and rollout batch sizes of 4096. The paper emphasizes that these projections should be interpreted as "opportunity envelopes" with emphasis on trends rather than absolute values.

Simulator Requirements: The paper states that a faithful simulation framework must satisfy three requirements:

Accurate device-level and system-level performance characteristics: The simulator must model GPU compute units (FLOP rates, tensor core utilization), memory hierarchies (HBM bandwidth, capacity, caching behavior), and interconnects (NVLink bandwidth, latency, topology). This enables it to estimate per-layer latencies for transformer forward passes, which depend on matrix multiplication dimensions, attention sequence lengths, and communication patterns in tensor-parallel and pipeline-parallel configurations.
A broad spectrum of model sharding strategies: At deployment scale, models are partitioned across many GPUs using combinations of tensor parallelism (splitting individual layers across GPUs), pipeline parallelism (splitting layer sequences across GPUs), and data parallelism (replicating the model across GPU groups, each processing a subset of the batch). The simulator must evaluate these strategies and their interactions—including operator overlap (overlapping communication with computation) and power-aware optimizations—to find the configuration that minimizes per-token latency for a given model size and GPU count.
Long-tailed response length distributions characteristic of RL rollout workloads: Unlike inference serving, where prompts are typically short and the model generates relatively brief responses, RL rollouts for reasoning tasks produce highly variable-length outputs. The paper observes this directly in Figure 2a, where RL-Zero baseline latency rises sharply as the policy learns to produce longer chain-of-thought traces. The simulator uses a "dynamic traffic generator" that estimates rollout batch sizes at each step based on a given response length distribution, capturing the effect of long-tailed latencies on overall throughput. This is important because speculative decoding's benefit depends on the decode-to-prefill ratio: if responses are short, the prefill phase (processing the prompt, which speculation does not accelerate) dominates, reducing the effective speedup. Long reasoning traces are favorable for speculation because they are decode-heavy.

Simulator Architecture: The paper describes the simulator as "a proprietary GPU performance simulator that incorporates detailed models of GPU compute units, memory hierarchies, and interconnects" and "leverages a kernel-aware analytical framework to evaluate state-of-the-art model partitioning strategies." In practical terms, this means the simulator does not run actual GPU kernels but analytically estimates their execution time based on:

Compute time: modeled as $\text{FLOPs} / \text{effective FLOP rate}$ , where the effective FLOP rate depends on the matrix dimensions (which determine tensor core utilization) and the precision (FP8 in the projections).
Memory time: modeled as $\text{bytes transferred} / \text{effective bandwidth}$ , where the effective bandwidth depends on whether data resides in HBM or must be moved over NVLink.
Communication time: modeled as $\text{bytes communicated} / \text{interconnect bandwidth}$ , accounting for topology (e.g., all-reduce ring vs. tree algorithms) and overlap with computation.

The "kernel-aware" aspect means the simulator accounts for kernel launch overheads, occupancy limits (how many thread blocks can run concurrently), and the fact that some operations (e.g., attention softmax) are memory-bound while others (e.g., large matrix multiplies) are compute-bound. The "dynamic traffic generator" samples response lengths from a distribution estimated from empirical RL training data, constructing per-step batch compositions that reflect realistic load imbalance.

What the Simulator Produces: The simulator outputs two types of speedup estimates:

Rollout speedup (Figure 3a, Figure 4): The factor by which speculative decoding reduces the latency of the generation stage alone, accounting for draft model overhead, verification cost, and the effect of batching and sharding on per-token latency. This is the speedup that would be measured if generation were the only stage.
End-to-end speedup (Figure 3b): The factor by which speculative decoding reduces total RL step time, accounting for the non-generation stages ( $T_{\text{data}}$ , $T_{\text{prepare}}$ , $T_{\text{logprob}}$ , $T_{\text{train}}$ ) that speculation does not accelerate. This incorporates the Amdahl's law dilution: the gap between the rollout speedup and the end-to-end speedup is determined by the generation share $R_{\text{gen}}$ .

The heatmap in Figure 3 shows both speedups as a function of draft length $k$ (on the y-axis) and acceptance length $\alpha$ (on the x-axis) for Qwen3-235B-A22B on 512 GPUs. The gray cells in the upper-left triangle mark infeasible configurations where acceptance length exceeds $k+1$ (since you cannot accept more tokens than are proposed plus the one bonus token). The key patterns are:

At $k=7$ , acceptance length 5 yields a rollout speedup of 4.07× but an end-to-end speedup of only 1.96×—the non-generation stages halve the gain.
At $k=3$ , acceptance length 3 yields a rollout speedup of 2.72× and an end-to-end speedup of 1.70×—a comparable end-to-end operating point with far lower speculative overhead.
The peak rollout speedup in the heatmap is 6.49× (at $k=7$ , acceptance 8, the maximum feasible configuration), but the corresponding end-to-end speedup is only 2.22×, confirming that non-generation stages form a hard ceiling that no amount of rollout-side optimization can exceed.

Sensitivity to Deployment Scale and Policy Lag (Figure 4): The simulator sweeps GPU counts (32, 128, 512, 2048) and maximum policy lags (0, 2, 4, 8) to characterize how deployment configuration affects speculative decoding speedup. The results reveal scale-dependent behavior:

For Qwen3-235B-A22B (Figure 4a): Rollout speedup is sensitive to both GPU count and policy lag. At 32 GPUs with zero lag, speedup is approximately 1.9×; at lag 8, it drops to roughly 1.3×—a 32% reduction. At 512 GPUs with zero lag, speedup is approximately 3.4×; at lag 8, it remains around 3.0×—a much smaller relative drop. The 2048-GPU configuration shows a non-monotonic pattern: at zero lag, speedup is ~3.0× (lower than 512 GPUs), but at lag 2 it recovers to ~3.5× (the highest in the plot). The paper attributes this to a tension between batch-per-instance effects (more GPUs means smaller local batches, which reduces utilization but creates more room for speculative acceleration to fill idle cycles) and sharding overheads (spreading the batch too thin forces suboptimal tensor/pipeline parallelism configurations).
For Qwen3-8B (Figure 4b): Rollout speedup is remarkably stable across all configurations, clustering within a narrow 2.8–3.2× band regardless of GPU count or policy lag. The paper explains this as a consequence of model scale: the 8B model occupies only 8 GPUs per instance in a 2048-GPU deployment, allowing many parallel instances, each with a reasonable batch size. The batch is never spread so thin that sharding becomes suboptimal, and the per-token latency is low enough that even substantial policy lag does not meaningfully change the generation share on the critical path.

The paper's key deployment-scale projection is: "At the most favorable simulated operating point (Qwen3-235B-A22B, 2048 GPUs, lag 2), rollout speedup reaches ~3.5×; combined with the high generation share characteristic of frontier-scale models, this translates to a projected ~2.5× end-to-end training speedup." This is the headline scaling claim: speculative decoding's benefit grows with model size because larger models have higher $R_{\text{gen}}$ (longer per-token latencies and longer reasoning traces) and because they are deployed at larger GPU counts where the batch-per-instance shrinks, creating more idle cycles for speculation to fill.

Limitations of the Simulator: The paper is explicit that the simulator provides "opportunity envelopes" rather than exact predictions. The projections do not account for several real-world factors: variable network congestion, GPU thermal throttling, transient load imbalance from straggler responses (the dynamic traffic generator captures steady-state distributions but not bursty behavior), and the overhead of the weight synchronization pipeline at large scale (which may become non-negligible when model size reaches hundreds of billions of parameters). The projections should therefore be understood as establishing trends (speculation becomes more beneficial at scale, policy lag degrades speedup less at larger GPU counts, and the optimal draft length remains small) rather than as precise performance guarantees.

Summary of Design Choices and Their Justifications

EAGLE-3 as the default drafting path over native MTP heads: chosen as "the harder case" because it requires training and maintaining an external draft model, which exercises the full system integration (weight synchronization, online adaptation, gradient-detached training). Findings carry over to the simpler MTP case.
DAPO prompt-aligned draft initialization over generic chat initialization: because acceptance length depends on distribution match between draft and policy, and in-domain training data (policy outputs on the exact RL training prompts) provides the closest possible alignment at initialization time.
Draft length $k=3$ as the default over longer drafts: because empirical sweep (Table 4) shows that the per-speculation-step cost grows faster with $k$ than the acceptance length, making $k=3 the optimal cost-per-accepted-token operating point. Longer drafts are counterproductive, especially in harder RL regimes.
Offline drafting (no online adaptation) as the default when DAPO initialization is available: because online adaptation provides negligible additional benefit when the draft is already well-aligned (Table 5), and avoiding it reduces computational overhead and system complexity.
Gradient-detached hidden-state pathway for online draft training: to ensure that draft training does not alter the policy gradient signal, preserving the verifier-exact guarantee. Without this, the policy would be implicitly trained to optimize for draft acceptance, changing the optimization objective.
Two-fold validation via latency and accuracy tracking (Figure 2): generation latency and validation accuracy are monitored throughout training to confirm that speculative decoding provides sustained speedup (latency remains below baseline) without altering the optimization trajectory (accuracy curves overlap). This dual verification is necessary because acceptance length alone does not guarantee either property—high acceptance can coexist with poor speedup if draft overhead is high, and low acceptance can theoretically drift the distribution if the rejection sampling implementation has bugs.
Simulator-based deployment projections over empirical scaling experiments: because running 235B-scale RL training at multiple GPU counts and policy lags would be prohibitively expensive. The simulator provides trend-level guidance for practitioners planning frontier-scale deployments without requiring them to reproduce the experiments at scale.

4. Key Insights and Innovations

Innovation 1: Speculative Decoding as a "Lossless" Primitive That Decouples Throughput from Distributional Fidelity in RL Systems

The paper's most distinctive conceptual contribution is reframing speculative decoding not as an inference optimization trick, but as a systems primitive that occupies a unique point on the throughput-effectiveness Pareto frontier—one that no prior RL acceleration method could reach. Before this work, the field understood RL rollout acceleration as an inherent trade-off: you could go faster by changing the training dynamics (async execution, off-policy replay, low-precision rollouts), but each change perturbed the sampling distribution and risked degrading the learning signal. The dominant assumption was that these perturbations were acceptable engineering compromises—frontier model reports routinely combine several of them (DeepSeek-AI, 2024; Qwen Team, 2025; Meta GenAI, 2025).

This paper demonstrates that this assumption is incomplete. By integrating speculative decoding into the RL loop, it shows that throughput can be improved without touching effectiveness at all. The rejection sampling procedure guarantees that every accepted token follows the target policy's exact distribution, making the rollouts statistically indistinguishable from autoregressive generation. The RL gradient is therefore unchanged—not approximately, not with bounded error, but exactly. This is not an incremental refinement of existing methods. It shifts the design space itself: rather than asking "how much distribution mismatch can we tolerate for how much speedup?", the question becomes "how much speedup can we extract before the generation share ceiling binds?", which is an entirely different optimization problem.

What makes this framing intellectually distinctive is that it reclassifies speculative decoding from an inference-serving technique (where it has been studied extensively; Leviathan et al., 2023; Chen et al., 2023; Cai et al., 2024; Li et al., 2024) into a training-system primitive with different requirements and constraints. In serving, the model is frozen, the draft is trained once, and there is no concept of a moving policy. In RL training, the moving policy creates entirely new challenges—draft-policy alignment drift, weight synchronization, log-probability correctness—that the paper identifies and solves at the systems level (Section 2.3, Figure 1). The insight is not that speculative decoding is new, but that its lossless property makes it the only known mechanism for accelerating on-policy RL rollouts without altering the optimization objective, and that deploying it in this context requires solving integration problems that the inference-serving literature never addressed.

The evidence for this claim is the validation accuracy curves in Figure 2b, which show the EAGLE-3 and autoregressive curves overlapping almost perfectly throughout training on RL-Think (both rising from ~0.60 to ~0.70) and RL-Zero (both rising from ~0.03 to ~0.33). This is the empirical confirmation of the lossless guarantee: speculative decoding changes when the model arrives at a given accuracy, not what accuracy it reaches.

Innovation 2: The Amdahl's Law Decomposition as a Unifying Diagnostic Framework for Rollout-Side Acceleration

The paper introduces a simple but powerful analytical tool: the Amdahl's law bound $S_{\text{step}} \leq 1 / (R_{\text{gen}}/\alpha + (1 - R_{\text{gen}}))$ (Section 2.2). This equation is not itself novel—Amdahl's law is a standard tool in computer architecture—but its application as a diagnostic framework for RL rollout acceleration is an intellectual contribution that unifies and explains essentially all of the paper's empirical findings through a single lens.

Prior work on RL rollout efficiency (Noukhovitch et al., 2025; Piché et al., 2025; Li et al., 2025a; Zheng et al., 2025a) focused on specific mechanisms and reported speedups in specific configurations, but lacked a common framework for understanding why a given mechanism delivers a given speedup and where the ceiling lies. The Amdahl's law decomposition provides exactly this: it separates the problem into two independently measurable quantities—the generation share $R_{\text{gen}}$ (how much of the RL step is generation) and the acceptance length $\alpha$ (how effectively the draft predicts the policy)—and shows that both must be optimized for speculation to yield meaningful end-to-end gains.

The diagnostic power of this framework is demonstrated by how cleanly it explains the paper's counterintuitive findings:

Why draft initialization matters so much (Table 3): DAPO initialization raises $\alpha$ from 2.88 to 3.32 on RL-Zero, directly increasing the denominator's $R_{\text{gen}}/\alpha$ term.
Why longer draft lengths can be slower than autoregressive (Table 4): Increasing $k$ raises $\alpha$ sub-linearly while increasing per-step cost, making the effective $\alpha$ per unit cost lower at $k=7$ than at $k=3$ —the bound captures this when you account for the fact that per-step cost is not constant.
Why async execution reduces speculation's benefit (Section 3.3): Async overlap reduces the effective $R_{\text{gen}}$ on the critical path from ~0.65–0.72 to ~0.14 (10.4s out of 75.0s), shrinking the optimizable fraction.
Why larger models benefit more (Section 4.3): Larger models have higher $R_{\text{gen}}$ due to longer per-token latencies and longer reasoning traces, giving speculation more room to operate.

This framework is not a theoretical contribution in the formal sense—it derives no new theorems about convergence or optimality. But it is a practical diagnostic tool that changes how practitioners should think about rollout acceleration. Rather than asking "is speculative decoding faster?", the framework tells them to measure $R_{\text{gen}}$ first—if generation is only 30% of step time, no amount of speculation will yield more than a modest speedup. Rather than treating acceptance length as the sole figure of merit, the framework shows that per-step overhead can erase high acceptance entirely, explaining why $n$ -gram drafting achieves non-trivial acceptance (2.47 on RL-Zero) yet is slower than autoregressive (Table 2). This reframes the engineering problem from "maximize acceptance" to "maximize net tokens per unit cost," which is a more useful objective.

Innovation 3: Scale-Dependent Characterization of Speculative Decoding's Operating Regime via Deployment-Scale Simulation

The paper's third distinctive contribution is its characterization of how speculative decoding's benefit varies with deployment scale, model size, and policy lag—not through expensive empirical scaling experiments (which would be prohibitively costly at 235B scale), but through a high-fidelity performance simulator that models GPU compute, memory, interconnect, and long-tailed response length distributions (Section 4).

This is more than a "we projected our results to larger scale" section. It surfaces a set of non-obvious, scale-dependent trends that would not be predictable from the 8B experimental data alone:

The non-monotonic relationship between GPU count and speedup for large models (Figure 4a): At zero policy lag, 512 GPUs (3.4× rollout speedup) outperforms 2048 GPUs (3.0×), but at lag 2, the ordering reverses (2048 GPUs reaches 3.5×, the highest in the plot). The paper attributes this to a tension between batch-per-instance effects (more GPUs → smaller local batches → more idle cycles for speculation to fill) and sharding overhead (too-thin batches force suboptimal parallelism configurations). This is a subtlety that would be invisible at small scale and that practitioners planning frontier deployments need to know.
The scale-dependence of sensitivity to policy lag (Figure 4a vs. 4b): The 235B model's rollout speedup degrades by ~32% as lag increases from 0 to 8 at 32 GPUs, but remains relatively stable at 512+ GPUs. The 8B model is essentially insensitive to lag at all GPU counts. This difference arises because larger models occupy more GPUs per instance, making batch composition and load balance more sensitive to the exact deployment configuration.
The draft-length sweet spot is narrow and scale-invariant (Figure 3): The heatmap shows that $k=3$ with acceptance length 3 yields comparable end-to-end speedup (1.70×) to $k=7$ with acceptance length 5 (1.96×), despite the latter requiring 67% higher acceptance. This confirms that the finding from Table 4—longer drafts are rarely worth it—generalizes to deployment scale and is not an artifact of the 8B experimental setup.
The generation share ceiling (Figure 3b vs. 3a): A rollout speedup of 6.49× (the peak in the heatmap) translates to only 2.22× end-to-end. The non-generation stages form a hard ceiling that no amount of rollout-side optimization can exceed. This quantifies the "opportunity envelope" the paper emphasizes: even at the most favorable simulated operating point, end-to-end speedup is bounded at ~2.5×.

The intellectual contribution here is not the simulator itself (which is proprietary and not described in sufficient detail to replicate), but the characterization methodology and the trend-level findings it enables. Prior work on speculative decoding for RL (FastGRPO by Zhang et al., 2025; ReSpec by Chen et al., 2025) focused on algorithmic optimizations at a single scale. This paper adds the dimension of deployment-scale analysis, showing that the same speculative decoding configuration can yield dramatically different speedups depending on GPU count, model size, and policy lag—and that these interactions are not intuitively predictable. The simulator-based approach, while not directly reproducible by external researchers, provides a template for how deployment-scale projection can complement empirical experimentation in systems ML research.

Innovation 4: Online Draft Adaptation as Insurance, Not Improvement—A Negative Result with Positive Implications

The paper's ablation on online draft adaptation (Table 5) is a negative result: when the draft is well-initialized (DAPO), online updating provides essentially zero additional speedup (1.77× → 1.78× on RL-Zero; 1.53× → 1.52× on RL-Think). The benefit appears only for the weaker UltraChat initialization (1.51× → 1.63× on RL-Zero; 1.19× → 1.26× on RL-Think), and even there, the gains are modest (+6–8%).

This finding is intellectually significant because it reframes online draft adaptation from a general optimization strategy (the natural assumption: keep the draft aligned with the moving policy to maintain acceptance) to a robustness mechanism (useful only when initial alignment is poor). This has direct implications for system design: if you can train the draft on in-domain policy outputs before RL training begins, you can skip the entire online adaptation infrastructure—the gradient-detached caching pathway, the draft weight synchronization, the additional training compute—and get essentially the same speedup with a simpler system. Online adaptation becomes a fallback for scenarios where in-domain draft initialization is impossible (e.g., when the policy's output distribution at RL training time is genuinely unknown before training starts).

This finding also challenges a potential assumption from concurrent work. FastGRPO (Zhang et al., 2025) emphasizes online draft learning as a core mechanism, and ReSpec (Chen et al., 2025) studies reward-weighted drafter adaptation. This paper's result suggests that, at least for the mathematical reasoning workloads and model scales studied, offline draft initialization quality dominates online adaptation benefits. This is not a contradiction—FastGRPO and ReSpec may study regimes where initial alignment is harder or where the policy distribution shifts more dramatically during training—but it establishes a boundary condition: online adaptation is not universally beneficial, and its value depends on the initial draft-policy alignment gap.

The insight generalizes beyond speculative decoding: in any RL training system where an auxiliary model (draft, verifier, value function) must track a moving policy, the decision of whether to update the auxiliary model online should be driven by the initial alignment quality, not by a blanket assumption that online updating is always better. The paper provides empirical evidence that, for well-initialized drafts, the policy's distribution shift during RL training is small enough that a frozen draft remains effective. This is a practical finding that reduces system complexity for a common deployment scenario.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses DAPO-Math-17K (Yu et al., 2025) for RL training—a dataset of 17,000 mathematical reasoning problems designed for GRPO-based post-training. For validation, the paper reports accuracy on AIME-2024, a separate competition-level math benchmark. The split is standard: DAPO-Math-17K provides training prompts for policy optimization; AIME-2024 provides held-out evaluation to track generalization. No explicit mention is made of a train/validation split within DAPO-Math-17K, since RL training uses the full dataset for rollout generation and the validation metric is computed on an entirely separate benchmark.
Base model(s). Experiments use two variants from the Qwen3 family (Qwen Team, 2025): Qwen3-8B (the instruct-tuned, reasoning-capable checkpoint) for the RL-Think setting, and Qwen3-8B-Base (the pretrained base model without instruction tuning) for the RL-Zero setting. Both are 8B-parameter dense models. The authors state they chose this family as representative of contemporary open-weight models at a scale where comprehensive RL training experiments remain feasible on 32 GPUs while still exhibiting the generation bottleneck characteristic of larger models. The RL-Think setting continues training from a model that already produces structured reasoning traces, while RL-Zero starts from a model with no reasoning fine-tuning, creating two distinct difficulty regimes for the draft model.
Metrics. The paper tracks three categories of metrics. For throughput: generation latency per RL step (seconds, measured as wall-clock time for the $T_{\text{gen}}$ stage), end-to-end RL step time (seconds, summing all five stages in Table 1), and generation speedup (ratio of autoregressive to speculative generation latency). For speculation quality: mean acceptance length $\alpha$ (average number of tokens accepted per speculative forward pass, computed from the rejection sampling procedure). For training quality: validation accuracy on AIME-2024 (fraction of problems answered correctly, graded via string matching on final answers). The paper carefully separates these: acceptance length measures draft quality, generation latency measures systems throughput, and validation accuracy measures whether the optimization trajectory is preserved.
Baselines. The primary baseline is autoregressive decoding—standard token-by-token generation without speculation, using the same vLLM backend and model configuration. This is the "unaccelerated" reference point for all speedup calculations. A secondary baseline is n-gram drafting, a model-free speculative approach that uses n-gram statistics (rather than a learned draft model) to propose candidate tokens. The n-gram baseline serves as a sanity check: it demonstrates that positive acceptance length alone (n-gram achieves 2.47 on RL-Zero, 2.05 on RL-Think) is insufficient for speedup if the draft's per-step overhead exceeds the savings from accepted tokens. The paper does not compare against other RL acceleration methods (async execution, off-policy replay, low-precision rollouts) in head-to-head experiments—the analysis in Section 2.1 positions speculative decoding as complementary to these methods rather than competing with them.
Generation budget / compute accounting. The paper measures throughput in wall-clock time per RL step, not in abstract FLOP counts. This is the appropriate metric for a systems paper: the goal is to reduce training wall-clock time, and FLOP accounting would obscure real-world overheads like weight synchronization, kernel launch latency, and load imbalance. Within the speculative decoding mechanism, the cost model assumes that one target model forward pass (verification) costs approximately the same as one autoregressive forward pass—a reasonable approximation because modern transformer implementations process multiple positions efficiently in parallel. The draft model's forward passes are assumed negligible relative to the target model's, which is justified for EAGLE-3 (the draft is a small auxiliary head operating on cached hidden states, not a full language model). The paper acknowledges that this cost model is approximate and that realized speedup is always below the Amdahl's law bound due to draft overhead, prefill time, and batching effects.
Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional ML sense—there is no hyperparameter tuning on held-out data, no multiple random seeds, and no error bars on the main results. This is a systems measurement paper, not a statistical ML paper: the primary results are throughput measurements (seconds per step) under controlled configurations, where variance comes from hardware fluctuations (GPU clock speeds, network congestion, thermal throttling) rather than from sampling different data splits. The validation accuracy curves (Figure 2b) serve as the "correctness check": if speculative decoding altered the optimization trajectory, the accuracy curves would diverge, and this would be visible without statistical testing because the curves either track or they don't. The paper's approach is to run complete RL training runs (500+ steps for RL-Think, 1000+ steps for RL-Zero) and report per-step latency and validation accuracy throughout, providing a time-series view rather than point estimates. For the simulator projections (Section 4), there is no statistical protocol—the simulator is deterministic given a configuration, producing a single speedup estimate per operating point. The paper acknowledges this limitation by characterizing results as "opportunity envelopes" with "emphasis on trends rather than absolute values."

Main Quantitative Results

The experiments are organized around two RL training regimes (RL-Think and RL-Zero) that create different difficulty profiles for the draft model, with a unified set of measurements across both. The paper's core empirical claims are: (1) speculative decoding reduces generation latency by 1.5–1.8× at 8B scale, (2) end-to-end step speedup is limited to 1.35–1.41× by non-generation stages, (3) validation accuracy is indistinguishable from autoregressive baselines, verifying the lossless guarantee, and (4) the realized speedup depends critically on draft initialization quality, draft length, and generation share.

I organize the results by first establishing the baseline timing breakdown (where does time go?), then presenting the headline generation speedups, then validating that training quality is preserved, and finally examining the operational factors that determine realized speedup.

The Generation Bottleneck: Where Time Goes in an RL Step

Before measuring speculative decoding's benefit, the paper must first establish that generation is indeed the bottleneck worth accelerating. Table 1 provides this baseline, reporting the per-stage wall-clock time breakdown for both RL-Think and RL-Zero under autoregressive decoding and speculative decoding (EAGLE-3, $k=3$ , DAPO initialization, offline drafting).

In the autoregressive baselines, generation ( $T_{\text{gen}}$ ) is the single largest stage in both settings:

RL-Think: Generation consumes 133.6 seconds out of 185.3 seconds total per step—72.1% of step time. Log-probability recomputation ( $T_{\text{logprob}}$ ) takes 17.9s (9.7%), training ( $T_{\text{train}}$ ) takes 31.4s (16.9%), preparation takes 2.1s (1.1%), and data loading takes 0.3s (0.2%).
RL-Zero: Generation consumes 100.0 seconds out of 151.2 seconds total—66.1% of step time. Log-probability recomputation takes 17.8s (11.8%), training takes 31.3s (20.7%), preparation takes 1.9s (1.3%), and data loading takes 0.2s (0.1%).

Two observations are notable. First, training time ( $T_{\text{train}}$ ) is nearly identical across RL-Think and RL-Zero (31.4s vs. 31.3s), which is expected since both use the same 8B model architecture and the GRPO loss computation cost depends primarily on model size and sequence length, not on the policy's reasoning capability. Second, generation time is substantially higher for RL-Think (133.6s vs. 100.0s), reflecting that the instruct-tuned model produces longer reasoning traces than the base model—the RL-Think policy already generates chain-of-thought, while RL-Zero starts from short outputs and gradually learns to produce reasoning (visible in the latency ramp in Figure 2a). This means the generation share $R_{\text{gen}}$ is higher for RL-Think (0.72 vs. 0.66), which, by the Amdahl's law bound, gives speculative decoding more room to improve end-to-end step time—though this is partially offset by RL-Think's lower acceptance length (2.77 vs. 3.32), as discussed below.

With EAGLE-3 speculative decoding enabled, the per-stage breakdown shifts:

RL-Think: Generation drops from 133.6s to 87.0s—a 1.54× generation speedup. End-to-end step time drops from 185.3s to 137.4s—a 1.35× overall speedup. The non-generation stages are essentially unchanged: log-probability recomputation stays at ~18s, training stays at ~30s, and data/prepare times are unchanged. This confirms that speculation affects only $T_{\text{gen}}$ and leaves other stages untouched.
RL-Zero: Generation drops from 100.0s to 56.6s—a 1.77× generation speedup. End-to-end step time drops from 151.2s to 107.5s—a 1.41× overall speedup. Again, non-generation stages are unchanged.

The gap between generation speedup and end-to-end speedup is entirely explained by Amdahl's law. For RL-Think with $R_{\text{gen}} = 0.721$ and $\alpha = 2.77$ (from Table 2), the Amdahl bound is $1 / (0.721/2.77 + 0.279) = 1 / (0.260 + 0.279) = 1 / 0.539 = 1.86\times$ . The observed 1.54× generation speedup and 1.35× end-to-end speedup are both below this bound, as expected given draft overhead and batching effects not captured by the simplified model. For RL-Zero with $R_{\text{gen}} = 0.661$ and $\alpha = 3.32$ , the bound is $1 / (0.661/3.32 + 0.339) = 1 / (0.199 + 0.339) = 1 / 0.538 = 1.86\times$ as well—similar ceiling despite different $R_{\text{gen}}$ and $\alpha$ values because they compensate: RL-Think has higher $R_{\text{gen}}$ but lower $\alpha$ , RL-Zero has lower $R_{\text{gen}}$ but higher $\alpha$ .

Generation Latency and Speedup: EAGLE-3 vs. Autoregressive vs. N-gram

Table 2 provides the head-to-head comparison of three generation methods at matched RL configurations. The key metrics are acceptance length (mean tokens per speculative step), generation latency per RL step (seconds), and generation speedup relative to the autoregressive baseline.

For RL-Zero:

Autoregressive: 100.0 seconds per step (baseline, speedup 1.0×).
N-gram drafting: Acceptance length 2.47, but generation latency increases to 140.2 seconds—a 0.7× speedup, meaning n-gram is 40% slower than autoregressive despite achieving non-trivial acceptance. This is the paper's clearest demonstration that acceptance length alone is not a sufficient metric: the n-gram draft's per-step overhead (building and querying n-gram statistics) consumes more time than the accepted tokens save.
EAGLE-3: Acceptance length 3.32, generation latency drops to 56.6 seconds—a 1.8× speedup over autoregressive and a 2.5× advantage over n-gram.

For RL-Think:

Autoregressive: 133.6 seconds per step.
N-gram drafting: Acceptance length 2.05, generation latency balloons to 262.9 seconds—a 0.5× speedup, meaning n-gram is twice as slow as autoregressive. The degradation is even worse than on RL-Zero, likely because RL-Think's longer, more structured reasoning traces are harder for simple n-gram statistics to predict, reducing acceptance while the overhead remains constant.
EAGLE-3: Acceptance length 2.77, generation latency drops to 87.0 seconds—a 1.5× speedup.

The acceptance length difference between RL-Zero (3.32) and RL-Think (2.77) is notable. The paper doesn't provide a detailed per-token analysis, but the likely explanation is that RL-Think's policy already produces coherent chain-of-thought reasoning, which is more structured and predictable than RL-Zero's evolving outputs—but also more diverse in its specific token choices, making exact token-level prediction harder. Higher-level semantic predictability does not necessarily translate to higher token-level acceptance.

Figure 2a extends these point measurements to the full training run, plotting generation latency at each training step for both autoregressive and EAGLE-3 decoding. The per-step curves reveal several dynamics invisible in the mean statistics:

RL-Think (left panel, 600 steps): Autoregressive latency fluctuates between approximately 120s and 160s per step, with a gentle downward trend in the later steps (possibly as the policy converges and stops increasing response length). EAGLE-3 latency tracks this fluctuation in a compressed band between roughly 75s and 105s. The speedup is sustained throughout training: mean 1.54×, maximum 1.81× (at some steps the draft aligns particularly well with the policy, yielding higher acceptance). There is no evidence of draft staleness causing speedup degradation—the EAGLE-3 curve does not drift upward relative to the autoregressive curve over time, confirming that the DAPO-initialized draft remains aligned throughout training without online updates.
RL-Zero (right panel, 1000 steps): This panel reveals a striking dynamic: the autoregressive baseline latency starts at approximately 40s per step (steps 0–50) and then rises sharply to approximately 100–120s over the first ~100–150 steps. This is the policy learning to produce chain-of-thought reasoning: initially, the base model outputs short, direct answers (sometimes just a number), but as GRPO rewards correct reasoning, the policy learns to generate longer, more structured traces, and generation latency tracks this increase. The EAGLE-3 curve follows the same trajectory but compressed: starting around 25s, rising to roughly 50–60s. The speedup is mean 1.79×, maximum 2.85×—the maximum occurring early in training when outputs are short and highly predictable, making the draft's job easier. Crucially, even as the policy's output distribution shifts dramatically (from short answers to long reasoning traces), the speedup persists—the EAGLE-3 curve remains consistently below the autoregressive curve with no convergence. This is evidence that the DAPO initialization, trained on the policy's initial outputs, provides sufficient alignment even as the policy evolves, and that the draft's ability to predict tokens degrades gracefully rather than catastrophically.

The paper reports the mean and max speedup values in the figure caption rather than in the main text. For RL-Think: mean 1.54×, max 1.81×. For RL-Zero: mean 1.79×, max 2.85×. The large gap between mean and max on RL-Zero reflects the early-training regime where outputs are short and predictable—a transient bonus that the mean statistic averages away.

Validation Accuracy: Verifying the Lossless Guarantee

Figure 2b plots validation accuracy on AIME-2024 against training step for both autoregressive and EAGLE-3 decoding. This is the critical experiment for the paper's central claim that speculative decoding preserves "verifier-exact training semantics"—that the optimization trajectory is indistinguishable from autoregressive training.

RL-Think (left panel, 600 steps): Both curves start at approximately 0.58–0.60 accuracy at step 0 (the instruct-tuned model already has some mathematical reasoning capability) and rise to approximately 0.68–0.70 by step 600. The EAGLE-3 and autoregressive curves overlap almost perfectly—there is no systematic divergence, no period where one curve leads or lags the other, and the final accuracy is indistinguishable (~0.70 for both). The training dynamics are preserved.
RL-Zero (right panel, 1000 steps): Both curves start at approximately 0.03–0.05 (near-random performance, as expected from a base model without reasoning training) and rise to approximately 0.32–0.33 by step 1000. The learning curve is nearly identical under both decoding modes: the same initial flat period (steps 0–200, where the model is learning basic reasoning structure), the same takeoff phase (steps 200–600, where accuracy rises rapidly), and the same plateau (steps 600–1000, where gains slow). The final accuracy is ~0.33 for both curves.

The paper does not report confidence intervals or statistical tests on these curves—they are presented as raw per-step accuracy measurements on the 30-question AIME-2024 set. With 30 questions, the standard error on a proportion near 0.33 is approximately $\sqrt{0.33 \times 0.67 / 30} \approx 0.086$ (8.6 percentage points), so individual step measurements are noisy. The relevant observation is not point-by-point equality but the absence of systematic divergence: the curves track each other throughout training, with no sustained gap in either direction.

A subtle point: the lossless guarantee from speculative decoding's rejection sampling procedure ensures that the distribution of individual tokens matches the target policy. However, the RL training signal depends on complete trajectories and their associated rewards. In principle, if the speculative decoding implementation had a bug—for example, if the rejection sampling used an incorrect residual distribution, or if the draft model's probabilities leaked into the log-probability recomputation—the trajectory-level distribution could diverge even if token-level distributions were correct. The validation accuracy experiment rules out such bugs empirically: if trajectories were systematically different under speculative decoding (e.g., systematically shorter, systematically different reasoning structures), the RL optimizer would receive a different gradient signal, and the learning curve would diverge. The overlapping curves are therefore strong evidence that the integration is correct.

Draft Initialization Quality (Table 3)

Table 3 compares two draft initializations at fixed draft length $k=3$ under offline drafting (no online updates during training):

UltraChat initialization: The EAGLE-3 draft is trained on general chat data (UltraChat and Magpie; Ding et al., 2023; Xu et al., 2025), which contains diverse conversational examples but no mathematical reasoning traces. On RL-Zero, this achieves acceptance length 2.88 and generation speedup 1.51×. On RL-Think, acceptance length drops to 2.40 and speedup to 1.19×—barely faster than autoregressive and substantially worse than the DAPO initialization.
DAPO initialization: The EAGLE-3 draft is trained on responses generated by the policy model on the DAPO-Math-17K training prompts—the exact prompts used for RL training. On RL-Zero, acceptance length rises to 3.32 and speedup to 1.77× (+17% over UltraChat). On RL-Think, acceptance length rises to 2.77 and speedup to 1.53× (+29% over UltraChat).

The key finding is that in-domain initialization matters more for RL-Think than RL-Zero (29% vs. 17% speedup improvement). The paper's interpretation: RL-Think starts from an instruct-tuned model that already produces structured chain-of-thought reasoning, making its outputs more domain-specific and less similar to general chat than RL-Zero's initial outputs (which start as short, simple responses before evolving). A chat-domain draft is therefore a worse match for RL-Think than for RL-Zero, making DAPO initialization proportionally more valuable.

A practical implication: if you are doing RL post-training from an already-capable model (RL-Think style), investing in in-domain draft initialization is essential—a generic draft will barely beat autoregressive decoding. If you are doing RL from a base model (RL-Zero style), the generic draft is less bad because the base model's initial outputs are less domain-specific, but in-domain initialization still provides a meaningful gain.

Draft Length (Table 4)

Table 4 sweeps draft length $k \in \{3, 5, 7\}$ with DAPO initialization under offline drafting. The central finding: $k=3$ is optimal, and larger drafts are counterproductive. Acceptance length increases with $k$ in all cases, but realized speedup decreases.

For RL-Zero:

$k=3$ : acceptance 3.32, speedup 1.77×
$k=5$ : acceptance 4.35, speedup 1.44×
$k=7$ : acceptance 5.06, speedup 1.21×

Acceptance grows steadily (3.32 → 4.35 → 5.06), but the improvement is sub-linear in $k$ : each additional draft token is accepted with lower probability than the previous one, because the draft's predictions become less accurate at longer horizons. Meanwhile, the per-speculation-step cost grows with $k$ (more draft forward passes, larger verification pass). The net tokens-per-unit-cost peaks at $k=3$ .

For RL-Think, the effect is more dramatic:

$k=3$ : acceptance 2.77, speedup 1.53×
$k=5$ : acceptance 3.23, speedup 0.84× (slower than autoregressive)
$k=7$ : acceptance 3.48, speedup 0.71× (substantially slower than autoregressive)

At $k=5$ and $k=7$ , speculative decoding with EAGLE-3 is a net negative on RL-Think: the draft overhead exceeds the savings from accepted tokens, and the generation stage takes longer than it would with simple autoregressive decoding. This is the paper's most striking negative result on draft length, and it carries an important practical lesson: the draft-length sweet spot is narrower than the acceptance-length curve would suggest, and testing longer drafts (which might seem intuitively beneficial because they increase acceptance) can make generation slower, not faster.

The asymmetry between RL-Zero and RL-Think is notable. On RL-Zero, $k=7$ still yields a positive speedup (1.21×), while on RL-Think it is a substantial slowdown (0.71×). The paper does not fully explain this asymmetry, but the likely mechanism is that RL-Think's outputs are longer and more structured, making the EAGLE-3 draft's predictions decay faster with horizon—the acceptance of the 5th and 7th draft tokens is proportionally lower on RL-Think than on RL-Zero, making the cost-per-accepted-token penalty from longer drafts more severe.

Online Draft Adaptation (Table 5)

Table 5 studies online draft adaptation—periodically retraining the draft on rollouts generated by the current policy—for both UltraChat and DAPO initializations at $k=3$ . The finding: online adaptation provides limited benefit when the draft is well-initialized, and modest benefit when it is poorly initialized.

For DAPO initialization:

RL-Zero: offline 1.77× → online 1.78× (negligible change; acceptance 3.32 → 3.29)
RL-Think: offline 1.53× → online 1.52× (negligible change; acceptance 2.77 → 2.74)

The DAPO-initialized draft is already well-aligned with the policy at the start of training, and the policy's distribution shift during RL training is small enough that the draft remains effective without updates. The tiny fluctuations in acceptance length (3.32 → 3.29, 2.77 → 2.74) are likely within measurement noise and confirm that online adaptation neither helps nor hurts in this regime.

For UltraChat initialization:

RL-Zero: offline 1.51× → online 1.63× (+8%; acceptance 2.88 → 3.04)
RL-Think: offline 1.19× → online 1.26× (+6%; acceptance 2.40 → 2.55)

Online adaptation partially compensates for the poor initialization by incrementally aligning the draft to the policy's evolving distribution. The gains are modest—8% and 6% speedup improvement—but they demonstrate that online adaptation works as intended: it raises acceptance length and realized speedup. However, even with online adaptation, the UltraChat draft never catches up to the DAPO-initialized offline draft (1.63× vs. 1.77× on RL-Zero; 1.26× vs. 1.53× on RL-Think), suggesting that initial alignment quality has a persistent effect that online updates cannot fully overcome within the training horizon studied.

The practical takeaway is that online adaptation serves as insurance against distribution mismatch, not as a general improvement strategy. If you can train the draft on in-domain policy outputs before RL training begins, online adaptation can be omitted entirely, simplifying the system (no gradient-detached caching pathway, no draft weight synchronization, no additional training compute). Online adaptation is worth enabling only when in-domain initialization is impossible—and even then, the gains are modest.

Interaction with Asynchronous Execution (Section 3.3, final paragraph)

The paper reports a single experiment on the interaction between speculative decoding and asynchronous RL: RL-Think with policy lag 1 in a 16-node non-colocated configuration (12 generation nodes, 4 training nodes). In this setting, much of generation is overlapped with training, and only the exposed portion (the time training nodes wait for rollouts) is on the critical path.

The paper reports that speculative decoding reduces exposed generation time from 10.4 seconds to 0.6 seconds per step—a ~17× reduction of the exposed portion. However, effective step time drops only from 75.0s to 60.5s, yielding an end-to-end speedup of 1.24×. In the autoregressive async baseline, total step time is 75.0s with generation overlap; with speculation, it falls to 60.5s. The learning trajectories remain similar under both decoding modes.

This result demonstrates that speculative decoding and asynchronous execution compose without interference, but the speedup is smaller than in synchronous RL because async overlap already reduces the effective generation share on the critical path. In synchronous RL-Think, generation was 72.1% of step time (133.6s out of 185.3s), and speculation provided a 1.35× end-to-end speedup. In async RL-Think, the exposed generation is only 10.4s out of 75.0s (13.9%), and speculation provides 1.24×. The 17× reduction of the exposed portion looks dramatic but translates to only a modest end-to-end gain because the exposed portion was already small.

The paper explicitly frames this as evidence of complementarity: "the two mechanisms are complementary: speculation makes each rollout cheaper, while async overlap hides remaining generation cost." The async experiment confirms that speculation does not interfere with async overlap (e.g., by changing the timing such that training nodes wait differently) and that the verifier-exact guarantee holds in async mode as well.

Deployment-Scale Simulator Projections (Section 4, Figures 3–4)

The simulator projections extend the empirical findings to deployment-scale regimes not directly tested. Figure 3 (heatmap) reports rollout and end-to-end speedup for Qwen3-235B-A22B on 512 GPUs in synchronous RL as a function of draft length ( $k \in \{1, 3, 5, 7\}$ ) and acceptance length ( $\alpha \in \{1, 1.5, 2, 2.5, 3, 4, 5, 6, 7, 8\}$ ). Gray cells mark infeasible configurations where acceptance exceeds $k+1$ .

Key patterns from Figure 3a (rollout speedup):

At $k=1$ (standard speculative decoding with single-token proposals), acceptance length 1.5 yields 1.45× rollout speedup.
At $k=3$ , acceptance length 3 yields 2.72×; acceptance length 4 yields 3.62×.
At $k=7$ , acceptance length 5 yields 4.07×; acceptance length 8 yields 6.49× (the maximum in the heatmap).
The gradient is steepest in the lower-left region: going from acceptance 1 to 2 at $k=3$ more than doubles rollout speedup (1.36× → 2.72×), while going from 5 to 8 at $k=7$ yields a smaller relative gain (4.07× → 6.49×).

Key patterns from Figure 3b (end-to-end speedup):

The non-generation stages dramatically compress the range. The 6.49× rollout peak translates to only 2.22× end-to-end.
At $k=3$ , acceptance length 3 yields 1.70× end-to-end; acceptance length 4 yields 1.89×.
At $k=7$ , acceptance length 5 yields 1.96× end-to-end; acceptance length 8 yields 2.22×.
Comparing $k=3$ , acceptance 3 (1.70× end-to-end) with $k=7$ , acceptance 5 (1.96× end-to-end): the longer draft yields only 15% better end-to-end speedup despite requiring 67% higher acceptance and substantially more draft overhead. This confirms the draft-length finding from Table 4 generalizes to deployment scale.

Figure 4 sweeps GPU count and policy lag for two model sizes at fixed draft length 5 and acceptance length 4. For Qwen3-235B-A22B (Figure 4a):

At 32 GPUs, lag 0: rollout speedup ~1.9×; lag 8: ~1.3× (32% degradation).
At 512 GPUs, lag 0: ~3.4×; lag 8: ~3.0× (12% degradation).
At 2048 GPUs, lag 0: ~3.0×; lag 2: ~3.5×; lag 8: ~3.0×.
The non-monotonic behavior at 2048 GPUs (lower at lag 0 than 512 GPUs, recovering at lag 2) is attributed to sharding inefficiency at zero lag (batch spread too thin) being compensated by pipeline concurrency at lag 2.
Larger GPU counts are more robust to policy lag: the speedup degradation with increasing lag is smaller at 512+ GPUs than at 32–128 GPUs.

For Qwen3-8B (Figure 4b):

All configurations (32–2048 GPUs, lag 0–8) cluster within 2.8–3.2× rollout speedup.
No meaningful sensitivity to GPU count or policy lag.
The paper attributes this to the 8B model's small per-instance GPU footprint (8 GPUs per instance in 2048-GPU deployment, allowing many parallel instances with reasonable batch sizes), which avoids the sharding and load-balance issues that affect the 235B model.

The paper's headline projection: "At the most favorable simulated operating point (Qwen3-235B-A22B, 2048 GPUs, lag 2), rollout speedup reaches ~3.5×; combined with the high generation share characteristic of frontier-scale models, this translates to a projected ~2.5× end-to-end training speedup."

Ablation Studies and Robustness Checks

The paper's ablation structure is integrated into the main experimental narrative (Tables 3–5) rather than presented as a separate section. Each ablation studies a single operational decision and its effect on acceptance length and speedup.

Draft initialization (Table 3): UltraChat (general chat) vs. DAPO (in-domain math reasoning) at fixed $k=3$ . DAPO initialization consistently outperforms UltraChat: +17% speedup on RL-Zero (1.51× → 1.77×), +29% on RL-Think (1.19× → 1.53×). The relative benefit is larger for RL-Think, where the policy's outputs are more domain-specific. This ablation establishes that in-domain draft initialization is a first-order determinant of speedup—a generic draft barely beats autoregressive on RL-Think.

Draft length (Table 4): $k \in \{3, 5, 7\}$ with DAPO initialization. $k=3$ is consistently optimal. On RL-Zero, speedup degrades from 1.77× to 1.44× to 1.21× as $k$ increases. On RL-Think, the degradation is severe: 1.53× → 0.84× → 0.71×, with $k=5$ and $k=7$ being slower than autoregressive. This ablation demonstrates that acceptance length is not a sufficient metric for draft quality— $k=7$ achieves the highest acceptance (5.06 on RL-Zero, 3.48 on RL-Think) but the worst speedup, because the per-step overhead grows faster than the acceptance gain.

Online draft adaptation (Table 5): Offline vs. online drafting for both UltraChat and DAPO initializations at $k=3$ . For DAPO, online adaptation provides negligible change (1.77× → 1.78× on RL-Zero; 1.53× → 1.52× on RL-Think). For UltraChat, online adaptation provides modest gains (1.51× → 1.63× on RL-Zero; 1.19× → 1.26× on RL-Think). This ablation establishes that online adaptation is an insurance mechanism, not a general improvement strategy—it helps when initial alignment is poor but adds no value when the draft is already well-initialized. This is a negative result with positive implications: systems can omit the complexity of online draft training when in-domain initialization is feasible.

N-gram drafting baseline (Table 2): N-gram speculative decoding achieves non-trivial acceptance (2.47 on RL-Zero, 2.05 on RL-Think) but is slower than autoregressive in both settings (0.7× and 0.5× speedup, respectively). This ablation serves as a sanity check: it demonstrates that the speedup from speculative decoding is not automatic—the draft model must be both accurate (high acceptance) and efficient (low per-step overhead relative to the savings). The n-gram draft fails on the second criterion: its per-step overhead (building and querying n-gram statistics on the fly) exceeds the time saved by accepted tokens.

Asynchronous execution interaction (Section 3.3): Speculative decoding in async RL-Think with policy lag 1 yields 1.24× end-to-end speedup, compared to 1.35× in synchronous mode. The exposed generation time is reduced from 10.4s to 0.6s (17× reduction of the exposed portion), but this translates to a modest end-to-end gain because the exposed portion was already small (13.9% of step time). This ablation demonstrates that speculation and async execution compose without interference, but the benefit of speculation shrinks as async overlap hides more generation time.

The paper does not include several ablations that would strengthen the analysis:

No DAPO vs. other in-domain data: The paper only compares DAPO initialization to UltraChat (general chat). It does not test intermediate levels of domain alignment (e.g., math data not from the exact RL prompts, or a mixture of math and chat data), which would characterize how sensitive speedup is to the degree of domain match.
No EAGLE-3 vs. other draft architectures: The paper focuses exclusively on EAGLE-3, stating that the MTP (native multi-token prediction) path is supported and findings carry over, but provides no empirical comparison. A head-to-head between EAGLE-3 and MTP at matched draft sizes would clarify whether the external draft model's overhead matters.
No draft model size ablation: The EAGLE-3 draft's capacity (number of parameters, number of layers) is not varied. A smaller draft might have lower per-step overhead but lower acceptance; a larger draft might have higher acceptance but higher overhead. The optimal draft size for a given model and task is unexplored.
No acceptance length decomposition by response position: The paper reports mean acceptance length but does not decompose it by position within the response (beginning vs. middle vs. end). Acceptance likely varies systematically—early tokens may be more predictable (fixed prompt, common opening phrases) while later tokens may be less predictable (diverse reasoning paths). Understanding this decomposition would inform whether draft length should vary with response position.
No training-time overhead measurement for online adaptation: When online adaptation is enabled, the additional computation (draft loss forward/backward pass, draft weight update) consumes time that could otherwise be used for policy training. The paper does not report whether this overhead is negligible or significant, which matters for the end-to-end speedup accounting.

Critical Assessment

The paper makes several central claims, and the experiments support them with varying degrees of completeness. I examine each claim in turn, identifying what the experiments actually demonstrate versus what the claims assert, and where the evidence is thinner than it appears.

Claim: "Speculative decoding improves rollout throughput by 1.8× at 8B scale." The experiments in Table 2 and Figure 2a directly support this claim for the specific configuration tested: Qwen3-8B, GRPO on DAPO-Math-17K, EAGLE-3 drafting with DAPO initialization and $k=3$ , running on 32 GB200 GPUs. The 1.8× figure comes from RL-Zero generation latency (100.0s → 56.6s). The RL-Think generation speedup is 1.5×—still meaningful, but smaller. The claim holds, but with the qualification that it varies substantially across RL regimes (1.5× vs. 1.8×) and depends on draft initialization (dropping to 1.2× with a generic chat draft on RL-Think). The paper's abstract emphasizes 1.8×, which is the best-case among the tested configurations, not the typical or worst-case.

Claim: "Speculative decoding yields up to 1.4× end-to-end step speedup." Table 1 confirms this: 1.35× on RL-Think, 1.41× on RL-Zero. The speedup is bounded by the Amdahl's law ceiling (non-generation stages consume 28–34% of step time), which the paper correctly identifies as the limiting factor. The claim holds, but the phrasing "up to" is important—1.41× is the ceiling at 8B scale with this specific stage breakdown. If $T_{\text{logprob}}$ or $T_{\text{train}}$ were proportionally larger (as they might be for models with different architectures or training configurations), the end-to-end speedup would be lower.

Claim: "Validation accuracy is indistinguishable from autoregressive baselines." Figure 2b provides strong evidence for this claim at 8B scale over the full training runs (600 and 1000 steps). The EAGLE-3 and autoregressive curves overlap closely in both RL-Think and RL-Zero. However, the evidence has limitations: (1) AIME-2024 has only 30 questions, so per-step accuracy measurements are noisy (standard error ~8.6 percentage points near 0.33 accuracy). Systematic divergence smaller than the noise floor would be invisible. (2) The training runs are relatively short (600–1000 steps)—if speculative decoding introduced subtle distributional biases that accumulate over longer training, this experiment would not detect them. (3) The validation metric is final-answer accuracy, not a more sensitive measure of distribution match (e.g., KL divergence between trajectory distributions, or per-token probability calibration). The claim of "indistinguishable" is empirically supported for the observed training horizon, but the experiment is not sensitive enough to guarantee identical optimization trajectories at finer granularity.

Claim: "Simulator projections show 2.5× end-to-end training speedup at 235B scale." This claim is based on Figure 3b and Figure 4a, but the evidence is qualitative, not quantitative. The simulator is proprietary, not described in sufficient detail for external validation, and the paper explicitly states that results should be interpreted as "opportunity envelopes" with "emphasis on trends rather than absolute values." The 2.5× figure is the paper's best estimate for the most favorable operating point (2048 GPUs, lag 2), but the simulator's accuracy at this scale is unknown—there is no validation against empirical measurements at 235B scale because those measurements are prohibitively expensive. The projections are useful as trend guidance (speculation becomes more beneficial at scale, optimal draft length remains small) but the specific 2.5× number should be treated as an approximate estimate, not a validated prediction.

Claim: "Speculative decoding composes with asynchronous execution as a complementary mechanism." The single async experiment (RL-Think, lag 1, 16 nodes) supports this claim in a limited regime. The 1.24× end-to-end speedup demonstrates that speculation and async overlap do not interfere, and that speculation provides some benefit even when generation is largely hidden. However, the experiment tests only one lag value (1) and one node configuration (12 generation, 4 training). The simulator projections in Figure 4 sweep a wider range of lags and GPU counts, showing that the benefit varies substantially—at 32 GPUs, lag 8 reduces rollout speedup to ~1.3× for the 235B model, while at 512 GPUs it remains near ~3.0×. The paper's abstract and conclusion phrase the composition claim broadly, but the evidence shows that the benefit of speculation under async execution is configuration-dependent: in some regimes, async overlap shrinks the critical-path generation share so much that speculation's benefit becomes marginal.

Genuine weaknesses in the experimental design:

Single model family, single scale. All empirical results are on Qwen3-8B. The paper provides no evidence that the findings generalize to other model architectures (e.g., Mixture-of-Experts models, where per-token latency and generation share might differ), other model sizes at the empirical level (e.g., 1B or 70B), or other RL algorithms (e.g., PPO with a critic model, which would change the $T_{\text{train}}$ breakdown). The simulator partially addresses the scale limitation, but only for the Qwen3 family and only in projection, not in measurement.
Single task domain (mathematical reasoning). The experiments use DAPO-Math-17K for training and AIME-2024 for validation—both mathematical reasoning benchmarks. The acceptance length and generation share depend on the task: mathematical reasoning produces long, structured chain-of-thought traces that may be more predictable (and thus more amenable to speculative decoding) than other task types. Agentic RL with multi-turn tool use (cited in Section 1 as an emerging bottleneck) may have very different generation patterns—shorter individual model calls, more variable outputs, different prefill-to-decode ratios—that could substantially change speculation's benefit.
No sensitivity analysis on GPU hardware. All experiments use GB200 NVL72 nodes with fifth-generation NVLink. The per-token latencies, weight synchronization time, and the relative cost of draft vs. target forward passes depend on hardware characteristics (HBM bandwidth, interconnect speed, tensor core throughput). The paper does not test on other hardware configurations (e.g., H100 nodes with slower interconnect), limiting the generalizability of the specific speedup numbers.
Limited training horizon. RL-Think runs for 600 steps, RL-Zero for 1000 steps. These are sufficient to demonstrate the speedup and verify that accuracy curves don't diverge, but they may not capture long-term effects. If the draft model's alignment with the policy degrades gradually over many thousands of steps (even the DAPO-initialized draft showed a tiny acceptance change from 3.32 to 3.29 with online adaptation in Table 5—suggesting some drift), the speedup might slowly degrade over longer training runs. The paper cannot rule this out with the current data.
Missing baseline: larger draft models. The paper's cost model assumes draft forward passes are negligible compared to target forward passes. This is reasonable for EAGLE-3 (a small auxiliary head), but the paper does not empirically verify this assumption by, for example, profiling the draft model's per-step time separately from the target verification time. If the draft overhead were non-negligible, it would reduce the effective $\alpha$ per unit cost, and the speedup would be lower than reported. The n-gram baseline demonstrates that overhead can dominate acceptance entirely, but there is no analogous check for EAGLE-3's overhead relative to its acceptance.

Experiments that would strengthen the paper:

Head-to-head comparison with MTP drafting. The paper states that findings carry over to the native MTP path (Gloeckle et al., 2024), where the model's built-in auxiliary heads serve as the draft. A direct EAGLE-3 vs. MTP comparison at matched draft capacity would validate this claim and help practitioners choose between the two approaches. MTP might have lower overhead (no separate draft model to load and run) but potentially lower acceptance (the auxiliary heads are trained concurrently with the base model, not specifically optimized for the RL task distribution).
Ablation on draft training data quantity. The DAPO initialization uses responses on the full 17K training prompts. How many prompts are needed to train an effective draft? If 1K prompts suffice, the draft initialization cost is much lower; if 17K is near the minimum, practitioners need to budget for substantial draft training before RL begins.
Measurement of token-level acceptance variation. Reporting acceptance length as a function of position within the response (first 10% of tokens, middle, last 10%) would reveal whether the draft is uniformly effective or whether acceptance collapses in certain regions (e.g., late in long reasoning traces where the policy explores diverse solution paths). This would inform adaptive strategies where draft length varies with position.
A longer training run with periodic acceptance monitoring. Extending RL-Zero to 5000+ steps and tracking acceptance length throughout would test whether the DAPO-initialized draft eventually goes stale (acceptance drifts downward) and whether online adaptation can recover from this staleness at longer horizons.

Conditional nature of the claims: All empirical speedup claims are conditional on the specific configuration: Qwen3-8B, GRPO on mathematical reasoning, EAGLE-3 drafting with in-domain initialization and $k=3$ , running on GB200 NVL72 hardware. The paper is appropriately explicit about these conditions, but the abstract and introduction necessarily generalize ("speculative decoding improves rollout throughput by 1.8×"), which risks overstatement. The evidence supports the claim that speculative decoding can provide meaningful throughput improvements in RL post-training, but the magnitude and even the sign of the effect (see $k=7$ on RL-Think: 0.71×, slower than autoregressive) depends on configuration choices that the paper helpfully characterizes. The paper's primary contribution is not the specific speedup numbers but the characterization of when and why speculation helps—the draft initialization, draft length, and generation share analysis—and this contribution is well-supported.

6. Limitations and Trade-offs

6.1 Single Model Family, Single Scale: No Empirical Evidence of Generalization Beyond Qwen3-8B

The assumption or constraint. All empirical experiments in Section 3 use exclusively the Qwen3-8B model family (Qwen Team, 2025)—specifically Qwen3-8B for RL-Think and Qwen3-8B-Base for RL-Zero. The paper does not present any empirical measurements on other model architectures (e.g., Mixture-of-Experts models, where per-token latency characteristics differ substantially), other model sizes at the measured level (e.g., 1B or 70B), or other model families (e.g., Llama, DeepSeek). The simulator projections in Section 4 extend the analysis to Qwen3-235B-A22B and to deployment scales up to 2048 GPUs, but these are projections, not measurements, and the paper explicitly characterizes them as "opportunity envelopes" with "emphasis on trends rather than absolute values" (Section 4 introduction). There is no empirical validation of the simulator's accuracy at any scale beyond 8B.

The consequence. A practitioner cannot determine from this paper alone whether the 1.5–1.8× generation speedup observed on Qwen3-8B will transfer to their specific model. Several properties of the Qwen3-8B architecture could affect speculative decoding's benefit in ways that would not generalize:

Architecture-specific acceptance rates. Different model families have different token prediction patterns, different degrees of output determinism, and different typical sequence lengths for the same task. A model that produces more predictable reasoning traces (higher token-level repetition, more formulaic structure) will yield higher acceptance lengths and greater speedup; a model with more diverse token choices will yield lower acceptance and potentially negative speedup (recall $k=7$ on RL-Think was 0.71×, slower than autoregressive, in Table 4). The paper provides no evidence about where Qwen3-8B falls on this spectrum relative to other models.
Architecture-specific generation share $R_{\text{gen}}$ . The Amdahl's law bound (Section 2.2) shows that end-to-end speedup is fundamentally limited by the fraction of step time spent on generation. Qwen3-8B has $R_{\text{gen}} \approx 0.66-0.72$ (Table 1). A model with faster per-token inference (e.g., a model using Grouped-Query Attention with fewer KV heads, or a model with more aggressive KV-cache compression) would have lower $R_{\text{gen}}$ and correspondingly lower end-to-end speedup from speculation. Conversely, a Mixture-of-Experts model with sparse activation would have higher per-token latency (more parameters to route through), potentially raising $R_{\text{gen}}$ but also changing the relative cost of the draft model's forward passes.
Draft architecture compatibility. The EAGLE-3 draft head attaches to the target model's hidden states. Its effectiveness depends on how informative those hidden states are for multi-token prediction, which varies with the target model's architecture (number of layers, hidden dimension, attention pattern). A target model with very different internal representations might require a different draft architecture or training recipe to achieve comparable acceptance.

The simulator partially addresses model scale, but at the cost of empirical fidelity—the 2.5× end-to-end projection for Qwen3-235B-A22B is a modeled estimate, not a measurement, and its accuracy is unknown.

What evidence exists in the paper. The paper contains no empirical evidence of generalization to other model families, architectures, or sizes. The simulator projections (Section 4, Figures 3–4) model Qwen3-235B-A22B and Qwen3-8B at larger GPU counts, but these are within the same model family and are not empirically validated. The paper does not cite prior work showing that EAGLE-3 speculative decoding generalizes across model families in RL training contexts (no such prior work exists, since this is the first RL-training integration of speculative decoding at this level of system completeness).

Mitigation status. The paper acknowledges the single-model limitation indirectly by framing the simulator projections as "trends rather than absolute values" (Section 4), but does not explicitly flag the lack of cross-model empirical evidence as a limitation. The claim that Qwen3-8B is "representative of the capabilities of many contemporary LLMs" (Section 4, implied by the model choice) is asserted rather than demonstrated. The paper does not suggest future work on cross-model validation. A practitioner deploying this method on a non-Qwen model would need to conduct their own empirical validation from scratch, as the paper provides no guidance on how speedup estimates transfer across architectures.

6.2 Single Task Domain: Only Mathematical Reasoning, No Evidence from Agentic or Multi-Turn Workloads

The assumption or constraint. All experiments use mathematical reasoning as the RL post-training task: DAPO-Math-17K (Yu et al., 2025) for training and AIME-2024 for validation. The paper's own motivation section (Section 1) explicitly identifies agentic RL as an emerging domain where rollout generation is especially expensive:

"The same issue is emerging in agentic RL, where long-horizon tasks require many repeated model calls across tool-use, retrieval, and web-interaction steps and further amplify the cost of every decoded token."

However, the paper provides no empirical results on agentic workloads, multi-turn dialogue, code generation, or any task domain other than single-turn mathematical reasoning. This is a significant gap because speculative decoding's effectiveness depends on properties of the generated text that vary across domains:

Response length and structure. Mathematical reasoning produces long, structured chain-of-thought traces with high token-level predictability (mathematical notation, repeated patterns like "Step 1:", "Therefore,"). This is favorable for speculative decoding: long sequences make the decode phase dominant (high $R_{\text{gen}}$ ), and structured patterns increase acceptance length. Agentic trajectories may involve many short model calls (e.g., "search for X," "click on Y") interleaved with tool outputs, producing a different prefill-to-decode ratio and potentially lower token-level predictability.
Prefill-to-decode ratio. Speculative decoding accelerates only the decode phase, not prefill (Section 2.2). In mathematical reasoning, prompts are relatively short (problem statements) and responses are long (multi-step reasoning), making the workload decode-heavy—ideal for speculation. In agentic RL, each model call may have a long context (accumulated tool outputs, conversation history) and a short generation (a single action or brief reasoning step), making the workload prefill-heavy. Speculative decoding would be less beneficial in this regime because a smaller fraction of generation time is spent on the decode phase.
Token-level predictability. The paper shows that even within mathematical reasoning, acceptance length varies substantially: 3.32 on RL-Zero vs. 2.77 on RL-Think (Table 2). This variation exists within a single task domain and is driven by differences in output structure (RL-Think produces more diverse chain-of-thought than RL-Zero's early outputs). The variation across task domains is likely much larger. For example, creative writing or open-ended dialogue may have much lower token-level predictability than mathematical reasoning, yielding lower acceptance and potentially negative speedup.

The consequence. The 1.5–1.8× generation speedup reported in the paper cannot be assumed to transfer to agentic RL, code generation, or other task domains that the paper's introduction identifies as important use cases. A practitioner deploying speculative decoding for agentic RL training cannot estimate their expected speedup from this paper's data, because the key determinants—acceptance length, decode-to-prefill ratio, and generation share—are all task-dependent and unmeasured outside mathematical reasoning.

What evidence exists in the paper. None. There are no experiments on non-math tasks. The paper does not even measure task-specific properties that would help practitioners extrapolate (e.g., token-level entropy of the policy's output distribution, which correlates with acceptance length). The simulator projections in Section 4 also assume mathematical reasoning workloads (they model the Qwen3 family on RL post-training without specifying a different task distribution).

Mitigation status. The paper does not acknowledge this as a limitation. The abstract, introduction, and conclusion present speculative decoding as a general RL rollout acceleration primitive without qualifying that all empirical evidence comes from a single task domain. The introduction's mention of agentic RL serves only as motivation for why generation efficiency matters, not as a domain where the method has been tested. A reader might reasonably assume from the paper's framing that the method has been demonstrated on agentic workloads, when it has not. No future work on cross-domain validation is suggested.

6.3 Draft Initialization Requires Access to In-Domain Policy Outputs Before Training Begins

The assumption or constraint. The paper's headline speedups (1.8× generation on RL-Zero, 1.5× on RL-Think; Table 2) are achieved with DAPO-initialized drafts—EAGLE-3 draft models trained on responses generated by the policy model on the exact same training prompts (DAPO-Math-17K) used for RL training. This initialization procedure requires that, before RL training starts, the practitioner: (1) has access to the policy model's outputs on the training prompts, and (2) can afford the computational cost of generating these outputs and training the draft on them.

When this assumption is violated—when in-domain policy outputs are unavailable—the paper shows that speedup degrades substantially. With a generic chat-domain initialization (UltraChat), speedup drops from 1.77× to 1.51× on RL-Zero (−15%) and from 1.53× to 1.19× on RL-Think (−22%) (Table 3). On RL-Think, the chat-initialized draft is only marginally faster than autoregressive (1.19×), and with online adaptation enabled, it recovers only to 1.26× (Table 5)—still far below the DAPO-initialized offline speedup of 1.53×.

The consequence. The effectiveness of speculative decoding as a rollout acceleration primitive depends on a circular requirement: you need the policy's outputs on the training prompts to train an effective draft, but the whole point of RL training is to improve the policy. In the paper's setup, this circularity is resolved by using the initial policy (before any RL training) to generate the draft training data. This works because the initial policy's output distribution is close enough to the distribution during RL training that the draft remains effective without online updates (Table 5 shows negligible benefit from online adaptation when DAPO-initialized).

However, this resolution relies on two conditions that may not hold in general:

The initial policy must already produce reasonable outputs on the training prompts. In RL-Zero, the base model produces short, often incorrect answers (validation accuracy starts at ~0.03 in Figure 2b), but these outputs are apparently sufficient to train an effective draft (acceptance 3.32, speedup 1.77×). The paper does not analyze what properties of these outputs make them suitable for draft training, so a practitioner cannot determine whether their initial policy's outputs will suffice.
The policy's output distribution must not shift too far during RL training. The paper's finding that online adaptation is unnecessary for DAPO-initialized drafts (Table 5) suggests the distribution shift during 600–1000 steps of GRPO on mathematical reasoning is modest. For longer training runs, different RL algorithms (e.g., PPO with a KL penalty that more aggressively constrains policy movement), or tasks where the policy undergoes more dramatic capability improvements, the shift might be larger, and the DAPO-initialized draft might go stale. The paper provides no evidence about where this staleness threshold lies.

In practice, the draft initialization cost (generating policy outputs on all training prompts, then training the EAGLE-3 draft) adds to the total computational budget for RL post-training. The paper does not quantify this cost or amortize it into the reported speedups. For a practitioner, the upfront cost of draft initialization might be significant—potentially comparable to a non-trivial number of RL training steps—and the net speedup (including initialization cost) would be lower than the per-step speedups reported.

What evidence exists in the paper. Table 3 directly demonstrates the speedup gap between in-domain (DAPO) and generic (UltraChat) initialization. Table 5 shows that online adaptation only partially recovers from poor initialization (1.19× → 1.26× on RL-Think with UltraChat). The paper does not quantify the computational cost of DAPO initialization (how many GPU-hours to generate policy outputs and train the draft) and does not amortize this cost into the reported speedup figures. The paper does not analyze what properties of the initial policy's outputs determine draft training effectiveness.

Mitigation status. The paper partially acknowledges this limitation through its analysis of online draft adaptation (Section 3.3), which it characterizes as "insurance against distribution mismatch rather than a general improvement strategy." The finding that online adaptation helps weak initializations (Table 5) is presented as a partial mitigation, but the recovery is incomplete—the online-adapted UltraChat draft never reaches DAPO-initialized performance. The paper does not explicitly flag the circularity of requiring in-domain policy outputs for draft initialization, does not quantify the initialization cost, and does not suggest methods for reducing this cost (e.g., using a smaller subset of prompts for draft training, or using a different draft architecture that requires less data). The implicit recommendation—"use DAPO-style initialization if you can"—assumes a level of access to policy outputs that may not be practical in all RL post-training scenarios, especially those involving proprietary models, very large training sets, or tasks where generating initial outputs is itself expensive.

6.4 Simulator-Based Projections Are Unvalidated and the Simulator Is Proprietary

The assumption or constraint. Section 4 uses a "proprietary GPU performance simulator" to project speculative decoding speedups to deployment-scale regimes (Qwen3-235B-A22B, up to 2048 GPUs, policy lags up to 8). The simulator is described at a high level (Section 4.1): it "incorporates detailed models of GPU compute units, memory hierarchies, and interconnects," "leverages a kernel-aware analytical framework to evaluate state-of-the-art model partitioning strategies," and uses a "dynamic traffic generator" to estimate rollout batch sizes from response length distributions. However, the paper provides no validation of the simulator's accuracy—no comparison of simulated latencies against empirical measurements at scales where both are available (e.g., the 8B experiments on 32 GPUs, which could serve as a held-out validation point for the simulator). The simulator is proprietary and not described in sufficient detail for external researchers to reproduce or independently validate its projections.

The consequence. The paper's most ambitious claims—2.5× end-to-end training speedup at 235B scale (Section 4.3), rollout speedups exceeding 3× (Figure 4a), and the scale-dependent sensitivity patterns—rest entirely on an unvalidated simulator. If the simulator systematically overestimates speedup (e.g., by underestimating communication overhead at large GPU counts, by assuming optimistic batching efficiency, or by not modeling real-world sources of latency like GPU thermal throttling or network congestion), the deployment-scale projections could be substantially inflated. Conversely, if the simulator underestimates speedup, the paper undersells the method's potential. There is no way for a reader to assess which direction the error lies or its magnitude.

Specific reasons the simulator might be inaccurate at deployment scale:

Communication modeling at scale. At 2048 GPUs, all-reduce operations for tensor parallelism and pipeline parallelism involve complex communication patterns (hierarchical reductions across NVSwitch domains, potential congestion on inter-node links). The simulator's interconnect model may not capture these effects accurately, especially if it uses simplified bandwidth/latency abstractions rather than cycle-accurate network simulation.
Long-tailed response length distributions. The "dynamic traffic generator" estimates batch sizes from a response length distribution, but real RL rollouts exhibit bursty, correlated behavior (many prompts may produce long responses simultaneously after a policy update that encourages longer reasoning). The paper does not specify what response length distribution is used as input to the simulator or how it was estimated—it is unclear whether it comes from the 8B experiments (and is assumed to scale to 235B), from prior work on large-model reasoning, or from other sources.
FP8 precision assumptions. The simulator uses FP8 precision, which introduces quantization error not present in the paper's empirical experiments (which presumably use BF16 or FP16, though the paper doesn't specify the training precision). The accuracy of speculative decoding depends on exact token probability matching for rejection sampling; quantization could perturb these probabilities and affect acceptance length in ways the simulator does not model.
Startup transients and load imbalance. Section 3.3's async experiment shows that even with well-tuned overlap, some generation time remains exposed (10.4 seconds → 0.6 seconds with speculation). The simulator may not capture the full complexity of pipeline bubbles, straggler responses, and synchronization overheads that create this residual exposure.

What evidence exists in the paper. No validation of the simulator against empirical measurements is provided. The paper does not report simulated speedups for the 8B, 32-GPU configuration used in the experiments, which would serve as a basic sanity check (do the simulator's estimates match the measured 1.8× generation speedup?). The simulator's internal models (GPU compute, memory, interconnect) are not described in enough detail to assess their fidelity. The paper does not cite prior work that has validated this simulator on related workloads.

Mitigation status. The paper acknowledges the limitation explicitly—"The speedups reported in this section should be interpreted as opportunity envelopes, with emphasis on trends rather than absolute values" (Section 4)—which appropriately tempers the strength of the claims. However, this caveat appears once in the section introduction and is not repeated with each specific numerical claim (e.g., the 2.5× figure in the abstract and conclusion is presented without this qualification). The paper does not suggest future work on simulator validation or on empirical scaling experiments that would ground the projections. For a practitioner, the deployment-scale projections provide directional guidance (speculation becomes more beneficial at larger scale, optimal draft length remains small, policy lag sensitivity decreases at larger GPU counts) but not actionable quantitative predictions on which to base infrastructure planning decisions.

6.5 The Hardest Problems (Weakest Policies) See Reduced Benefit, and the Method Cannot Create Capability

The assumption or constraint. The paper's two RL regimes create different difficulty profiles for speculative decoding. RL-Think starts from a model that already produces structured reasoning (validation accuracy ~0.60, Figure 2b), while RL-Zero starts from a base model with essentially no reasoning capability (accuracy ~0.03). The speculative decoding speedup is higher on RL-Zero (1.8× generation, 1.41× end-to-end) than RL-Think (1.5× generation, 1.35× end-to-end) in the headline numbers (Tables 1–2). This might suggest that speculation helps more when the policy is weaker.

However, a closer look reveals the opposite pattern when draft quality is marginal. On RL-Think with $k=5$ and $k=7$ drafts, speculative decoding becomes slower than autoregressive (0.84× and 0.71× speedup, Table 4), while on RL-Zero with the same draft lengths, speedup remains positive (1.44× and 1.21×). The UltraChat-initialized draft is also worse on RL-Think (1.19×) than RL-Zero (1.51×) (Table 3). This suggests that RL-Think's outputs are harder to predict with a draft model than RL-Zero's outputs—despite RL-Think being the more capable policy. The paper's explanation: RL-Think's structured reasoning traces are more diverse in their specific token choices, making exact token-level prediction harder even though the high-level reasoning structure is more coherent.

The consequence. There is a non-obvious relationship between policy capability and speculative decoding benefit. A more capable policy (RL-Think, accuracy ~0.60–0.70) does not necessarily yield higher acceptance or speedup—in fact, the paper's data suggests the opposite: the weaker policy (RL-Zero, accuracy ~0.03–0.33) yields better speedup, likely because its outputs are less diverse at the token level (shorter, more formulaic, less creative reasoning). If this pattern generalizes, speculative decoding might be most beneficial early in RL training (when the policy is weak and outputs are simple/predictable) and less beneficial or even counterproductive later (when the policy has learned to generate diverse, sophisticated reasoning). This would limit speculation's value for long-running RL training campaigns that aim to push policy capability far beyond its starting point.

More fundamentally, speculative decoding does nothing to help the policy solve problems it cannot already solve. If the policy's pass@1 on a problem class is near zero (like RL-Zero at the start of training, accuracy ~0.03), no amount of speculative decoding changes that—the generation is just faster, not better. The paper's framing emphasizes that speculative decoding is a "lossless" acceleration primitive, meaning it preserves the policy's output distribution exactly. This is a strength (it doesn't perturb training semantics) but also a limitation: speculative decoding cannot improve the quality of the rollouts, only the speed at which they are generated. Other acceleration methods that do change the distribution (e.g., higher-temperature sampling, best-of-N selection with a verifier) might find correct solutions that the policy would miss, trading distributional fidelity for exploration—a trade-off that speculative decoding explicitly forgoes.

What evidence exists in the paper. Table 4 shows counterproductive speedup on RL-Think with longer drafts. Table 3 shows larger initialization sensitivity on RL-Think. Figure 2a shows that the RL-Zero autoregressive baseline latency rises sharply as the policy learns reasoning, suggesting outputs become longer and potentially less predictable over time—yet the EAGLE-3 speedup is sustained (mean 1.79× throughout). This is evidence that the DAPO-initialized draft remains effective even as the policy evolves, but the paper does not analyze whether the speedup would eventually degrade with enough training steps. The validation accuracy curves (Figure 2b) confirm that speculative decoding does not improve the policy's final capability—both curves end at the same accuracy.

Mitigation status. The paper does not explicitly address the relationship between policy capability and draft effectiveness. The finding that RL-Think is harder to accelerate than RL-Zero is presented as an empirical observation, not analyzed as a fundamental limitation. The paper's conclusion presents speculative decoding as broadly applicable, without qualifying that its benefit may shrink as the policy becomes more capable and its outputs become less token-level predictable. No experiments probe how acceptance length evolves over very long training runs (thousands of steps) or whether there is a policy capability threshold beyond which speculation provides negligible benefit. The limitation that speculation cannot improve rollout quality is inherent to the method and not something the paper attempts to mitigate—it is a deliberate design choice (lossless acceleration), but the paper does not discuss the opportunity cost of forgoing methods that trade distributional fidelity for better exploration.

6.6 Wall-Clock Latency vs. Throughput: Sequential Draft Dependencies and Batching Effects Are Minimally Analyzed

The assumption or constraint. The paper measures throughput in "generations per second" or "step time," which is appropriate for a training system where the goal is to maximize the rate of RL steps. However, speculative decoding introduces sequential dependencies that can affect latency in ways not fully captured by throughput metrics:

Draft-then-verify is inherently sequential within each request. For a single prompt, the draft model must propose $k$ tokens autoregressively before the target model can verify them. This is $k$ sequential draft forward passes followed by one target forward pass, compared to $k+1$ sequential target forward passes in autoregressive decoding. The latency reduction comes from replacing expensive target forward passes with cheap draft forward passes, but the sequential dependency remains—you cannot parallelize the draft proposals across the $k$ positions.
Batching effects interact with draft overhead. In a serving system like vLLM, multiple requests are batched together to maximize GPU utilization. The draft model's forward passes consume GPU compute that could otherwise be used for target model forward passes from other requests in the batch. If the draft model is not perfectly compute-bound (i.e., its forward passes don't fully utilize the GPU), it may reduce overall throughput by creating pipeline bubbles. Conversely, if the draft model is small enough that it fits in otherwise-idle compute units, its overhead may be negligible. The paper does not profile this interaction.
Prefill is not accelerated and may become a bottleneck. The paper notes that speculative decoding targets only the decode phase (Section 2.2). For prompts with long contexts (common in agentic RL, where tool outputs accumulate), prefill can be a significant fraction of generation time. Speculative decoding does nothing to reduce prefill latency, and if prefill and decode are batched together (as is typical in continuous batching serving systems), the decode-phase speedup may not translate to proportional end-to-end latency reduction because the prefill phase occupies GPU time that blocks decode.

The consequence. The measured speedups (1.5–1.8× generation, 1.35–1.41× end-to-end) are specific to the batching configuration, GPU count, and prompt/response characteristics of the experimental setup. A practitioner deploying speculative decoding in a different configuration—different batch size, different GPU count, different prefill-to-decode ratio, different draft model size relative to the target—may see substantially different speedups. The paper provides no profiling of how draft model overhead scales with batch size, no analysis of GPU utilization under speculative vs. autoregressive decoding, and no measurement of prefill time as a fraction of generation time.

The n-gram baseline (Table 2) provides a cautionary example: n-gram drafting achieves non-trivial acceptance (2.47 on RL-Zero) but is slower than autoregressive because its per-step overhead (building and querying n-gram statistics) dominates. The paper uses this to argue for EAGLE-3's efficiency, but does not provide analogous overhead profiling for EAGLE-3 itself. A reader cannot determine from the paper how close EAGLE-3 is to the overhead threshold where speculation becomes counterproductive—the n-gram result shows that threshold exists, but EAGLE-3's position relative to it is unmeasured.

What evidence exists in the paper. Table 1 provides a stage-level time breakdown showing that $T_{\text{gen}}$ (the only stage speculation accelerates) accounts for 65–72% of step time. Table 2 reports generation latency for autoregressive, n-gram, and EAGLE-3, but does not decompose generation latency into prefill, draft, and verification components. Figure 2a shows generation latency per step over training, but again without sub-component decomposition. The simulator projections in Section 4 include draft model overhead in their modeling (the simulator "incorporates detailed models of GPU compute units"), but since the simulator is unvalidated (see Limitation 6.4), a reader cannot assess whether its modeling of batching effects and draft overhead is accurate.

Mitigation status. The paper acknowledges that speculative decoding "targets only $T_{\text{gen}}$ , and within generation, only the autoregressive decode phase (not prefill)" (Section 2.2), which is a precise scoping of the limitation. However, the paper does not quantify how much of generation time is prefill vs. decode in their experiments, does not profile draft model overhead separately from verification time, and does not analyze how batching affects the realized speedup. The Amdahl's law bound (Section 2.2) assumes "each speculation step costs the same as one autoregressive forward pass" and notes that "draft model overhead, prefill time, and batching effects reduce the realized speedup below this bound"—but these reductions are never measured. The paper does not suggest profiling methodologies or future work on understanding the latency-throughput tradeoffs of speculative decoding in batched serving systems.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes speculative decoding from an inference-serving optimization into a first-class systems primitive for RL post-training, establishing that lossless acceleration can be deployed inside a production RL stack without altering the optimization trajectory. The shift is primarily methodological rather than algorithmic: the paper does not invent a new speculative decoding technique, but it demonstrates—through working integration and careful characterization—that throughput and distributional fidelity need not trade off in the specific bottleneck that dominates RL training wall-clock time.

The conceptual contribution is the recognition that speculative decoding occupies a unique point on the effectiveness-throughput Pareto frontier that no prior RL acceleration method could reach. Asynchronous execution, off-policy replay, low-precision rollouts, and selective prompt filtering—all the standard levers for accelerating RL training—each perturb the sampling distribution to some degree, creating an effectiveness cost that must be weighed against the throughput gain. Speculative decoding, by contrast, provides throughput improvement with zero effectiveness cost, at least in the verifier-exact sense that the policy sees indistinguishable trajectories. This does not render prior methods obsolete—asynchronous execution, for example, composes with speculation (Section 3.3) and addresses different parts of the critical path—but it adds a new axis to the design space that was previously unavailable.

The paper's diagnostic framework—the Amdahl's law decomposition $S_{\text{step}} \leq 1 / (R_{\text{gen}}/\alpha + (1 - R_{\text{gen}}))$ (Section 2.2)—is likely to have more lasting impact than any specific speedup number. This equation, while simple, provides a unifying lens for understanding rollout-side acceleration that the field previously lacked. It explains why generation share $R_{\text{gen}}$ is the first quantity any practitioner should measure before deciding whether speculation is worth deploying: if $R_{\text{gen}}$ is 30%, the maximum possible end-to-end speedup from perfect speculation is only $1/(1-0.3) = 1.43\times$ , regardless of how good the draft model is. It explains why acceptance length $\alpha$ is a necessary but insufficient metric—the n-gram draft achieves $\alpha = 2.47$ on RL-Zero yet is slower than autoregressive (Table 2), because the effective $\alpha$ per unit cost is what matters. And it explains why draft length optimization (Table 4) finds a sweet spot at $k=3$ rather than monotonically improving with larger $k$ : the per-step verification cost grows with $k$ , making the cost-adjusted acceptance length peak at small draft sizes.

This diagnostic framework redirects research attention in concrete ways. Research directions that become more attractive:

Verifier and draft model quality as the primary constraint on speculation's benefit. The Amdahl's law bound shows that, for a given $R_{\text{gen}}$ , the end-to-end speedup is determined almost entirely by $\alpha$ . Improving $\alpha$ —through better draft architectures, better draft training data, or draft-policy alignment techniques—is the most direct path to higher speedup. The paper's finding that DAPO initialization improves speedup by 17–29% over generic chat initialization (Table 3) provides a concrete magnitude for how much draft quality matters.
Characterizing and reducing the non-generation stages that form the Amdahl ceiling. The 1.35–1.41× end-to-end speedup at 8B scale (Table 1) is limited not by speculation's effectiveness (generation speedup is 1.5–1.8×) but by the 28–34% of step time spent on log-probability recomputation and training. Any technique that reduces $T_{\text{logprob}}$ or $T_{\text{train}}$ —even without accelerating generation—directly raises the speedup ceiling by increasing the effective $R_{\text{gen}}$ . This makes training-side optimizations (kernel fusion, mixed-precision log-probability computation, more efficient advantage estimation) complementary to rollout-side acceleration in a mathematically precise way.
Draft initialization as a first-class engineering concern in RL training pipelines. The paper's finding that in-domain initialization (DAPO) provides large, persistent benefits while online adaptation provides negligible additional gain when initialization is good (Table 5) suggests that the field should invest in draft data generation and training as a standard pre-training step, analogous to how instruction tuning data is prepared before SFT. The draft model is not an afterthought to be trained during RL; it is a component whose quality at initialization largely determines the speedup throughout training.

Research directions that become less attractive:

Developing ever-more-sophisticated speculative decoding algorithms for RL without addressing the Amdahl ceiling. The simulator projections (Figure 3b) show that even a perfect draft (acceptance length 8 at $k=7$ , yielding 6.49× rollout speedup) translates to only 2.22× end-to-end speedup because of the non-generation stages. For models or training configurations where $R_{\text{gen}}$ is low, the ceiling is so tight that no amount of draft quality improvement can yield meaningful end-to-end gains. Research on speculative decoding for RL should therefore be coupled with analysis of $R_{\text{gen}}$ in the target deployment regime; otherwise, it risks optimizing a metric (rollout speedup) that is ceiling-bound.
Treating online draft adaptation as a universal requirement for RL integration. The paper's Table 5 is a clear negative result: online adaptation provides essentially zero benefit when the draft is well-initialized (DAPO), and only modest benefit when initialization is poor (UltraChat, +6–8% speedup). This suggests that the research community's focus on online draft learning algorithms (cf. FastGRPO by Zhang et al., 2025; ReSpec by Chen et al., 2025) may be solving a problem that is better addressed through better initialization. For many practical RL training scenarios, a one-time investment in draft data generation before training begins may be more cost-effective than building and maintaining online draft training infrastructure.
Long draft lengths as a research direction for RL speculative decoding. The paper's Table 4 shows that $k=5$ and $k=7$ drafts are slower than autoregressive on RL-Think (0.84× and 0.71×), and even on RL-Zero, the speedup degrades from 1.77× at $k=3$ to 1.21× at $k=7$ . The simulator heatmap (Figure 3) confirms that the draft-length sweet spot is narrow and scale-invariant. Research effort on techniques for making longer drafts work (e.g., better multi-token prediction heads, tree-based verification) would need to overcome the fundamental cost scaling: each additional draft token adds verification overhead, and the marginal acceptance probability per additional token declines, making the cost-per-accepted-token optimal at small $k$ .

The paper also provides a reconciliation mechanism for the apparent tension between speculative decoding's theoretical lossless guarantee and the practical concern that draft models might produce subtly different trajectory distributions that affect RL training. The validation accuracy curves (Figure 2b) demonstrate that, at least for mathematical reasoning GRPO over 600–1000 steps, the trajectories are sufficiently identical that the optimizer follows the same path. This is not a proof that speculative decoding is always lossless in the RL-relevant sense—the rejection sampling guarantee is token-level, and trajectory-level equivalence requires that the implementation correctly composes the token-level guarantees—but it provides strong empirical evidence that a careful integration (Section 2.3, Figure 1) achieves trajectory-level preservation in practice. The paper thus resolves a concern that might otherwise prevent practitioners from adopting speculative decoding in RL: the fear that "lossless" is a theoretical property that breaks in the messy reality of distributed training systems.

Follow-Up Research This Work Enables

Cross-model and cross-architecture validation of the Amdahl's law framework. The paper's entire empirical evidence base is Qwen3-8B on mathematical reasoning. The Amdahl's law bound is architecture-agnostic, but the key inputs— $R_{\text{gen}}$ (generation share) and $\alpha$ (acceptance length)—are model-specific and task-specific. A systematic study measuring $R_{\text{gen}}$ and $\alpha$ for a range of model architectures (dense vs. MoE, different model sizes from 1B to 70B, different attention patterns) and task domains (code generation, multi-turn agentic trajectories, long-form QA) would establish whether the paper's specific speedup numbers generalize and whether the optimal draft length of $k=3$ is universal. A strong study would: (1) measure $R_{\text{gen}}$ from the autoregressive stage breakdown for each configuration, (2) train an EAGLE-3 draft with in-domain initialization for each, (3) sweep $k \in \{1, 3, 5, 7\}$ to find the optimal per configuration, (4) report acceptance length, generation speedup, and end-to-end speedup, and (5) test whether the Amdahl bound accurately predicts the ceiling. This would transform the paper's single-point findings into a predictive framework that practitioners can use to estimate speculation's benefit for their specific deployment without running full RL training experiments.

Draft data efficiency: how many policy outputs are needed for effective initialization? The paper's DAPO initialization uses policy outputs on the full 17K training prompts, but the computational cost of this initialization (generating all outputs plus training the draft) is never quantified. An ablation study varying the number of training prompts used for draft initialization—from 100 to 17,000—and measuring the resulting acceptance length and speedup would establish the draft data scaling law. If 1,000 prompts yield 95% of the full 17K speedup, the initialization cost drops dramatically and becomes practical for much larger training sets. If 17K is near the minimum needed, practitioners must budget substantial compute for draft training before RL begins. A strong study would also test whether the policy outputs used for draft training need to come from the exact RL training prompts, or whether a held-out set of similar prompts (e.g., other math problems from the same distribution) is equally effective—this matters for scenarios where the RL training prompts are not available before training starts (e.g., when prompts are generated online or drawn from a dynamic distribution).

Draft staleness over long training horizons and the online adaptation threshold. The paper's training runs are 600–1000 steps, and the DAPO-initialized draft shows no degradation in speedup over this horizon (Figure 2a). But for large-scale RL training campaigns that run for tens of thousands of steps, the policy may drift far enough that even a well-initialized draft becomes stale. An experiment running RL-Zero or RL-Think for 10,000+ steps with periodic acceptance length monitoring would identify when (if ever) the DAPO-initialized draft's acceptance length drops below the cost-effectiveness threshold. If acceptance remains stable for 10K steps, the paper's finding that online adaptation is unnecessary generalizes to long horizons. If it drops, the experiment would characterize the decay rate and test whether periodic offline re-initialization (re-training the draft on the current policy's outputs at checkpoints) or continuous online adaptation can recover the speedup. This is a stress-test that determines whether the paper's "insurance" framing of online adaptation (Table 5) holds at scale or whether online adaptation becomes necessary for long-running training.

Interaction between speculative decoding and KL-constrained RL algorithms. The paper uses GRPO (Shao et al., 2024), which does not include an explicit KL penalty against a reference policy. Many production RL pipelines use PPO with a KL penalty (or variants like RLOO, REINFORCE with baseline) that constrains how far the policy can move from a reference model. KL constraints may limit the policy's distribution shift during training, which could increase the effective horizon over which a frozen draft remains aligned—potentially making the DAPO-initialized draft effective for even longer training runs. Conversely, if the KL penalty is computed against a reference policy that differs from the draft's training distribution, the draft might become misaligned with the policy even if the policy hasn't moved far from the reference. An experiment comparing GRPO vs. PPO (with varying KL penalty strength) under speculative decoding, measuring acceptance length and speedup throughout training, would characterize how the choice of RL algorithm affects speculation's benefit. A strong study would also test whether the KL penalty can be computed using the draft model's log-probabilities (avoiding the $T_{\text{logprob}}$ recomputation stage entirely) and whether the resulting approximation error is acceptable—this is a natural extension that the paper's system architecture almost supports but does not explore.

Agentic RL workloads: measuring the prefill-to-decode ratio and its effect on speculation benefit. The paper's introduction identifies agentic RL as an emerging bottleneck but provides no experiments on agentic tasks. Agentic RL differs from mathematical reasoning in two ways critical for speculative decoding: (1) each model call may involve a long context (accumulated tool outputs, conversation history) and a short generation (a single action or reasoning step), making the workload prefill-heavy and reducing the decode-phase fraction that speculation accelerates; (2) the token-level predictability of agent actions may be fundamentally different from mathematical reasoning traces. An experiment applying speculative decoding to an agentic RL benchmark (e.g., WebArena, ToolBench, or a custom multi-turn tool-use environment) with the same system architecture would measure: (a) the prefill-to-decode time ratio in the autoregressive baseline, (b) the acceptance length for an in-domain-initialized EAGLE-3 draft, (c) the resulting generation and end-to-end speedup. If the prefill phase dominates (e.g., 80% of generation time), and speculation only accelerates the remaining 20% decode phase, the Amdahl ceiling would be severe—even perfect speculation could not exceed a 1.25× end-to-end speedup. This negative result would be important: it would establish a domain boundary where speculative decoding is not the right acceleration primitive, and where other methods (e.g., prefill optimization, KV-cache compression, or prompt compression) should be prioritized instead.

Combining speculative decoding with other rollout acceleration methods to push past the Amdahl ceiling. The paper's Amdahl's law analysis shows that end-to-end speedup is bounded by the non-generation stages $T_{\text{logprob}}$ and $T_{\text{train}}$ . But these stages need not be immutable. If $T_{\text{logprob}}$ can be reduced (e.g., by approximating log-probabilities using the draft model, by subsampling tokens for KL computation, or by quantizing the log-probability forward pass), the effective $R_{\text{gen}}$ increases, and the ceiling rises. Similarly, if $T_{\text{train}}$ can be reduced (e.g., through gradient accumulation across multiple rollout batches, or through low-precision optimizer states), the ceiling rises further. An experiment that combines speculative decoding with one or more of these complementary optimizations would test whether the combined speedup is multiplicative—for example, if speculation provides 1.8× generation speedup and log-probability quantization provides 1.5× $T_{\text{logprob}}$ speedup, the combined end-to-end speedup might approach the product rather than the sum. The key measurement would be the end-to-end step time breakdown under the combined configuration, compared against each optimization applied individually. The paper's system architecture (Figure 1) is modular enough to support this—the vLLM rollout engine, the MegatronLM log-probability recomputation, and the GRPO training are separate stages with clean interfaces—making such combination experiments straightforward to implement.

Practical Applications and Downstream Use Cases

Cost-efficient RL post-training for mid-scale models (8B–70B parameters) on reasoning tasks. The most direct application of this paper is accelerating GRPO-style mathematical reasoning post-training for models at the scale where the experiments were conducted. Table 1 shows a 1.35–1.41× end-to-end step speedup on 8B models—meaning a training run that would take 7 days with autoregressive decoding takes approximately 5 days with speculative decoding, saving roughly 2 days of GPU time per training run at 32-GPU scale. This translates to approximately 1,500 GPU-hours saved per run. For teams running multiple ablations, hyperparameter sweeps, or iterative training pipelines (e.g., multiple rounds of RL with intermediate SFT), the cumulative savings are substantial. The finding that DAPO-style in-domain initialization provides nearly all the benefit without requiring online draft maintenance (Tables 3, 5) makes this practical: the upfront cost of generating policy outputs and training the draft is a one-time investment amortized over the full training run. The system integration described in Section 2.3 and Figure 1 provides a concrete, replicable architecture for teams using NeMo RL with a vLLM backend, and the principles (weight synchronization, log-probability correctness, draft-policy alignment) transfer to other RL frameworks.

Frontier-scale RL training where generation dominates wall-clock time and GPU allocations are large. The simulator projections in Section 4 suggest that speculative decoding's benefit grows with model scale: at 235B parameters on 2048 GPUs with modest policy lag (2), rollout speedup reaches ~3.5× and end-to-end training speedup reaches approximately 2.5×. While these specific numbers are unvalidated projections (see Limitation 6.4), the trend is mechanically sound: larger models have longer per-token latencies and produce longer reasoning traces, both of which increase $R_{\text{gen}}$ and give speculation more room to operate. For organizations training frontier models (100B+ parameters) with thousands of GPUs, even a 1.5× end-to-end speedup translates to weeks of wall-clock time saved and millions of dollars in compute cost. The paper's finding that the optimal draft length remains small ( $k=3$ ) and that in-domain initialization is critical applies regardless of scale, giving practitioners concrete guidance without requiring them to reproduce the simulator experiments. A frontier training team adopting this method would: (1) generate policy outputs on their training prompts before RL begins (the draft initialization cost, which at 235B scale is non-trivial but a one-time investment), (2) train an EAGLE-3 draft head on those outputs, (3) deploy with $k=3$ and the gradient-detached architecture from Figure 1, and (4) monitor acceptance length and validation accuracy throughout training to verify that the speedup is sustained and the optimization trajectory is preserved.

Iterative self-improvement pipelines where the same model undergoes multiple rounds of generation and training. In self-improvement paradigms (e.g., ReST, STaR, or iterative DPO/RL), the model generates outputs on training prompts, those outputs are filtered or scored, and the model is fine-tuned on the high-quality subset—a process that repeats for multiple rounds. Each round involves a generation phase (where the current model produces outputs) and a training phase. Speculative decoding accelerates the generation phase of each round, and because the policy changes between rounds (due to fine-tuning), the draft initialization question recurs: after each training round, the policy's output distribution shifts, and the draft from the previous round may become stale. The paper's finding that DAPO-initialized drafts remain effective for 600–1000 GRPO steps without online updates (Figure 2a, Table 5) suggests that for small, incremental policy updates (as in iterative fine-tuning with modest learning rates), the draft may not need re-initialization between rounds. However, for larger distribution shifts (e.g., after a full round of SFT on new data), re-initializing the draft on the updated policy's outputs would be advisable. A self-improvement team would: (1) train an initial draft on the base policy's outputs, (2) use it for the first round's generation phase, (3) after training, evaluate whether acceptance length on the new policy's outputs has dropped significantly, (4) if so, re-initialize the draft; if not, continue using the existing draft for the next round. The decision rule is: re-initialize when acceptance length drops below the threshold where speculation's per-step overhead outweighs the savings (approximately $\alpha < 2$ based on the n-gram result in Table 2, where $\alpha = 2.05-2.47$ was insufficient for positive speedup).

On-policy RL for tasks where distributional fidelity is unusually important. Some RL applications are particularly sensitive to distribution mismatch between the rollout-generating policy and the policy being optimized. Examples include RLHF for safety-critical behaviors (where off-policy corrections might systematically underweight unsafe trajectories), RL for factuality (where low-precision rollouts might hallucinate facts that the full-precision policy would not), and RL in domains with sharp reward functions (where small distribution shifts can cause large changes in expected reward). In these settings, the "lossless" property of speculative decoding—statistically indistinguishable trajectories from autoregressive generation—is not merely convenient; it is necessary to maintain the integrity of the training signal. Other acceleration methods (async execution, off-policy replay, low-precision rollouts) introduce distribution mismatch that, while often acceptable in practice, creates uncertainty about whether the final policy quality was limited by the acceleration method rather than the RL algorithm. Speculative decoding eliminates this uncertainty: the validation accuracy curves in Figure 2b demonstrate that the optimization trajectory is identical to the autoregressive baseline, so any degradation in final policy quality cannot be attributed to the rollout acceleration. For high-stakes RL training where correctness guarantees matter, this is a decisive advantage over methods that trade fidelity for speed.