Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster

🎯 Pitch

Large language models can answer reasoning questions nearly 20% faster by glimpsing their own noisy, partially generated future rationale instead of writing it all out. FastCoT fuses fast parallel Jacobi decoding with standard autoregressive generation, treating discarded approximate tokens as a shortcut that slashes inference time with negligible accuracy loss. This model-agnostic trick requires no retraining and suggests that for capable models, a rough sketch of the reasoning is often enough to land on the correct answer.

1. Executive Summary

This paper proposes FastCoT, a model-agnostic framework that accelerates chain-of-thought reasoning inference without training or model modification by integrating parallel Jacobi decoding—which generates multiple approximate future tokens in a single forward—alongside standard autoregressive decoding, treating the unverified approximate tokens not as discarded by-products but as a "glimpse of future" that enables the LLM to reach answers with fewer total forward passes. Evaluated on Llama 2 (7B/13B) and Llama 1 (7B/13B) across CSQA, StrategyQA, and AQuA, FastCoT achieves up to a ~20% reduction in wall-clock inference time (e.g., 326.93s vs. 365.20s on Llama2-13B with CSQA) with negligible performance loss under 3%, establishing that partial—even noisy—rationale snippets suffice for correct answer extraction only when the LLM is already capable of producing the full rationale autoregressively.

2. Context and Motivation

The Core Problem: CoT Reasoning Is Accurate but Slow, and Prior Speedups Ignore Reasoning Tasks

The fundamental tension this paper addresses is familiar to anyone who has deployed LLMs for complex reasoning: chain-of-thought (CoT) prompting dramatically improves accuracy on reasoning benchmarks, but it does so by forcing the model to generate a long, sequential rationale before producing the final answer. Since most state-of-the-art LLMs are autoregressive transformers — meaning they predict one token at a time, with each token's generation depending on all previous tokens — the extra rationale tokens translate directly into extra forward passes through the model. Each forward pass involves loading the full model weights and computing attention over an ever-growing context, making the computational cost of CoT reasoning substantially higher than directly answering a question.

This is not a minor overhead. In the paper's experimental setting with Llama2-13B on CSQA, the Vanilla CoT baseline consumes 365.20 seconds of wall-clock time across the test set. The authors' core question, stated bluntly in Section 1, is:

"can a large language model (LLM) benefit from its inner approximate reasoning rationale?"

In other words: must the model laboriously spell out every intermediate reasoning step with perfect fidelity, or can it reach the correct answer using only a partial, noisy sketch of the rationale — the kind of sketch that a faster but less precise decoding strategy might produce?

This question matters for several concrete reasons:

Industrial throughput and cost. For applications involving large volumes of reasoning queries (customer support, automated tutoring, legal document analysis), the inference-time cost of generating full rationales can dominate deployment budgets. A 20% reduction in inference time, as FastCoT achieves, directly translates to 20% more queries served per GPU-hour — a substantial cost saving at scale.
User experience and latency. CoT reasoning is inherently serial: the user must wait for the entire rationale to stream out before seeing the answer. Reducing the total wall-clock time improves interactivity, particularly for latency-sensitive applications like voice assistants or real-time tutoring systems.
Theoretical significance. The question tests a hypothesis about how LLMs use their own generated context. If partial, even corrupted rationales can trigger correct answers, it suggests that the rationale's role is not to logically derive the answer from first principles (as a human might) but rather to steer the model's internal representations toward a region of embedding space where the correct answer becomes the highest-probability continuation. This has implications for how we understand CoT's mechanism: it may function more like a learned retrieval cue or a structured prompt extension than a verified logical proof.

Prior Approaches: Separate Worlds of XoT Prompting and Decoding Acceleration

The paper identifies two largely disconnected research threads that approach the CoT efficiency problem from opposite directions, neither of which directly addresses the core question:

XoT Prompt Engineering (Improving Accuracy, Not Speed)

A rich body of work, collectively termed "XoT" by the authors, extends the basic CoT idea in various ways to improve reasoning accuracy: Self-Consistency (Wang et al., 2022) samples multiple rationales and takes a majority vote over answers; Tree-of-Thought (Long, 2023) and Graph-of-Thought (Besta et al., 2023) structure the reasoning process as a search over intermediate states; Self-Ask (Press et al., 2022) decomposes questions into sub-questions; Maieutic Prompting (Jung et al., 2022) generates recursive explanations. Other work focuses on rationale distillation — using CoT-generated rationales from a large teacher model to train smaller student models (Magister et al., 2022; Hsieh et al., 2023; Li et al., 2023) — which improves the student's accuracy but does not address inference speed for the teacher.

The critical limitation of all XoT work, from FastCoT's perspective, is that it uniformly increases inference cost. Self-Consistency requires generating multiple complete rationales per question. Tree-of-Thought requires many forward passes to explore branches. Rationale distillation still requires the teacher to generate full rationales during training. As the authors note:

"these methods are capable of yielding more accurate results compared to the non-XoT approach. However, they compromise the speed of inference."

This creates an asymmetry: the research community has invested heavily in making CoT more accurate, but has done almost nothing to make it faster — despite speed being equally important for practical deployment.

Accuracy-Lossless Decoding Acceleration (Ignoring Reasoning Tasks)

A parallel line of work focuses exclusively on speeding up autoregressive generation while maintaining exact output equivalence with greedy decoding. The two dominant paradigms are:

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Xia et al., 2022) uses a smaller, faster draft model to propose several tokens at once, which the large model then verifies in parallel. Verified tokens are accepted; rejected tokens trigger re-sampling from the large model. This achieves speedups by parallelizing verification while preserving the identical output distribution.

Jacobi decoding (Santilli et al., 2023), the direct predecessor to FastCoT, takes a different approach: instead of using a separate draft model, it feeds the model's own previous predictions as "guesses" for future positions, then runs one forward pass to simultaneously refine all positions. Over multiple iterations, this converges to the same tokens that autoregressive decoding would produce. Santilli et al. demonstrated this for machine translation tasks.

Both paradigms share a key assumption: the output must be identical to what autoregressive decoding would produce. The approximate intermediate tokens generated during Jacobi iterations are treated as transient artifacts — discarded once verification determines they don't match the autoregressive greedy output. Santilli et al. explicitly focus only on tokens that survive verification; the "by-products" are ignored.

The critical gap that FastCoT identifies is at the intersection of these two worlds:

"To the best of our knowledge, we are the first to propose exploiting the concept of approximate rationale to expedite the completion of XoT-series tasks."

No prior work had asked whether the inaccurate, unverified tokens produced during speculative or Jacobi decoding — the very tokens that accuracy-lossless methods discard — might actually contain useful information for guiding the model toward the correct answer. This is the paper's core conceptual move: re-interpreting Jacobi decoding's by-products from "verification failures" to "a glimpse of future" that can accelerate reasoning specifically.

The Missing Evidence: Can LLMs Reason From Partial Rationales?

Before proposing FastCoT, the authors needed to establish a motivating empirical claim: that LLMs don't actually need complete, perfectly accurate rationales to produce correct answers. If an LLM required every token of the full rationale, then any method that skips or corrupts rationale tokens would necessarily degrade accuracy, making the speed-accuracy tradeoff unfavorable.

To test this, the paper conducts a corrupting rationale experiment (Section 5.2) that serves as the empirical foundation for the entire method. The procedure is:

Run Vanilla CoT (autoregressive decoding) on each question to generate a complete, accurate rationale.
Corrupt the rationale by replacing a fraction of its tokens with [PAD] tokens. Two corruption patterns are used with equal probability: (a) sequential masking from the end of the rationale toward the beginning (simulating truncation after partial generation), and (b) random uniform masking across the rationale (simulating the scattered errors produced by Jacobi decoding's approximate tokens).
Repeat the corruption 100 times with different random seeds per question, producing diverse sets of partially corrupted rationales.
Feed the original prompt + corrupted rationale to the same LLM and ask it to produce the answer.

The results, displayed in Figure 5, are striking:

"when only 40% of the rationale generated by autoregressive decoding is revealed in StrategyQA dataset, the performance is already saturated."

In other words, the model achieves essentially the same accuracy whether it sees 40% of the rationale or 100% — the remaining 60% is redundant for answer extraction. The curves for CSQA, StrategyQA, and AQuA all show the same pattern: accuracy rises quickly with overlap ratio and then plateaus well before reaching 1.0 (full rationale).

This finding has a crucial implication: the value of a rationale token for answer extraction is not uniform. Some tokens (likely keywords, entities, logical connectors, or the structural framing of the reasoning path) carry most of the information; others are filler, stylistic flourishes, or redundant restatements. If a faster decoding method can generate the high-value tokens — even approximately — while skipping the low-value ones, it can match autoregressive CoT accuracy with fewer forward passes.

However, the paper is careful not to overclaim. The corrupting rationale experiment uses oracle rationales (generated by the model itself under autoregressive decoding, then artificially corrupted). This demonstrates that the model can reason from partial information when the partial information is a subset of a genuine rationale. It does not demonstrate that the model can reason from arbitrary partial information, nor that the approximate tokens from Jacobi decoding will be sufficiently similar to genuine rationale tokens to trigger the same behavior. That is the empirical question FastCoT proper must answer.

Where Existing Work Falls Short: Three Specific Gaps

The paper identifies three precise shortcomings in prior approaches that motivate FastCoT's design:

1. Decoding acceleration ignores reasoning tasks entirely. Speculative decoding and Jacobi decoding have been demonstrated on machine translation (Santilli et al., 2023), open-ended text generation, and similar tasks where the output is consumed as-is. The unique structure of CoT reasoning — where tokens serve as an intermediate representation to condition the model's own subsequent output, not as the final deliverable — has never been considered in the decoding acceleration literature. This matters because the tolerance for token-level inaccuracy in an intermediate representation may be very different from the tolerance for inaccuracy in the final output. An incorrect word in a translation is a bug. An incorrect word in an intermediate reasoning step might be harmless if the model can see past it.

2. Jacobi decoding's by-products are wasted. Santilli et al.'s original formulation of Jacobi decoding for translation treats the iterative refinement process as a means to an end: converge to the autoregressive greedy output using fewer sequential steps. The approximate tokens generated in early iterations are discarded once verification rejects them. FastCoT's key insight is that for reasoning tasks, these discarded tokens are potentially valuable even though they're wrong — they provide a "glimpse of future" that can accelerate answer generation. This reframing transforms Jacobi decoding from a verified-decoding algorithm into a feature extraction mechanism where the feature (approximate future context) is useful regardless of whether it passes verification.

3. No prior work combines approximate decoding with CoT. As the authors state in Section 2:

"we are the first paper to attempt to utilize these approximate tokens."

This is not just a claim about their specific method — it's an observation about a blind spot in the literature. The decoding acceleration community optimized for exact output matching. The CoT community optimized for accuracy, treating generation speed as a secondary concern. Neither community had reason to consider the possibility that fast-but-inaccurate decoding could be beneficial for reasoning tasks. FastCoT positions itself at this unexplored intersection.

How FastCoT Positions Itself

FastCoT does not compete with XoT prompt engineering methods — it is explicitly "orthogonal to the prompt engineering approach" and "parallel to these engineering efforts." Any prompt method that produces better rationales (few-shot CoT, Self-Consistency, Tree-of-Thought, etc.) could in principle be combined with FastCoT's decoding acceleration, since FastCoT operates purely at the generation mechanism level.

Similarly, FastCoT is positioned as complementary to, not competitive with, lossless acceleration methods. The paper acknowledges that "most state-of-the-art causal transformers are autoregressive models, meaning they can only predict tokens one at a time. This leads to slow inference and may not fully utilize the capabilities of GPUs" (Section 1). FastCoT's contribution is not to propose a better verification mechanism or a faster draft model, but to show that for reasoning tasks specifically, we can go beyond lossless decoding by treating Jacobi decoding's by-products as useful context rather than errors to be discarded.

The key boundary condition that the paper establishes — and which a careful reader should note — is that FastCoT's acceleration depends on the model being able to reach the correct answer from partial rationale context. The corrupting rationale experiment demonstrates this is possible for the datasets and models tested, but this is an empirical property of the model-dataset pair, not a guaranteed property of all reasoning tasks. If a task requires precise intermediate calculations where every token matters (e.g., multi-digit arithmetic), the "glimpse of future" approach may degrade accuracy unacceptably. The paper does not test such tasks, so this remains a limitation to be aware of.

The paper also positions itself pragmatically: FastCoT requires no training, no auxiliary models, and minimal implementation changes ("almost no modification to the source code of the language model itself based on the huggingface implementations"). This distinguishes it from speculative decoding approaches that require training or selecting a suitable draft model, and from non-autoregressive translation models that require specialized architectures and careful hyperparameter tuning. FastCoT is designed to be "model-agnostic" — a drop-in acceleration layer that works with any causal transformer.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The system being built is a drop-in decoding engine that replaces standard autoregressive token-by-token generation during chain-of-thought reasoning, without changing the underlying language model at all. It solves the problem that CoT reasoning is accurate but slow—each rationale token requires a separate forward pass through a massive model—by generating multiple approximate future tokens in each forward pass and using those partial, noisy tokens to trigger the final answer earlier than the model would normally reach it.

3.2 Big-Picture Architecture (Diagram in Words)

The FastCoT system has five major components that orchestrate a modified inference loop:

Prompt Constructor — takes a reasoning question and builds the initial prompt (identical to standard CoT prompting, including few-shot examples if used). This component is unchanged from the baseline.
Approximate Tokens Buffer (ATB) — a data structure holding the partial, evolving sequence of generated tokens. It maintains two contiguous regions: exact tokens (those verified to match autoregressive greedy output) followed by approximate tokens (unverified Jacobi decoding outputs that serve as "glimpses of future"). The buffer is initialized with the tokenized question itself (a design choice explored empirically).
Parallel Decoding Engine — executes a single forward pass through the LLM that simultaneously generates one exact next token (via standard autoregressive prediction) and multiple approximate future tokens (via Jacobi decoding's iterative refinement mechanism). This is the computational core: one GPU inference call produces tokens at multiple positions in the sequence.
Verification Module — after each forward pass, compares the newly generated tokens against the previous iteration's tokens to determine how many consecutive positions match exactly. Matched tokens are promoted from "approximate" to "exact" status; mismatched tokens remain approximate. This module also updates the ATB's boundary pointer I (the position where exact tokens end and approximate tokens begin).
Iteration Controller — decides when to stop the decoding loop based on configurable conditions (a pre-computed maximum iteration count, detection of an EOS token in the exact portion, or other triggers). When a stop condition fires, it appends the answer trigger prompt ("So the answer is") to all generated tokens and runs one final forward pass to extract the answer.

Information flows as follows: a question enters → the Prompt Constructor builds the initial prompt and tokenizes it → the tokenized prompt initializes the ATB (so the "approximate tokens" at iteration 0 are just the question itself) → the Parallel Decoding Engine runs a forward pass, producing one exact token and c approximate tokens (where c is the Jacobi context window size) → the Verification Module determines how many tokens at the boundary can be promoted to exact status → the ATB is updated, with the exact-region pointer I advancing by however many tokens were verified → the Iteration Controller checks stop conditions → if not stopping, the ATB's current contents feed into the next forward pass → when stopping, the answer trigger is appended and one final forward pass extracts the final answer.

The key departure from standard Jacobi decoding is that the approximate tokens are never discarded. They persist through iterations, gradually being overwritten as the exact region expands, and when the loop terminates, the surviving approximate tokens serve as the partial rationale context that the answer-extraction forward pass conditions on.

3.3 Roadmap for the Deep Dive

First, Parallel Jacobi Decoding formalized (Section 3.1 and our extension) — because the entire method builds on the Jacobi iteration, we need precise notation for what a single forward pass computes, what "context window" means, and why the time cost of the extra window positions is negligible. This establishes the computational primitive.
Second, the FastCoT iteration loop — the complete per-iteration procedure: how the ATB is initialized, how a forward pass produces both exact and approximate tokens, how verification determines the new exact-token boundary, and how the buffer is updated. This is the operational core.
Third, the verification mechanism — the critical subroutine that determines which tokens graduate from "approximate" to "exact." This needs its own treatment because it defines the interface between Jacobi decoding's outputs and FastCoT's buffer management, and because the verification criterion (longest prefix match) has specific properties that affect downstream behavior.
Fourth, iteration stop conditions and answer triggering — the three mechanisms for deciding when to terminate the loop and how the answer-trigger prompt extracts the final answer from whatever rationale has been generated so far (both exact and approximate).
Fifth, auxiliary design choices — the ATB initialization strategy (why the question itself rather than [PAD] tokens), the two KV-cache padding mechanisms for batched inference, and the context window size tradeoffs revealed by the ablation study. These are implementation decisions that matter for practical deployment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper that repurposes an existing decoding algorithm (Jacobi decoding) for a new application domain (CoT reasoning acceleration) by recognizing that the algorithm's previously discarded outputs (approximate intermediate tokens) are actually valuable as partial reasoning context. The contribution is not a new mathematical technique but rather a re-interpretation of Jacobi decoding's by-products combined with an engineering framework that manages and leverages these by-products effectively.

Parallel Jacobi Decoding: The Computational Primitive

Before explaining FastCoT's full loop, we must understand exactly what one iteration of Jacobi decoding computes, because FastCoT's speed advantage comes entirely from how it uses this primitive.

Standard autoregressive greedy decoding computes one token per forward pass. Given a prompt $x$ and previously generated tokens $y_{1:i-1}$ , the model outputs a probability distribution over the vocabulary for position $i$ :

$p_\theta(y_i \mid y_{1:i-1}, x)$

The next token $y_i$ is selected as:

$y_i \leftarrow \arg\max \, p_\theta(y_i \mid y_{1:i-1}, x)$

where $\theta$ represents the model parameters, $x$ is the prompt, $y_{1:i-1}$ are the tokens generated so far, and $p_\theta$ is the model's predicted probability distribution over the vocabulary.

What this computes: for a given prefix, the model predicts which token comes next by running the entire prefix through the transformer and reading the logits at the final position, then selecting the highest-probability token (greedy). The output is exactly one new token.

Why this form: autoregressive factorization is the standard causal language modeling objective—predict each token given all previous tokens. Greedy selection (argmax) is the simplest deterministic decoding strategy, producing the same output every time for a given input. This serves as the accuracy baseline that FastCoT aims to approximate with fewer forward passes.

Jacobi decoding modifies this by considering not just the next token but a window of $c$ future positions simultaneously. A single forward pass computes:

\begin{cases} y_i &\leftarrow \arg\max \, p_\theta(y_i \mid x, y_{1:i-1}) \\[4pt] \widehat{Y}^t_{i+1} &\leftarrow \arg\max \, p_\theta(y_{i+1} \mid x, y_{1:i-1}, \widehat{Y}^{t-1}_{i}) \\[4pt] \widehat{Y}^t_{i+2} &\leftarrow \arg\max \, p_\theta(y_{i+2} \mid x, y_{1:i-1}, \widehat{Y}^{t-1}_{i:i+1}) \\[4pt] &\;\; \vdots \\[4pt] \widehat{Y}^t_{i+c} &\leftarrow \arg\max \, p_\theta(y_{i+c} \mid x, y_{1:i-1}, \widehat{Y}^{t-1}_{i:i+c-1}) \end{cases}

where $i$ is the current exact-token position (the start of the Jacobi window), $c$ is the context window size, $t$ is the iteration index, $y_i$ is the exact next token (conditional only on verified past tokens), and $\widehat{Y}^t_{i+1}, \ldots, \widehat{Y}^t_{i+c}$ are the approximate tokens for positions $i+1$ through $i+c$ (each conditional on the previous iteration's approximate tokens at the preceding positions, denoted $\widehat{Y}^{t-1}_{i}$ , $\widehat{Y}^{t-1}_{i:i+1}$ , etc.).

What this computes: one forward pass through the model produces tokens at $c+1$ positions in the sequence—one exact token at the current position $i$ (which uses only verified past context) plus $c$ approximate tokens at future positions $i+1$ through $i+c$ (each of which uses the previous iteration's guesses for the positions before it as context). The key asymmetry is that $y_i$ conditions on ground-truth past tokens ( $y_{1:i-1}$ ), while $\widehat{Y}^t_{i+k}$ conditions on $\widehat{Y}^{t-1}_{i:i+k-1}$ —the previous iteration's approximate guesses, which may be wrong. This is what makes the future-position tokens "approximate": they are predictions based on possibly-incorrect context, so they may differ from what full autoregressive decoding would produce at those positions.

Why this form: Jacobi decoding is an iterative fixed-point method. The idea is that if we initialize the future-position tokens with some guess and repeatedly refine them (each iteration, each position gets to see the most recent guesses for all preceding positions as context), the guesses will converge to the same tokens that autoregressive decoding would produce. The convergence is guaranteed because the first position in the window only conditions on verified tokens, so it's always correct; the second position conditions on the previous iteration's first position, which becomes correct once the first position stabilizes; and so on. Over iterations, correctness propagates left-to-right through the window.

The paper introduces a compact notation for one iteration:

$y_i, \widehat{Y}^t \leftarrow \text{JD}(x, y_{<i}, \widehat{Y}^{t-1}, i, c)$

where $\text{JD}$ stands for one Jacobi Decoding forward pass, $x$ is the prompt, $y_{<i}$ is all verified tokens before position $i$ , $\widehat{Y}^{t-1}$ is the approximate token buffer from the previous iteration, $i$ is the start position, and $c$ is the window size. The output is one new exact token $y_i$ and an updated approximate buffer $\widehat{Y}^t$ .

What the superscript $t$ adds over Santilli et al.'s original formulation: the paper explicitly indexes iterations with $t$ to emphasize that the approximate tokens evolve over time. In the original Jacobi decoding paper, the focus is on the converged final state. FastCoT's innovation is to care about the intermediate states $\widehat{Y}^1, \widehat{Y}^2, \ldots$ and use them as partial reasoning context even before convergence.

The context window size is not time-consuming (Figure 2). A natural concern is that computing predictions for $c+1$ positions should cost roughly $c+1$ times as much as computing for one position. The paper demonstrates this is false for practical window sizes. With Llama2-13B on an A100 GPU, measured across prompt lengths of 400, 600, and 800 tokens, the forward time is essentially flat as the Jacobi context window size increases from 0 (pure autoregressive, computing one position) to 38 (computing 39 positions). The forward time hovers around 35-45ms regardless of window size.

The explanation: when the total decoding length is relatively short (compared to the prompt length), the additional matrix operations for the extra window positions are too small to saturate the GPU's CUDA and Tensor cores. The forward pass is bottlenecked by the large matrix multiplications over the prompt tokens (which are shared across all positions) and the model weights (which are also shared). Adding a few dozen more positions in the decoding segment increases the computation incrementally without crossing the threshold where it would become the dominant cost. This is a hardware-dependent property—it may not hold on all GPU architectures or for all context length ratios—but for the configurations tested, it means the "extra" Jacobi window positions come nearly for free in terms of per-iteration latency.

The FastCoT Iteration Loop: Complete Per-Iteration Procedure

FastCoT's decoding loop is a modified version of the standard Jacobi iteration, with two key changes: (1) the approximate tokens are preserved across iterations and play an explicit role in the final answer extraction, and (2) the iteration loop can terminate early (before full convergence) because the approximate tokens plus the generated exact tokens already contain enough information to answer the question.

Step 1: ATB initialization (iteration $t = 0$ ). Before any forward passes, the Approximate Tokens Buffer $Y$ is initialized with the tokenized prompt. Formally, $Y = [Y^0_{0:I}, \widehat{Y}^0_{I:}]$ where $I$ is a vector of integers indicating the first approximate-token position for each instance in the batch. At $t = 0$ , the paper sets $I = 0$ for all instances, meaning the entire buffer starts as "approximate"—there are no exact tokens yet.

The initial content of the approximate portion is the question itself:

"We choose to initialize Y with populating tokenized token id sequence of the question itself based on empirical results for each question. Compared with the commonly used initialization method, i.e., initializing using a special token [PAD], our method would provide a more diverse initial solution for early stage's Jacobi decoding."

This is a design choice worth understanding. Standard Jacobi decoding (Santilli et al.) initializes future positions with [PAD] tokens—a uniform, uninformative guess. FastCoT initializes with the question text, which is semantically related to the answer and rationale. The intuition: Jacobi decoding's iterative refinement converges faster when the initial guess is closer to the target. By seeding the window with question tokens, the first few iterations' approximate predictions are more likely to contain relevant keywords from the question domain, even if they're not yet correct reasoning tokens. The paper's empirical results support this choice, though no explicit ablation comparing question-initialization vs. PAD-initialization is shown.

Step 2: Forward pass (iteration $t$ ). The Parallel Decoding Engine executes one LLM forward pass, formalized as:

$y_I, \widehat{Y}^{t+1}_{I+1:I+c+1} \leftarrow \text{PD}(x, y_{<I}, \widehat{Y}^t_{I:I+c}, I, c)$

where $\text{PD}$ is the parallel decoding operation (equivalent to Jacobi decoding but with the paper's notation), $x$ is the prompt, $y_{<I}$ are all verified exact tokens before position $I$ , $\widehat{Y}^t_{I:I+c}$ is the current approximate-token window of size $c$ starting at position $I$ , and $I$ identifies where the Jacobi window begins.

The output is one new exact token at position $I$ (denoted $y_I$ ) and $c$ updated approximate tokens at positions $I+1$ through $I+c+1$ (denoted $\widehat{Y}^{t+1}_{I+1:I+c+1}$ ).

What this physically computes in one GPU call: the entire sequence $[x, y_{<I}, \widehat{Y}^t_{I:I+c}]$ is fed through the transformer. The causal attention mask ensures that:

Position $I$ (the exact token to generate) attends to $x$ and $y_{<I}$ only—it cannot see any approximate tokens, preserving the lossless property for this position.
Position $I+k$ (where $1 \leq k \leq c$ ) attends to $x$ , $y_{<I}$ , and the approximate tokens at positions $I$ through $I+k-1$ (from the previous iteration's buffer $\widehat{Y}^t$ )—it can see approximate context but not future approximate tokens.

The model produces logits at all $c+1$ positions. At position $I$ , greedy argmax selects $y_I$ (exact). At positions $I+1$ through $I+c+1$ , greedy argmax selects the updated approximate tokens $\widehat{Y}^{t+1}_{I+1:I+c+1}$ .

Why this asymmetric attention pattern matters: the exact token at position $I$ is guaranteed to match what autoregressive decoding would produce at that position (it sees the same context). The approximate tokens are not guaranteed to match autoregressive output (they see potentially incorrect context). However, because the exact token at position $I$ is now correct, it provides a firmer foundation for the next iteration's predictions at positions $I+1$ and beyond—hence the left-to-right convergence property.

Step 3: Verification. After the forward pass, the Verification Module determines how many of the newly generated approximate tokens actually match the previous iteration's tokens at the same positions. It does this by finding the longest prefix match at the boundary between exact and approximate regions:

$K \leftarrow \arg\max_K \left(Y^{t+1}_{I:I+K} = Y^t_{I:I+K}\right)$

where $K$ is the number of consecutive matching tokens starting from position $I$ , $Y^{t+1}_{I:I+K}$ is the slice of the newly generated buffer from position $I$ to $I+K$ , and $Y^t_{I:I+K}$ is the same slice from the previous iteration's buffer.

What this computes: starting at the boundary position $I$ , compare the new token at each position with the old token at that position. As long as they match, increment $K$ . Stop at the first mismatch. $K$ is the count of consecutive matches.

Why prefix matching is the right verification criterion: in Jacobi decoding, convergence is defined as a fixed point—a state where another iteration produces exactly the same tokens at every position. When position $I$ stabilizes (matches its previous value), it means the exact token at $I$ is finalized (will not change in future iterations). When positions $I+1, I+2, \ldots$ also stabilize, they too are finalized. The prefix match captures exactly the set of positions that have reached this fixed point. Positions beyond $K$ are still in flux—they may change in subsequent iterations as better context becomes available.

A subtle point: the first position in the matching prefix ( $I$ itself) always matches because it's computed from verified exact context (unchanged from the previous iteration's exact token). So $K \geq 1$ always, and the exact region always advances by at least one token per iteration. This guarantees forward progress—FastCoT never stalls.

Step 4: Cache update (ATB update). After verification computes $K$ , the ATB is updated:

Tokens at positions $I$ through $I+K-1$ are promoted from approximate to exact status (they are now part of the verified prefix $y$ ).
Tokens at positions $I+K$ and beyond remain approximate (they are still $\widehat{Y}$ ).
The boundary pointer advances: $I \leftarrow I + K$ .

The updated buffer $Y^{t+1} = [y_{0:I}, \widehat{Y}^{t+1}_{I:}]$ is ready for the next iteration.

Step 5: Stop condition check and loop. The Iteration Controller checks whether any termination condition is met (detailed below). If not, the loop returns to Step 2 with the updated ATB as input for the next forward pass.

Verification Mechanism: How Tokens Graduate From Approximate to Exact

The verification step is the interface between Jacobi decoding's iterative convergence and FastCoT's buffer management. It deserves careful attention because it determines what the model "sees" as verified context in subsequent iterations.

The exact match criterion. Verification is purely lexical: two tokens "match" if and only if their token IDs are identical. There is no semantic similarity threshold, no embedding-space distance, no soft acceptance. A token either is the same integer as the previous iteration's token at that position, or it is not. This is the same criterion used in Santilli et al.'s original Jacobi decoding.

Why lexical matching: the goal of the exact portion of the buffer is to be guaranteed identical to autoregressive greedy decoding. Greedy decoding is deterministic—given the same prefix, it always selects the same next token (the argmax of the logit distribution). If a position has stabilized (the model predicts the same token twice in a row), and the prefix leading to that position is all exact tokens, then the prediction at that position is the same as what autoregressive decoding would produce. Lexical matching is the necessary and sufficient condition for this guarantee.

The longest-prefix property. Verification checks for matches sequentially starting from position $I$ . It does not skip positions. If position $I$ matches and position $I+2$ matches but position $I+1$ does not, then $K = 1$ —only position $I$ is promoted. This is correct because position $I+1$ 's prediction in the current iteration conditioned on the old approximate token at position $I$ , which may have been wrong. Even if position $I+2$ coincidentally matches the old value at that position, we cannot trust it because its context (the intermediate tokens) is unreliable. The prefix property ensures that promoted tokens are built on a chain of verified context.

Consequence for convergence speed. In practice, $K$ is often greater than 1. When the Jacobi initialization (the question text) is close enough to the eventual rationale that the model's predictions stabilize quickly, multiple tokens per iteration graduate to exact status. This is where the speedup over autoregressive decoding comes from: instead of one token per forward pass, FastCoT sometimes gets 2, 3, or more verified tokens per pass. The paper quantifies this in Table 1 with the "IS/TI" column (Iterations Saved / Total Iterations): for Llama2-13B on CSQA, 896 iterations are saved out of 14,684 total—meaning roughly 6% of iterations produced more than one verified token, and those extra tokens are "free" from a forward-pass-counting perspective.

Tokens beyond the match prefix are approximate but not discarded. This is FastCoT's crucial departure from standard Jacobi decoding. In the original formulation, tokens that fail verification are discarded—they don't become part of the output. In FastCoT, they persist in the approximate portion of the buffer. They will be overwritten in subsequent iterations (as the Jacobi window advances and new predictions are made at those positions), but at any given moment, the approximate portion contains the model's best current guess for what comes next—a "glimpse of future" that, while not guaranteed correct, is often directionally relevant.

Iteration Stop Conditions and Answer Triggering

FastCoT does not run Jacobi iterations until full convergence (i.e., until all $c$ window positions have stabilized). Instead, it terminates early based on one of three conditions, then extracts the final answer from whatever partial rationale exists in the ATB.

Stop condition 1: Pre-computed iteration budget (for large datasets). The paper observes that CoT performance gradually converges over Jacobi iterations—the accuracy curve rises and then flattens (Figure 6). Rather than running a fixed large number of iterations for every question, FastCoT dynamically determines a per-dataset iteration budget:

"For a large dataset, we randomly select a small portion of it as $S_\text{cal}$ and perform the CoT task within this subset for each iteration. We calculate the minimum number of iterations required to achieve a certain performance loss threshold and use this value as the upper bound for iterations."

The procedure: (1) Take a small calibration subset $S_\text{cal}$ from the dataset. (2) Run FastCoT on $S_\text{cal}$ for many iterations, measuring accuracy at each iteration. (3) Find the smallest iteration count where accuracy is within some threshold (the "performance loss" budget, typically <3% below the converged accuracy). (4) Use this iteration count as the hard stop for the full dataset.

What this computes: an empirical iteration-accuracy Pareto frontier. The calibration subset tells us "how many iterations are enough" for this model-dataset pair. The performance loss threshold controls the speed-accuracy tradeoff: a tighter threshold means more iterations (higher accuracy, lower speedup), while a looser threshold means fewer iterations (lower accuracy, higher speedup). The paper's reported results use a threshold that yields <3% accuracy loss.

Why this approach: running all questions to full Jacobi convergence would partially defeat the purpose—the speedup comes from stopping before the full rationale is generated. But stopping too early degrades accuracy. The calibration-subset approach provides a data-driven stopping point without requiring per-question optimization.

Stop condition 2: EOS token in the exact region. If the exact-token portion of the ATB contains an End-Of-Sentence token (the special token that language models use to indicate "I'm done generating"), the loop terminates. This indicates the model has naturally concluded its generation via the verified decoding path. In practice, this condition fires less often than the iteration budget, because FastCoT typically stops before the model would naturally finish its full rationale.

Stop condition 3: Answer extraction. When any stop condition fires, FastCoT does not simply output whatever tokens are in the ATB. Instead, it constructs a special prompt for answer extraction:

"we combine the complete prompt, along with all generated tokens, and include the answer trigger, to instruct the LLM to generate the final answer directly."

The answer trigger is a short text string appended to the end of all generated tokens. The paper uses: "So the answer is". This is not chosen arbitrarily—it matches the phrasing that appears naturally in few-shot CoT prompts. In a typical few-shot CoT example, the rationale ends with some variant of "So the answer is (C)" or "Therefore, the answer is...". The answer trigger leverages this learned pattern: the model, having seen the partial rationale (exact + approximate tokens) and then the trigger phrase, is primed to complete the pattern by outputting the answer.

Why this is necessary: without the answer trigger, the model would simply continue generating rationale tokens autoregressively from wherever the exact region ends. The trigger explicitly switches the model's mode from "continue the reasoning" to "extract the answer from the reasoning so far." This is the mechanism that converts a partial, potentially noisy rationale into a concrete answer prediction.

The final forward pass for answer extraction. The trigger-extended sequence is fed through the LLM in one additional forward pass. The model's output at the position following the trigger is taken as the answer. This forward pass is not counted in the iteration budget—it's a separate, necessary cost—but it's a single pass, much cheaper than generating the remaining rationale tokens autoregressively.

Auxiliary Design Choices: ATB Initialization, KV-Cache Padding, and Context Window

Several implementation-level decisions are critical for FastCoT's practical performance and are explained in the paper:

ATB initialization with question tokens (not [PAD]). As noted above, the choice to initialize the approximate buffer with the tokenized question rather than uniform [PAD] tokens is based on empirical results. The mechanism: Jacobi decoding's iterative refinement is a fixed-point iteration. The convergence rate depends on how close the initial guess is to the fixed point. The question text shares vocabulary, entities, and syntactic patterns with the eventual rationale (both are about the same topic), making it a better initial guess than an uninformative [PAD] token repeated $c$ times. This is particularly important in early iterations when the exact region is short and the approximate tokens must bootstrap from minimal context.

KV-cache management for batched inference. A practical complication of Jacobi decoding in batch settings: different instances in the batch will have different exact-token counts at each iteration. Some questions generate rationales faster (more tokens verified per iteration), so after several iterations, the exact-region lengths diverge across the batch. This causes two problems for the transformer's key-value (KV) cache:

Differing historical KV-cache lengths. Instance A might have 50 exact tokens in its history while Instance B has 40. The KV cache storing past attention keys and values for each instance has different lengths, but batched tensor operations require uniform dimensions.
Differing numbers of Jacobi window tokens. Even with a fixed window size $c$ , the total sequence length (prompt + exact tokens + approximate tokens) varies across instances because the exact-token counts differ.

The paper introduces two padding mechanisms to handle these discrepancies (Figure 7):

Type 1 Padding: handles disparities in historical KV-cache lengths. For instances with shorter exact-token histories, padding tokens are appended to the end of their cached key-value tensors (in the "past" portion, not the current input). These padding tokens are masked out during attention computation so they don't affect the model's output. The computational cost is shown in Table 2: 5.33 seconds total across the CSQA test set for Llama2-13B—about 1.6% of the total 326.93s runtime.

Type 2 Padding: handles disparities in the number of input tokens within the current Jacobi window. Instances with fewer tokens to process in the current forward pass have their input tensors padded to match the maximum in the batch, again with attention masking. The cost is 0.84 seconds—negligible (0.26% of runtime).

Strip KV: after each forward pass, the KV cache must be trimmed to retain only the exact-token portion. The approximate-token positions computed during Jacobi decoding should not persist in the cache for future iterations (they'll be recomputed with better context). This "stripping" operation takes 18.58 seconds for Llama2-13B on CSQA—about 5.7% of total runtime. It's a non-trivial cost that is specific to Jacobi-decoding-based methods. The paper is transparent about this overhead in Table 2, showing that while GPU inference time drops from 358.15s (autoregressive) to 274.08s (FastCoT)—a 23.5% reduction—auxiliary operations (padding, stripping, context decoding) consume some of those savings, yielding the net 10.47% total speedup.

Context window size effects on reasoning performance (Figure 8). The paper conducts an ablation sweeping the Jacobi context window size $c$ from 0 to 25 and measuring reasoning accuracy at various iteration counts:

Early iterations (0–20): larger context windows correlate with worse performance. The paper's explanation: in early iterations, the Jacobi-decoded approximate tokens are low-quality (they condition on minimal, potentially uninformative context). A larger window means more of these low-quality tokens are present in the model's context, diluting the signal from the verified exact tokens. The model is effectively being shown more noise.
Mid iterations (10–65): the relationship reverses. After sufficient iterations, the Jacobi iterative solutions have improved in quality (the fixed-point iteration has made progress). Now, a larger context window provides more useful "glimpse of future" information, and accuracy scales positively with window size. However, the paper notes "there is no significant difference observed for context windows larger than 20"—diminishing returns set in.
Late iterations (65+): the performance gap between different context window sizes gradually diminishes. This is expected: as the exact region grows, the model has more verified context to condition on, making the approximate tokens less influential regardless of their quality.

This non-monotonic relationship—large windows hurt early, help later—implies that a dynamic window size that starts small and grows over iterations could be optimal. The paper flags this in the Limitations section as future work, framing it as a Markov Decision Process where a reinforcement learning agent could learn to adjust the window size per iteration.

4. Key Insights and Innovations

Innovation 1: Re-Framing Jacobi Decoding's By-Products as Useful Context Rather Than Errors to Discard

The most fundamental conceptual move in this paper is not a new algorithm but a re-interpretation of an existing algorithm's outputs. Jacobi decoding—as introduced by Santilli et al. (2023) for machine translation—is an accuracy-lossless decoding method: its goal is to produce tokens identical to autoregressive greedy decoding using fewer sequential steps. The "approximate tokens" generated during intermediate Jacobi iterations are transient noise that gets discarded once verification determines they don't match the converged output. They are, by design, invisible to the end user and irrelevant to the algorithm's performance claim.

FastCoT asks a question that the original Jacobi decoding framework had no reason to consider: what if these discarded tokens are actually valuable for certain tasks? The paper's reframing is from "by-products to be discarded" to "a glimpse of future"—an intentional, information-bearing signal that the model can use to reach its answer earlier. This is not an incremental improvement to Jacobi decoding. It is a category shift: the approximate tokens move from being defined by their failure to match a ground-truth output (a purely negative property) to being defined by their partial, noisy correlation with that output (a positive property that can be exploited).

Why is this non-obvious? Because the entire decoding acceleration literature—speculative decoding (Leviathan et al., 2023; Chen et al., 2023), blockwise parallel decoding (Stern et al., 2018), and Jacobi decoding itself—is built on the premise that correctness is binary: a token either matches what autoregressive decoding would produce, or it is wrong. The field's universal design goal is to produce the exact same output distribution as autoregressive decoding, just faster. FastCoT challenges this premise specifically for chain-of-thought reasoning tasks by arguing that correctness is the wrong metric for the intermediate rationale tokens. What matters is whether the partial, approximate tokens are sufficient to steer the model toward the correct final answer—not whether they are identical to what a full autoregressive rationale would contain.

The evidence for this conceptual move comes from the corrupting rationale experiment (Figure 5): when 60% of an oracle rationale's tokens are randomly masked, the model still reaches saturated accuracy. This demonstrates that rationale tokens are not equally informative, and that the model can tolerate substantial noise in its intermediate context without degrading answer quality. The paper does not cite prior work that made this demonstration for CoT reasoning, nor prior work that connected it to decoding acceleration. The corrupting rationale experiment is not a contribution in itself—it's a diagnostic probe—but it justifies the entire reframing: if the model can reason from corrupted rationales, then Jacobi decoding's approximate tokens (which are "corrupted" relative to the autoregressive output) might be sufficient for answer extraction even though they would fail verification.

This reframing has significance beyond the specific method. It opens a broader research question: what other intermediate computations in LLM pipelines are unnecessarily constrained by exact-correctness requirements when approximate outputs would suffice for downstream use? The paper hints at this in the Limitations section when discussing RL-based window size control, but the conceptual move—treating intermediate representational states as useful even when inexact—is the deeper contribution.

Innovation 2: The First Systematic Evidence That LLMs Can Extract Correct Answers From Partial, Noisy Chain-of-Thought Rationales

While the corrupting rationale experiment (Section 5.2) is presented as a motivating observation, it constitutes a standalone empirical finding that challenges implicit assumptions in the CoT literature about why and how rationales work. Prior work on chain-of-thought prompting—from the original Wei et al. (2022) paper through extensions like Self-Consistency (Wang et al., 2022), Tree-of-Thought (Long, 2023), and rationale distillation (Magister et al., 2022; Hsieh et al., 2023)—treats the rationale as a logically necessary intermediate computation. The implicit model is that each reasoning step is causally required for the next, and that disrupting any step would cascade into an incorrect answer. The corrupting rationale experiment falsifies the strongest version of this assumption.

The experiment's design is simple but revealing: take a complete, correct rationale (generated by the model itself via autoregressive decoding), randomly mask up to 60% of its tokens, and ask the model to answer from the corrupted version. The result—saturated accuracy with only 40% of tokens revealed on StrategyQA—suggests that CoT rationales contain substantial redundancy, and that the model's answer-extraction mechanism does not require step-by-step logical verification of the reasoning chain. Instead, the rationale appears to function as a structured prompt extension that biases the model's internal representations toward a region of embedding space where the correct answer token becomes the most probable continuation. Some tokens in the rationale carry most of this biasing signal; others are filler or redundant restatements.

This finding is significant not because it shows rationales are unimportant—the model clearly needs some rationale context to perform well—but because it quantifies the tolerance to corruption and demonstrates that the tolerance is task-dependent (the saturation point varies across CSQA, StrategyQA, and AQuA in Figure 5). Prior work had not systematically measured how much of a rationale could be removed or corrupted before accuracy degrades. The paper does not claim this as a universal property—it acknowledges that tasks requiring precise intermediate calculations (multi-digit arithmetic, symbolic manipulation) might show very different corruption tolerance—but for the commonsense and strategy reasoning tasks tested, the redundancy is substantial.

Methodologically, this finding justifies the entire FastCoT approach: if the model can tolerate random masking of 60% of rationale tokens, it can plausibly tolerate the structured noise introduced by Jacobi decoding's approximate tokens (which are not random but are systematically related to the true rationale through iterative refinement). The corrupting rationale experiment provides the necessary condition for FastCoT to work; the main experiments (Figure 6) provide the sufficiency demonstration by showing that the approximate tokens actually do enable answer extraction with competitive accuracy.

Innovation 3: Demonstrating That Speed-Accuracy Tradeoffs in CoT Reasoning Can Be Controlled Through Decoding Strategy Choice Rather Than Prompt Engineering

The XoT literature has explored many ways to improve CoT accuracy (more elaborate prompts, multi-path reasoning, self-consistency, verification), and the decoding acceleration literature has explored many ways to reduce generation latency (speculative decoding, non-autoregressive models, KV-cache optimization). But prior to FastCoT, no work had demonstrated that the decoding algorithm itself could be used as a knob to control the speed-accuracy tradeoff for reasoning tasks. The assumption—implicit across both literatures—was that decoding strategy was a systems-level concern orthogonal to task accuracy: you pick a decoding algorithm (autoregressive, beam search, speculative), and that algorithm either preserves exact output (lossless methods) or degrades accuracy in some uncontrolled way (lossy methods like temperature sampling). FastCoT shows that a lossy decoding strategy—Jacobi decoding with early stopping and approximate rationale context—can produce a controlled, predictable speed-accuracy Pareto frontier where the accuracy loss is bounded and calibratable.

This is a different kind of contribution than proposing a new prompt or a new model architecture. It is a systems-level insight about the structure of the CoT inference problem: the rationale generation and answer extraction phases are not equally sensitive to token-level accuracy. The paper's iteration-stop mechanism (calibrated on a subset of the dataset to achieve a target performance loss threshold) operationalizes this insight into a deployable strategy. The fact that Figure 6 shows FastCoT converging to essentially full CoT accuracy after enough iterations—but reaching near-peak accuracy substantially earlier—demonstrates that the early iterations' approximate tokens already contain most of the answer-relevant information.

The practical significance is substantial: if one can reduce inference time by 20% (e.g., 326.93s vs. 365.20s on Llama2-13B with CSQA) for <3% accuracy loss, this changes the deployment economics of CoT reasoning at scale. The paper's exhaustive wall-clock timing breakdown (Table 2) shows that the speedup is not merely theoretical—it survives the overhead of KV-cache management, padding, and verification that Jacobi decoding introduces. This distinguishes FastCoT from methods that show FLOPs reductions but fail to translate them into wall-clock gains due to implementation overhead.

The limitation is that this speed-accuracy control mechanism depends on the specific model-dataset pair—the calibration subset approach requires running FastCoT to convergence on a representative sample to determine the iteration budget. This is feasible for batch processing but less so for interactive settings with diverse, unpredictable queries. The paper acknowledges this implicitly by noting that RL-based dynamic window control is future work.

Innovation 4: The Counterintuitive Finding That Adding More Tokens to a Forward Pass Doesn't Increase Latency—and Why This Matters Architecturally

Figure 2 is arguably the paper's most practically significant single result, even though it appears in the Preliminary section. It demonstrates that for Llama2-13B on an A100 GPU, increasing the Jacobi context window size from 0 (pure autoregressive, generating 1 new token) to 38 (generating 39 new tokens) adds essentially zero to the forward-pass latency. The forward time hovers around 35–45ms across the entire range, with no visible upward trend.

This is a counterintuitive empirical finding—most practitioners would expect that computing predictions for 39 positions costs substantially more than computing for 1 position—and the paper provides a clear causal explanation rooted in GPU microarchitecture: when the decoding length is short relative to the prompt length, the additional matrix multiplications for the extra positions are too small to saturate the GPU's compute units. The forward pass is bottlenecked by operations over the prompt tokens and model weights (which are shared regardless of how many new positions are predicted), not by the incremental cost of the new positions.

Why is this a conceptual innovation rather than just a convenient hardware fact? Because it reveals a structural opportunity that the decoding acceleration literature had not fully exploited. Prior speculative decoding methods rely on a separate draft model to propose multiple tokens, introducing model-management complexity and memory overhead. Jacobi decoding's original formulation treats the context window as a necessary cost of parallel verification—it must be paid to get the convergence speedup. FastCoT shows that for reasoning tasks (where prompts are long and generation lengths are moderate), the context window is essentially free in terms of per-iteration latency, meaning that the approximate future tokens come at zero marginal computational cost. This transforms the tradeoff analysis: it's not "spend more compute per iteration to get more tokens"—it's "get more tokens per iteration for the same compute, and some of those tokens provide useful information even if they're inexact."

The finding is hardware-dependent (different GPU architectures may show different saturation points) and prompt-length-dependent (very short prompts would make the decoding portion a larger fraction of total computation). But for the deployment scenario the paper targets—long CoT prompts feeding moderate-length reasoning chains—the near-zero marginal cost of the Jacobi window is what makes FastCoT's approach viable. Without Figure 2, the entire method would be suspect: "you're computing c+1 positions per forward pass, surely that's c+1 times more expensive?" The paper's explicit measurement and explanation of why this intuition fails is a contribution that future work can build on when designing parallel-decoding strategies for other long-context applications.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three reasoning benchmarks: CSQA (CommonsenseQA; Talmor et al., 2018) — a commonsense reasoning dataset with multiple-choice questions requiring background world knowledge; StrategyQA (Geva et al., 2021) — a dataset requiring implicit multi-step reasoning strategies where questions are answerable with yes/no but demand inferential chains; and AQuA (Ling et al., 2017) — an algebraic word problem dataset requiring mathematical reasoning to produce numeric or multiple-choice answers. All experiments use the official test split for each dataset. The paper does not report exact test-set sizes, but the CSQA test set contains 1,221 questions, StrategyQA contains 490 questions (the test split from the original paper), and AQuA contains 254 test questions. These datasets collectively span commonsense, strategic, and mathematical reasoning, providing coverage across reasoning types.
Base model(s). Experiments run on four LLaMA-family models: Llama2-13B, Llama2-7B, Llama-13B, and Llama-7B (Touvron et al., 2023b). The authors do not explicitly justify this choice beyond the models being publicly available and representing a range of scales (7B to 13B parameters across two generations of the LLaMA architecture). The selection of only decoder-only causal transformer models is inherent to the method — FastCoT relies on causal attention masking for its parallel decoding, which is the standard architecture for autoregressive language models. The absence of models larger than 13B or from other families (e.g., Mistral, PaLM, GPT) is a limitation discussed in the Critical Assessment.
Metrics. Performance is evaluated along two axes: accuracy and efficiency. Accuracy is measured as the proportion of questions for which the final extracted answer matches the ground truth — a standard exact-match metric for multiple-choice and yes/no questions. The paper reports this as a percentage. Efficiency is measured using two complementary metrics: wall-clock time (total seconds to process the full test set, measured end-to-end including all overhead) and iteration count (the number of forward passes through the LLM required to complete generation). The wall-clock measurement environment is an A100-SXM 80GB GPU with a 32-core CPU and 64 GiB host memory. Wall-clock time is the primary efficiency metric because it captures all practical overhead (KV-cache operations, padding, verification) that FLOPs-based accounting would miss. The paper also reports Performance Loss (PL), defined as the absolute percentage-point difference in accuracy between FastCoT and Vanilla CoT (autoregressive decoding). Finally, the IS/TI metric (Iterations Saved / Total Iterations) quantifies how many forward passes were avoided compared to pure autoregressive decoding.
Baselines. The paper defines four comparison points:
1. Vanilla CoT (AR) — standard chain-of-thought prompting with autoregressive greedy decoding. This is the accuracy ceiling and the efficiency floor. It generates the complete rationale token-by-token and extracts the answer from the full rationale.
2. FastCoT (w/o by-products) — a strong ablation that truncates the autoregressive-generated rationale to the same number of exact tokens that FastCoT produces at each iteration, then extracts the answer using the same answer trigger. Critically, this baseline has no approximate tokens — it sees only the verified exact prefix of the rationale. Comparing FastCoT against this baseline isolates the contribution of the approximate glimpse-of-future tokens: if FastCoT outperforms it, the approximate tokens are providing useful information beyond what the exact prefix alone contains.
3. FastCoT — the full method with both exact tokens (from verified Jacobi decoding) and approximate tokens (from the Jacobi context window) fed into answer extraction.
4. Non-CoT — direct answering without any chain-of-thought rationale. This represents the accuracy floor (no reasoning support) and is shown as a horizontal reference line in Figure 6. It is only relevant for establishing that CoT reasoning provides a substantial accuracy benefit worth preserving.
Notably absent from the baselines are: (a) speculative decoding with a draft model (which would be a competing accuracy-lossless acceleration method), (b) standard Jacobi decoding run to full convergence (which would show the upper bound of what Jacobi decoding can achieve while remaining lossless), and (c) simple rationale truncation at varying lengths without the Jacobi iterative refinement (which would test whether the iterative refinement process is necessary or whether any partial rationale would work).
Generation budget / compute accounting. The paper's primary unit of compute is the forward pass (one call to the LLM). For autoregressive decoding, one forward pass produces exactly one new token. For Jacobi decoding, one forward pass produces c+1 tokens (one exact + c approximate), where c is the context window size. The wall-clock time accounts for all forward passes plus auxiliary operations (KV-cache padding, verification, context decoding, stripping). The context window size c is held fixed during each experiment; the paper does not precisely specify its value for the main results, but the ablation in Figure 8 sweeps from 0 to 25, and the practical range is implied to be modest (given the hardware analysis showing no latency increase up to c=38). The forward pass cost is shown to be independent of c for the tested ranges (Figure 2), so the budget accounting is effectively "number of LLM forward calls," with each call taking approximately constant time regardless of how many Jacobi window positions are computed.
Cross-validation / statistical protocol. The paper does not use k-fold cross-validation or statistical significance testing. The test sets are processed in their entirety, and accuracy is reported as a single percentage over the full test split. For the iteration budget calibration (Stop Condition 1 in Section 4.4), the paper "randomly select[s] a small portion of [the dataset] as S_cal" and computes the minimum iterations needed to stay within a performance loss threshold. This calibration subset is separate from the test set, but the paper does not specify the calibration subset size, the random seed, or whether multiple calibration runs were averaged. The corruption experiment (Figure 5) averages over 100 random seeds per corruption ratio, which is a reasonable protocol for that diagnostic probe, but this protocol is not extended to the main results. The absence of confidence intervals or error bars on accuracy and timing measurements is a weakness discussed in the Critical Assessment.

Main Quantitative Results

The Corrupting Rationale Experiment (Justifying the Approach)

The paper opens its experimental section not with FastCoT results but with the corrupting rationale experiment (Section 5.2, Figure 5), which serves as the empirical justification for the entire method. The headline finding:

"when only 40% of the rationale generated by autoregressive decoding is revealed in StrategyQA dataset, the performance is already saturated."

Figure 5 plots normalized performance (accuracy with corrupted rationale divided by accuracy with full rationale) against the overlap ratio (fraction of original rationale tokens preserved, i.e., 1 - corruption_ratio). Across all three datasets (CSQA, StrategyQA, AQuA), the curves rise sharply from 0 to roughly 0.4 overlap ratio and then flatten — additional rationale tokens beyond ~40% provide diminishing returns. The saturation point varies slightly by dataset: StrategyQA saturates earliest (around 0.4), while CSQA and AQuA continue to show modest gains up to roughly 0.6-0.8 overlap. This dataset-dependent saturation is important because it predicts that FastCoT's effectiveness should vary by reasoning type — the paper does not explicitly draw this connection, but it follows from the logic.

The experiment uses oracle rationales generated by the model itself via autoregressive decoding. The corruption process masks tokens using two patterns applied with equal probability: (1) sequential masking from the end of the rationale (simulating truncation / partial generation), and (2) random uniform masking across the rationale (simulating the scattered errors Jacobi decoding's approximate tokens might introduce). The paper runs 100 random seeds per corruption ratio per question, producing a distribution over corrupted-rationale accuracies.

What this demonstrates: The model can tolerate substantial noise in intermediate reasoning context without losing answer accuracy. Rationale tokens are not equally informative — some (likely keywords, entities, and structural markers) carry most of the signal. The remaining tokens are redundant or stylistic. This is a necessary condition for FastCoT to work: if even oracle rationales corrupted at 40% overlap caused accuracy to collapse, there would be no hope that Jacobi decoding's approximate tokens (which are not subsets of the true rationale but rather iteratively refined approximations) could support accurate answer extraction.

What this does NOT demonstrate: The experiment uses oracle rationales (correct, complete reasoning chains produced by the model itself) with tokens artificially replaced by [PAD]. This is not the same as Jacobi decoding's approximate tokens, which are the model's best guesses at future positions given partial context. Jacobi tokens are systematically related to the true rationale (they come from the same model distribution), but they are not a random subset of it — they may contain confabulations, inconsistent entities, or logical contradictions that [PAD] masking does not introduce. The corrupting rationale experiment is therefore an upper bound on how well partial context can work; FastCoT's actual performance will be lower if its approximate tokens are systematically worse than randomly masked oracle tokens.

Accuracy vs. Iteration Curves: Does the "Glimpse of Future" Help?

The paper's central experimental result appears in Figure 6, a grid of 8 subplots showing accuracy vs. iteration count for each model-dataset pair. Each subplot contains four curves:

FastCoT (blue): the full method with exact + approximate tokens.
FastCoT (w/o by-products) (orange): exact tokens only, no approximate tokens.
CoT (green dashed horizontal line): Vanilla CoT accuracy ceiling (autoregressive, full rationale).
Non-CoT (red dashed horizontal line): accuracy without any reasoning rationale.

Additionally, two vertical dashed lines mark the "converge points" — the iteration at which FastCoT and FastCoT (w/o by-products) reach their asymptotic accuracy (operationally, the point beyond which additional iterations produce negligible improvement).

Headline finding: FastCoT consistently reaches near-ceiling accuracy substantially before the full rationale would be generated, and in most configurations, FastCoT's curve lies above FastCoT (w/o by-products), demonstrating that the approximate glimpse-of-future tokens contribute signal beyond what the exact prefix alone provides.

Model-by-model patterns (from Figure 6):

Llama2-13B on CSQA (top-left): FastCoT rises from ~30% at iteration 0 to plateau around 58-60% by iteration 40-50 (well below the ~62% CoT ceiling, a 2-4 percentage point gap). FastCoT (w/o by-products) tracks slightly below FastCoT through most iterations, with the gap widening in early iterations and narrowing later. Both converge around iteration 60-80. The performance loss is approximately 1.27% per Table 1.

Llama2-13B on StrategyQA (top-right): An unusual pattern emerges: both FastCoT and FastCoT (w/o by-products) show non-monotonic behavior — accuracy initially rises, then declines in the iteration range ~20-60 before recovering. The CoT ceiling is ~67%. FastCoT peaks around 65-66% (1-2 points below ceiling) near iteration 100. The authors attribute this non-monotonicity to "the faithfulness of CoT" (Lanham et al., 2023; Radhakrishnan et al., 2023) — meaning the model's generated rationales may not always faithfully reflect the reasoning that leads to the correct answer. Adding more exact rationale tokens may sometimes introduce misleading context. FastCoT outperforms FastCoT (w/o by-products) through most iterations, with a gap of 2-5 percentage points in the mid-range.

Llama2-7B on both CSQA and StrategyQA (middle row): FastCoT consistently outperforms FastCoT (w/o by-products) throughout the iteration range. On CSQA, the gap is 3-8 percentage points in early iterations, narrowing as both approaches approach the ~53% CoT ceiling (itself lower than Llama2-13B's ~62%, reflecting the 7B model's weaker reasoning). On StrategyQA, the gap is 2-6 points. FastCoT converges around iteration 60-80 while the w/o by-products baseline takes longer to reach the same accuracy.

Llama-13B and Llama-7B (bottom two rows): The pattern holds but with interesting differences. For Llama-13B on CSQA, FastCoT substantially outperforms the w/o by-products baseline by 5-10 points in early and mid iterations. For Llama-13B on StrategyQA, the CoT ceiling is only ~60% (lower than Llama2-13B's ~67%, reflecting architectural differences between Llama generations), and FastCoT converges to near-ceiling accuracy faster than the w/o baseline. For Llama-7B on both datasets, accuracy is lower overall (CoT ceilings of ~22% on CSQA and ~32% on StrategyQA), and FastCoT still maintains an advantage over the w/o baseline.

Key cross-model observations:

The approximate tokens' benefit is model-dependent. For Llama2 models, the FastCoT vs. FastCoT (w/o by-products) gap is smaller and sometimes disappears in later iterations (e.g., Llama2-13B on StrategyQA shows the curves overlapping after iteration 80). For Llama1 models, the gap is larger and more persistent. The paper does not explain this discrepancy, but it may relate to architectural differences between Llama and Llama2 (different training data, different tokenizers, different attention mechanisms).
Convergence speed varies by model and dataset. Llama2-13B converges fastest (40-60 iterations to plateau), while Llama-7B requires 60-100 iterations. StrategyQA generally requires more iterations than CSQA for equivalent convergence, consistent with StrategyQA's more complex multi-step reasoning.
The accuracy ceiling gap. FastCoT never fully reaches the Vanilla CoT accuracy ceiling. The gap is quantified in Table 1's PL column: ranging from 1.23% (Llama-7B on CSQA, best case) to 2.66% (Llama2-13B on AQuA, worst case among the reported main results). This gap is the price of the speedup — the question is whether the speed-accuracy tradeoff is acceptable for the application.

Wall-Clock Time Results: The Practical Speedup

Table 1 reports the core efficiency numbers. For each model-dataset pair, it gives:

FastCoT Time: total wall-clock seconds for the full method on the test set.
CoT Time: total wall-clock seconds for Vanilla CoT (autoregressive decoding) on the same test set.
Save Time: the absolute reduction in seconds (CoT Time minus FastCoT Time).
Time Ratio: the percentage reduction (Save Time / CoT Time).
PL (Performance Loss): absolute accuracy percentage-point loss compared to Vanilla CoT.
IS/TI: Iterations Saved vs. Total Iterations (how many forward passes were avoided).

Headline results (from Table 1):

Llama2-13B on CSQA: 326.93s (FastCoT) vs. 365.20s (CoT). Time ratio: 10.47% reduction. PL: 1.27%. IS/TI: 896/14,684 (6.1% of forward passes saved).
Llama2-13B on AQuA: 402.90s vs. 502.38s. Time ratio: 19.80% reduction. PL: 2.66%. IS/TI: 1,742/18,640 (9.3% of forward passes saved).
Llama2-13B on StrategyQA: 313.43s vs. 330.38s. Time ratio: 5.13% reduction. PL: 1.47%. IS/TI: 906/13,992 (6.5% saved).
Llama2-7B on CSQA: 184.98s vs. 242.47s. Time ratio: 23.71% reduction. PL: 1.50%. IS/TI: 877/12,866 (6.8% saved).
Llama2-7B on AQuA: 228.04s vs. 257.46s. Time ratio: 11.42% reduction. PL: 2.24%. IS/TI: 1,400/15,358 (9.1% saved).
Llama2-7B on StrategyQA: 233.19s vs. 256.05s. Time ratio: 8.92% reduction. PL: 2.21%. IS/TI: 1,163/8,958 (13.0% saved).
Llama-13B on CSQA: 366.15s vs. 420.59s. Time ratio: 13.00% reduction. PL: 1.84%. IS/TI: 986/16,904 (5.8% saved).
Llama-13B on AQuA: 460.99s vs. 589.44s. Time ratio: 21.80% reduction. This is the single largest speedup reported. PL: 2.50%. IS/TI: 1,773/21,674 (8.2% saved).
Llama-13B on StrategyQA: 293.38s vs. 372.71s. Time ratio: 21.29% reduction. PL: 1.67%. IS/TI: 959/13,036 (7.4% saved).
Llama-7B on CSQA: 286.30s vs. 297.13s. Time ratio: 3.64% reduction. PL: 1.23%. IS/TI: 1,216/18,662 (6.5% saved).
Llama-7B on AQuA: 348.87s vs. 388.48s. Time ratio: 10.20% reduction. PL: 2.24%. IS/TI: 1,807/23,126 (7.8% saved).
Llama-7B on StrategyQA: 293.72s vs. 303.05s. Time ratio: 3.20% reduction. PL: 2.01%. IS/TI: 1,069/19,216 (5.6% saved).

Patterns in the timing data:

AQuA shows the largest relative speedups (19.80%, 11.42%, 21.80%, 10.20% across models). This dataset involves algebraic word problems, which likely produce longer rationales with more mathematical notation — meaning more tokens total and more opportunity for the Jacobi window to capture multiple verified tokens per iteration.
The speedup is consistently larger for 13B models than 7B models (e.g., Llama2-13B on CSQA: 10.47% vs. Llama2-7B: 23.71% — wait, this is actually the opposite pattern — the 7B model shows larger relative speedup on CSQA. Let me recheck: Llama2-7B on CSQA shows 23.71%, which is indeed larger than Llama2-13B's 10.47%. But on AQuA, Llama-13B shows 21.80% vs. Llama2-7B's 11.42%). The pattern is not cleanly parameter-count-dependent, suggesting dataset-specific interaction effects with model scale.
Performance loss is consistently modest: 1.23% to 2.66% absolute across all configurations. No configuration exceeds 3% accuracy loss, consistent with the paper's claim of "negligible performance drop."
The IS/TI ratio reveals that Jacobi decoding's multi-token verification saves 5.6% to 13.0% of forward passes. This is the "free" speedup — tokens produced and verified in the same forward pass that autoregressive decoding would have required separate passes for. However, this alone doesn't explain the full wall-clock speedup (which can be >20%). The remainder comes from early stopping: terminating the loop before the full rationale would be generated autoregressively, saving entire forward passes at the end of generation.

Time Composition: Where Does the Time Go?

Table 2 breaks down the 326.93s FastCoT runtime for Llama2-13B on CSQA into its constituent parts and compares against the 365.20s autoregressive baseline:

Time Type	FastCoT	AR
Inference	274.08s	358.15s
Type1 Padding	5.33s	0s
Type2 Padding	0.84s	0s
Decode	1.20s	3.06s
Context Decode	5.42s	0s
Strip KV	18.58s	0s
Other	21.48s	3.99s
Total	326.93s	365.20s

The core GPU inference time drops dramatically: 274.08s (FastCoT) vs. 358.15s (AR) — a 23.5% reduction in time spent on model forward passes. This is the "gross" speedup before overhead. The forward-pass savings come from two sources: (1) Jacobi decoding verifying multiple tokens per iteration (reducing the total number of forward passes), and (2) early stopping (not generating the tail of the rationale).

The overheads eat into the gross speedup:

Strip KV (18.58s, 5.7% of total): The largest single overhead. After each forward pass, the KV cache must be trimmed to retain only the exact-token portion, discarding approximate-token entries that will be recomputed in the next iteration. This operation is unique to Jacobi-decoding-based methods and has no counterpart in autoregressive decoding.
Context Decode (5.42s, 1.7% of total): Computing the approximate tokens in the context window after inference. This is the incremental cost of the Jacobi window positions — not the forward pass itself (which Figure 2 showed is nearly free), but the post-processing to extract and manage those approximate token predictions.
Type1 Padding (5.33s, 1.6% of total): Handling disparities in KV-cache lengths across batch instances. The paper notes that different questions reach different exact-token counts at each iteration, creating uneven cache lengths that must be padded for batched tensor operations.
Type2 Padding (0.84s, 0.3% of total): Handling disparities in Jacobi window token counts.
Decode (1.20s vs. AR's 3.06s): Token ID-to-text conversion overhead. FastCoT's is lower because it generates fewer total tokens (due to early stopping).
Other (21.48s vs. AR's 3.99s): Unspecified overhead that is substantially larger for FastCoT. The paper does not decompose this category, but it likely includes verification logic, ATB management, iteration control, and the final answer-trigger forward pass.

The net speedup (326.93s vs. 365.20s, 10.47%) is the gross inference speedup (23.5%) minus these overheads. The paper's transparency about this breakdown is a strength — it shows that Jacobi decoding's theoretical advantage is partially consumed by bookkeeping costs that are implementation-specific and could potentially be reduced with more optimized systems engineering.

Practical implication: The Strip KV overhead (5.7%) is the largest target for further optimization. If KV-cache operations could be parallelized with inference or made more efficient, the net speedup could approach the gross inference speedup of 23.5%. The paper does not explore this optimization, but identifying the bottleneck is valuable for future work.

Ablation Studies and Robustness Checks

The paper includes several analyses that probe the sensitivity and robustness of the method, though none are structured as formal ablation studies with controlled variable isolation:

Context window size vs. iteration and accuracy (Figure 8): This is the paper's most detailed ablation, measuring accuracy at multiple iteration counts for context window sizes c = 0, 5, 10, 15, 20, 25 on Llama2-13B with CSQA. The key findings:

Early iterations (0-20): Accuracy is inversely related to window size. c=0 (no approximate tokens) performs best; c=25 performs worst. The paper attributes this to low-quality approximate tokens in early iterations diluting the signal from exact tokens. The model sees more noise when the window is larger because the Jacobi iterative solutions haven't yet converged to meaningful content.
Mid iterations (10-65): The relationship reverses. Larger windows (c ≥ 15) outperform smaller windows. By iteration 20, c=5 through c=25 all outperform c=0. The approximate tokens have improved in quality (the fixed-point iteration has made progress) and now provide useful "glimpse of future" signal.
Late iterations (65+): The curves converge. By iteration 65, all window sizes from c=5 to c=25 cluster within ~2 percentage points of each other. The exact-token prefix has grown long enough that the approximate tokens contribute diminishing additional value.
Window sizes above 20 show no further benefit: The curves for c=20 and c=25 are essentially identical at all iteration counts, suggesting a saturation point where additional approximate tokens provide redundant or irrelevant information.

This finding is non-obvious and practically significant: It demonstrates that the optimal context window size is iteration-dependent. A fixed window size throughout the entire process (as the paper uses in its main experiments) is suboptimal — a dynamic strategy that starts with small windows and grows them as iterations progress could capture the best of both regimes (avoiding early noise while benefiting from later signal). The paper acknowledges this in the Limitations section but does not implement it.

The c=0 comparison (in Figure 8) is essentially a re-run of the FastCoT (w/o by-products) baseline and confirms that approximate tokens provide value — at iteration 30, c=20 shows roughly 58% accuracy vs. c=0 at roughly 54%, a 4-point gap attributable purely to the approximate tokens.

ATB initialization strategy: The paper states that initializing the Approximate Tokens Buffer with the tokenized question itself (rather than [PAD] tokens) "would provide a more diverse initial solution for early stage's Jacobi decoding." This claim is based on "empirical results" but no explicit ablation comparing question-initialization vs. PAD-initialization is shown. The absence of this comparison is a notable gap — it's a design choice that affects early-iteration token quality and could influence the context-window-size tradeoff.

Answer trigger design: The paper uses "So the answer is" as the answer trigger string but does not ablate this choice. Alternative triggers (e.g., "Therefore", "The answer is", "Answer:", or simply EOS) could produce different accuracy results. The choice is justified by consistency with few-shot CoT prompt templates (where similar phrasing naturally appears), but no empirical validation is reported. This is a mild concern — the trigger's effectiveness likely depends on whether the phrasing was present in the model's training data, and different models may have different sensitivity to trigger wording.

Two-fold corruption pattern (in the corrupting rationale experiment): The paper uses two corruption patterns (sequential-from-end and random-uniform) with equal probability. The results in Figure 5 aggregate over both patterns, so we cannot see whether one pattern is more damaging than the other. This matters because Jacobi decoding's approximate tokens are more structured than random masking — they are the model's own predictions, which may cluster errors in certain token types (e.g., named entities, numbers) rather than distributing them uniformly. If the model is more tolerant of random corruption than systematic error, the corrupting rationale experiment overestimates FastCoT's viability.

Batch processing and padding mechanisms (Figure 7): The two padding schemes are evaluated for their time cost (Table 2), but there is no ablation comparing single-instance vs. batched processing. Batched inference introduces the padding overheads that account for ~6.17s (Type1 + Type2) for Llama2-13B on CSQA. If FastCoT were run on a single instance at a time, these overheads would disappear, but throughput would drop. The paper does not explore this tradeoff.

Model generation effects not isolated: The accuracy-vs-iteration curves (Figure 6) conflate two effects: (1) the benefit of having more exact rationale tokens in the context, and (2) the benefit of seeing approximate future tokens. The FastCoT (w/o by-products) baseline isolates effect (1), but there is no baseline that isolates effect (2) in the absence of additional exact tokens — for instance, adding noisy tokens that are not Jacobi-generated (random tokens, repeated question tokens, etc.) to see whether any future-looking context helps or whether it must be specifically Jacobi-decoded approximate tokens.

Negative results that are acknowledged but not quantified: The Llama2-13B on StrategyQA curve (Figure 6, top-right) shows a non-monotonic accuracy decline in the iteration range ~20-60, where adding more exact rationale tokens reduces accuracy. This is a genuine negative result — it suggests that for some model-dataset pairs, the CoT rationales contain misleading or unfaithful reasoning steps that harm answer extraction. The paper attributes this to "the faithfulness of CoT" and cites relevant literature (Lanham et al., 2023; Radhakrishnan et al., 2023), but does not quantify the effect magnitude or investigate whether the approximate tokens exacerbate or mitigate it. This is a finding that deserves more attention than the paper gives it, because it implies a boundary condition: FastCoT's usefulness may be limited when the model's own reasoning is systematically unfaithful.

Critical Assessment

Does the paper demonstrate that FastCoT accelerates CoT inference with negligible performance loss?

Supported, with qualifications about the definition of "negligible." The wall-clock results (Table 1) unambiguously show speedups: 3.20% to 23.71% across configurations. The performance loss ranges from 1.23% to 2.66% absolute, which the paper characterizes as "negligible." Whether this is genuinely negligible depends on the application: for high-stakes reasoning where every percentage point matters (medical diagnosis, legal analysis), even 2.66% may be unacceptable. For throughput-oriented applications where serving cost is the primary concern, trading 2.5% accuracy for 21.8% speedup (as on Llama-13B with AQuA) is clearly favorable. The paper does not provide guidance on this tradeoff or discuss application-dependent acceptability thresholds.

The "negligible" claim is further qualified by the observation that FastCoT does not always outperform its own ablation (FastCoT w/o by-products) at all iteration counts. On Llama2-13B with StrategyQA (Figure 6, top-right), the blue and orange curves cross multiple times, and in the iteration range 20-40, FastCoT (w/o by-products) actually achieves higher accuracy. This means the approximate tokens are not uniformly beneficial — they can sometimes harm accuracy, likely when the Jacobi iterative solutions contain misleading "glimpse of future" tokens that steer the model toward wrong answers. The paper acknowledges this only obliquely via the "faithfulness" discussion but does not characterize when or why the approximate tokens become counterproductive.

Does the paper demonstrate that the "glimpse of future" concept (approximate tokens) specifically drives the speedup?

Partially. The FastCoT vs. FastCoT (w/o by-products) comparison in Figure 6 shows that the approximate tokens contribute to accuracy — FastCoT typically reaches higher accuracy at the same iteration count than the w/o baseline. This is the "glimpse of future" proving its value for accuracy. However, the speedup (Table 1) comes primarily from the Jacobi decoding mechanism itself (verifying multiple tokens per iteration, IS/TI) and from early stopping — not directly from the approximate tokens. The approximate tokens enable more aggressive early stopping by compensating for the missing exact rationale tokens, but the paper does not decompose the speedup into "savings from multi-token verification" vs. "savings from approximate-token-enabled early stopping." This decomposition would clarify how much of the 20% speedup could be achieved by standard Jacobi decoding (accuracy-lossless) vs. how much requires the FastCoT-specific approximate-token retention.

The corrupting rationale experiment (Figure 5) shows that partial rationales can work, but does not directly test the Jacobi approximate tokens. The corruption uses oracle rationales with [PAD] masking; Jacobi approximate tokens are different in kind (model predictions, not masked ground truth). An experiment that compared accuracy using (a) oracle rationale truncated to match FastCoT's exact-token count, (b) oracle rationale with random masking matching FastCoT's total token count, and (c) actual FastCoT approximate tokens, would more precisely characterize how much signal the approximate tokens carry relative to an idealized partial-rationale baseline. This experiment is absent.

Does the paper support its claim of being "model-agnostic"?

Supported within the tested scope, but the scope is narrow. The paper tests four models, all from the LLaMA family (Llama-7B, Llama-13B, Llama2-7B, Llama2-13B). All are decoder-only causal transformers with similar architectures. The method relies on causal attention masking and standard autoregressive token prediction — properties that indeed generalize to most modern LLMs — but the paper has not demonstrated FastCoT on encoder-decoder architectures (T5, BART), mixture-of-experts models (Mixtral), or models with significantly different tokenizers or context-length characteristics. The huggingface implementation approach is described as requiring "almost no modification," which is plausible for any model using the standard transformers library, but this claim is not empirically validated beyond the four tested models.

The datasets are all English-language reasoning tasks with relatively short rationales. CommonsenseQA, StrategyQA, and AQuA are multiple-choice or yes/no tasks where rationales are typically 20-100 tokens — short enough that early stopping saves a limited number of forward passes. The method's behavior on tasks requiring very long rationales (multi-paragraph mathematical proofs, code generation with explanation, multi-step planning) is unknown. For very long rationales, the token savings from early stopping would be proportionally larger, potentially yielding greater speedups — but the approximate token quality might also degrade if the Jacobi window cannot capture long-range dependencies.

What experiments would have strengthened the paper?

A direct comparison with speculative decoding. Since speculative decoding is the dominant accuracy-lossless acceleration method, showing how FastCoT's speed-accuracy tradeoff compares would situate it in the literature. Speculative decoding would preserve full accuracy but likely show smaller speedups (since it still generates the complete rationale). The comparison would answer: "Is it better to generate the full rationale faster (speculative) or to generate a partial rationale with approximate future context (FastCoT)?"
A FLOPs or energy consumption measurement. Wall-clock time on a specific GPU configuration is implementation-dependent. FLOPs or GPU energy consumption would provide a hardware-agnostic efficiency metric and reveal whether FastCoT's overhead (Strip KV, padding) represents wasted computation or merely unavoidable bookkeeping.
Per-question accuracy breakdown by rationale length. If FastCoT's speedup comes from stopping early, the accuracy loss should be concentrated on questions requiring long rationales (where early stopping truncates more of the reasoning). Conversely, questions with short rationales might show no accuracy loss at all. A scatter plot of per-question accuracy difference vs. full-rationale length would characterize this relationship and help practitioners predict when FastCoT is appropriate.
Ablation on the answer trigger. Testing multiple trigger strings would establish whether FastCoT's performance is sensitive to this design choice or robust to variation.
Statistical significance. With test sets of 254-1,221 questions, a 1-3% accuracy difference may or may not be statistically significant. Confidence intervals on the accuracy numbers would allow readers to assess whether the observed gaps (FastCoT vs. CoT, FastCoT vs. FastCoT w/o by-products) are reliable or within sampling noise.
Dynamic context window experiment. The paper's own Figure 8 analysis strongly suggests that a window that starts small and grows would outperform any fixed window. Implementing and evaluating this would test the paper's understanding of its own method and could yield additional speedup or accuracy gains.

Genuine weaknesses that should be acknowledged:

Single GPU architecture (A100). The finding that context window size doesn't affect latency (Figure 2) is hardware-dependent. On GPUs with different memory bandwidth or tensor core configurations, the saturation point may differ, potentially reducing or eliminating the "free" approximate tokens. The paper does not test on other hardware.
No comparison with simple rationale truncation. A trivial baseline would be: run autoregressive CoT for a fixed number of tokens (matching FastCoT's exact-token count at termination), then extract the answer. This would show whether the Jacobi iterative process and approximate tokens provide any benefit over simply generating a shorter rationale. If truncation at the same token count performs similarly, then the entire Jacobi apparatus is unnecessary — early stopping alone explains the gains.
The calibration subset approach is under-specified. The paper says it uses "a small portion" of the dataset for calibration but doesn't report the size, the selection procedure, or how the performance loss threshold maps to the actual iteration budget. This makes the method difficult to reproduce without access to the code.
No evaluation on open-ended generation tasks. CoT reasoning with multiple-choice answers is a specific subset of LLM use cases. FastCoT's approach of stopping early and using approximate tokens to trigger answers may not transfer to tasks where the output must be a complete, fluent text passage (summarization, creative writing, dialogue). The paper does not discuss this boundary condition.
The "glimpse of future" is bounded by the Jacobi context window depth. For questions requiring reasoning steps that depend on information >20 tokens ahead, the approximate tokens will not capture the relevant future context, and FastCoT's advantage over truncation may disappear. The paper does not characterize this depth limit or relate it to actual rationale structure.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Not Accounted For in the Headline Speedup Numbers

The assumption or constraint: FastCoT's iteration-stop condition for large datasets requires a calibration phase: randomly select a subset S_cal from the dataset, run FastCoT to convergence on this subset, measure accuracy at each iteration, and compute the minimum iteration count that stays within a target performance loss threshold (Section 4.4, Stop Condition 1). This calibration cost is not included in any of the wall-clock time measurements reported in Table 1 or the speedup percentages.

The consequence: If a practitioner wants to deploy FastCoT on a new dataset or with a new model, they must first pay the calibration cost—running FastCoT to convergence on a representative sample—before applying the method. For datasets with small test splits (StrategyQA: 490 questions, AQuA: 254 questions), the calibration subset alone might consume a meaningful fraction of the total dataset processing time. If the calibration subset is, say, 20% of the dataset, and calibration runs to full convergence (which takes longer than the budgeted iterations), then the true end-to-end speedup over autoregressive decoding shrinks—potentially substantially. The paper does not provide any numbers for calibration cost or guidance on how large S_cal needs to be for reliable budget estimation. The headline 10-20% speedups should be understood as incremental savings after calibration, not total deployment savings from scratch.

What evidence exists in the paper: The calibration procedure is described qualitatively in Section 4.4: "For a large dataset, we randomly select a small portion of it as S_cal and perform the CoT task within this subset for each iteration." No size for S_cal is reported. No calibration wall-clock time is measured. No amortization analysis is provided (e.g., "calibration costs X seconds, which is amortized after Y test-set queries"). Table 1's timing numbers are explicitly for the test set only, with the iteration budget pre-determined.

Mitigation status: Not addressed. The paper does not discuss amortization, does not propose a method for reusing calibration results across related datasets, and does not assess whether a lightweight difficulty predictor (analogous to the approach in the example paper's Section 3.2) could eliminate the need for per-dataset calibration. The authors do not flag this as a limitation in Section 7, which is a notable omission given that the calibration cost could dominate the measured savings for small datasets or one-off inference tasks.

6.2 FastCoT Assumes White-Box Access to the Model and Modifies the Generation Loop

The assumption or constraint: FastCoT fundamentally works by intercepting and modifying the autoregressive generation process—replacing standard next-token prediction with parallel Jacobi decoding, managing a custom approximate-tokens buffer, and injecting a separate verification and cache-stripping step after each forward pass. As the authors explicitly state in Section 7 (Limitations):

"Since our method involves modifying the text generation process by the model, it cannot be applied to black-box large language models."

The consequence: FastCoT is inapplicable to any LLM accessed solely through a commercial API (OpenAI's GPT-4, Anthropic's Claude, Google's Gemini, Cohere's Command, etc.). These APIs expose only a text-in/text-out interface with no control over the internal decoding loop, KV-cache, or token-by-token verification logic. For the large and growing fraction of LLM usage that goes through such APIs—including most production deployments of reasoning tasks—FastCoT offers no benefit. This is not a minor scope limitation; it restricts the method to self-hosted open-weight models, which represent a shrinking fraction of practical LLM usage as API-based models improve in capability and convenience.

Furthermore, even for open-weight models, FastCoT requires sufficiently low-level framework access to implement Jacobi decoding's asymmetric attention masking, custom KV-cache management (the Strip KV and padding operations detailed in Table 2), and the verification loop. The paper's implementation "based on the huggingface implementations" (Section 5.1) suggests this is feasible for models in the transformers ecosystem, but models with custom inference stacks (vLLM, TensorRT-LLM, llama.cpp) may require substantial re-engineering to support FastCoT's generation loop modifications.

What evidence exists in the paper: Section 7 acknowledges this limitation directly. The method is tested on four LLaMA-family models, all self-hosted via huggingface. No API-based model is evaluated, nor could be. The implementation details in Section 4 (Figures 3, 4) and the time-composition breakdown (Table 2) reveal the depth of the integration required—Strip KV alone is 18.58 seconds of custom cache manipulation that does not exist in standard inference pipelines.

Mitigation status: Explicitly acknowledged but not mitigated. The paper frames this as an inherent constraint ("it cannot be applied to black-box large language models") rather than a problem to solve. No future work is proposed to adapt the concept to API-accessible models (e.g., by using truncated autoregressive output as a proxy for Jacobi approximate tokens, or by negotiating with API providers for intermediate-token access). This is a fundamental architectural limitation, not an implementation detail—FastCoT's core mechanism requires control over the decoding loop, which API-based deployment models deliberately withhold.

6.3 Evaluation Is Confined to a Single Model Family and Three English Reasoning Benchmarks

The assumption or constraint: All experiments use LLaMA-family models (Llama-7B, Llama-13B, Llama2-7B, Llama2-13B) and three English-language reasoning datasets (CSQA, StrategyQA, AQuA). These datasets share structural properties: all are multiple-choice or yes/no tasks with relatively short rationales (tens to low hundreds of tokens), all require commonsense or mathematical reasoning rather than factual recall or multi-document synthesis, and all produce rationales where the final answer appears after a predictable structural cue ("So the answer is...").

The consequence: Three distinct generalisation questions are left open by this narrow evaluation scope, each with practical deployment implications:

Architecture generalisation: LLaMA-family models share a specific decoder-only transformer design with rotary position embeddings (RoPE), SwiGLU activations, and a particular pretraining data mixture. Do encoder-decoder models (T5, Flan-T5) benefit from FastCoT? Do mixture-of-experts models (Mixtral) with different attention patterns show different Jacobi convergence behavior? Do models with different tokenizers produce approximate tokens of different quality? The paper provides no evidence. The claim of being "model-agnostic" (Abstract) is tested only within one architectural lineage.

Task generalisation: The three benchmarks test reasoning where the answer is a discrete choice (A/B/C/D or yes/no), the rationale follows a largely linear structure, and the "glimpse of future" from a short Jacobi window (c ≤ 25) can plausibly capture the relevant upcoming context. What happens on tasks with:

Long-horizon reasoning, such as multi-step mathematical proofs (where the final answer depends on steps 50+ tokens ahead, well beyond the tested window sizes)?
Open-ended generation tasks (summarization, creative writing, code generation) where there is no single "answer" to extract and where the full output—not just a final label—is the deliverable?
Factual recall tasks where rationale tokens are not redundant (as shown in Figure 5) but individually load-bearing (e.g., each step of a calculation, each retrieved fact in a multi-hop QA chain)?

The corrupting rationale experiment (Figure 5) shows that saturation point varies by dataset—StrategyQA saturates earlier than AQuA—hinting that the redundancy property FastCoT exploits is task-dependent. Tasks with low redundancy (where every rationale token carries unique, non-redundant information) would likely show larger accuracy degradation from early stopping and approximate-token substitution.

Language and cultural generalisation: All datasets are in English. CoT reasoning quality and rationale structure vary across languages (Shi et al., 2022; Lai et al., 2023), and the Jacobi convergence behavior—which depends on token-level prediction accuracy—may differ for languages where the base model has weaker next-token prediction capabilities.

What evidence exists in the paper: The paper tests four models across three datasets—12 model-dataset pairs total—all within the LLaMA English reasoning envelope. No experiments on other architectures, other languages, or non-reasoning tasks are reported. The authors do not claim generalisation beyond the tested scope, but they also do not flag the narrow evaluation as a limitation in Section 7, which is a notable omission.

Mitigation status: Not addressed. The paper does not discuss domain boundaries or propose validation protocols for new task types. A practitioner considering FastCoT for, say, medical reasoning with BioBERT-based models or code generation with CodeLlama would need to conduct their own full evaluation from scratch, including re-calibrating the iteration budget and verifying that the corrupting rationale property (Figure 5) holds for their task.

6.4 The KV-Cache Stripping Overhead Consumes Nearly 6% of Total Runtime, and This Overhead Does Not Diminish With More Iterations Saved

The assumption or constraint: FastCoT's generation loop requires, after every forward pass, a "Strip KV" operation that extracts only the exact-token portion of the key-value cache and discards approximate-token entries. This operation is unique to Jacobi-decoding-based methods (standard autoregressive decoding has no equivalent, since every generated token becomes part of the permanent cache). The paper measures this overhead at 18.58 seconds for Llama2-13B on CSQA—5.7% of total FastCoT runtime (326.93s) and the single largest non-inference cost component (Table 2).

The consequence: The Strip KV overhead is effectively a fixed per-iteration tax that scales with the number of iterations, not with the number of tokens generated. Even as Jacobi decoding saves iterations (the IS/TI metric in Table 1 shows 5.6-13.0% of forward passes saved), each remaining iteration still pays the Strip KV cost. This creates an asymmetry: the gross inference savings from Jacobi decoding (274.08s vs. 358.15s, a 23.5% reduction for Llama2-13B on CSQA) are partially offset by the Strip KV overhead that would not exist in autoregressive decoding.

Worse, the Strip KV overhead may increase as the exact-token region grows, because the operation involves trimming a larger cache tensor. The paper does not measure Strip KV time as a function of iteration count, but this would be an informative ablation: if Strip KV costs more in later iterations (when exact-token sequences are longer), then the overhead is not simply per-iteration but per-iteration-and-sequence-length, compounding the tax.

For latency-sensitive applications, the Strip KV overhead also affects per-query response time, not just total throughput. A single query that benefits from 30% fewer forward passes may still feel slow if the per-iteration KV-cache manipulation adds noticeable latency between token generation bursts.

What evidence exists in the paper: Table 2 provides the Strip KV cost exactly (18.58s) as part of the total time composition breakdown for Llama2-13B on CSQA. The paper is transparent about this cost, which is a strength. However, no ablation explores how Strip KV time varies with:

Model size (does 7B vs. 13B change the cost proportionally?)
Sequence length (does the overhead grow with generated tokens?)
Batch size (is the cost amortized across batched instances?)

The paper also does not compare against speculative decoding's overhead profile, which might have a different cost structure (draft model inference vs. KV-cache manipulation).

Mitigation status: Acknowledged only through measurement, not through mitigation. The paper does not propose optimizations to the Strip KV operation (e.g., maintaining a separate exact-only cache in parallel with the full Jacobi cache, or using sparse attention patterns that avoid recomputation). The Limitations section (7) does not mention Strip KV overhead as a target for improvement, which is a missed opportunity—given that it consumes 5.7% of runtime, even a 2× reduction would translate to a ~3% net speedup improvement, closing some of the gap between gross and net performance.

6.5 The Fixed Context Window Size Is Demonstrated to Be Suboptimal, but No Adaptive Mechanism Is Provided

The assumption or constraint: FastCoT uses a fixed Jacobi context window size c throughout the entire iteration process (Section 4.3, Section 5.5). The paper's own ablation (Figure 8) demonstrates conclusively that the optimal window size depends on the iteration stage: small windows (or c=0, i.e., no approximate tokens) perform best in early iterations, while larger windows (c ≥ 15) outperform small windows in mid-iterations (roughly iterations 10-65). The paper's interpretation is that early-iteration approximate tokens are low-quality noise that dilutes the exact-token signal, while later-iteration approximate tokens (after Jacobi refinement has improved them) provide useful glimpse-of-future information.

The consequence: The main experimental results (Figure 6, Table 1) use a single fixed window size for all iterations. Given the demonstrated non-monotonic relationship between window size and accuracy (Figure 8), a fixed window is guaranteed to be suboptimal: either it is too large in early iterations (degrading accuracy by introducing noise) or too small in later iterations (missing the opportunity to accelerate convergence by providing more future context). The paper's measured speedups and accuracy numbers therefore represent a lower bound on what FastCoT could achieve with a properly tuned dynamic window size policy.

The magnitude of this suboptimality is not quantified—the paper does not compare the fixed-window results against an oracle that uses the per-iteration best window size from Figure 8. Given that Figure 8 shows a ~4-percentage-point gap between c=0 and c=20 at iteration 30 (roughly 54% vs. 58% accuracy), the fixed-window choice could be leaving 2-5 percentage points of accuracy on the table—enough to close most of the performance loss gap reported in Table 1 (1.23-2.66%). If a dynamic policy could recapture even half of that gap, FastCoT might achieve higher accuracy than Vanilla CoT (since Figure 6 shows FastCoT often stabilizes at slightly below the CoT ceiling) while still providing speedup.

What evidence exists in the paper: Figure 8 provides the key evidence. The paper itself interprets the result correctly in Section 5.5: "the lower context window size works better than the higher one in the early stages of the iterations... But after about 15 iterations, Jacobi decoding would generate enough informative tokens beneficial to downstream CoT tasks." Section 7 (Limitations) explicitly acknowledges this gap:

"Although we did not discuss it in our work, we believe that the process of controlling the context window throughout the iteration can be seen as a Markov Decision Process. It would be an intriguing problem to utilize reinforcement learning algorithms to regulate the context window size during the iteration while defining appropriate rewards."

Mitigation status: Acknowledged as future work, with a plausible technical direction (RL-based dynamic window control). The paper does not implement any adaptive mechanism, even a simple heuristic (e.g., c = min(25, floor(iteration / 2))). This is the most actionable identified limitation—the paper's own data clearly shows the problem and a straightforward heuristic might capture most of the potential gain—yet the main results are reported with the suboptimal fixed-window policy. A practitioner implementing FastCoT should strongly consider dynamic window sizing, but the paper provides no validated policy to adopt.

6.6 No Comparison Against the Trivial Baseline of Generating a Shorter Rationale Autoregressively

The assumption or constraint: FastCoT's core mechanism combines two distinct sources of speedup: (1) Jacobi decoding's multi-token verification (IS/TI in Table 1), which reduces the number of forward passes needed to generate a given number of exact tokens, and (2) early stopping before the full rationale is generated, which saves forward passes at the cost of generating fewer exact tokens. The paper compares FastCoT against a FastCoT (w/o by-products) baseline that truncates the Jacobi-generated exact tokens to the same count, isolating the contribution of the approximate tokens. However, the paper never compares against the trivial baseline of simply generating a shorter rationale using standard autoregressive decoding and stopping early.

The consequence: This missing baseline conflates two questions that should be answered separately: (a) "Does Jacobi decoding's multi-token verification provide speedup for CoT reasoning?" and (b) "Do the approximate glimpse-of-future tokens enable better accuracy at the same exact-token count?" The paper focuses on question (b) via the FastCoT vs. FastCoT (w/o by-products) comparison. But question (a)—which accounts for the IS/TI savings—has no clean isolation.

Consider an alternative approach: take standard autoregressive CoT, generate exactly as many tokens as FastCoT produces exact tokens (the same termination point), then apply the same answer trigger. This baseline would have identical token quality to FastCoT's exact-token region (both are generated autoregressively), but would require the same number of forward passes as tokens (whereas FastCoT requires fewer passes due to Jacobi multi-token verification). If this baseline shows accuracy comparable to FastCoT—despite taking more forward passes to generate the same tokens—then the multi-token verification is the sole source of FastCoT's speedup, and the approximate tokens contribute nothing to accuracy beyond what the same number of exact tokens would provide. Conversely, if FastCoT substantially outperforms this baseline at the same exact-token count, the approximate tokens are demonstrably providing signal beyond exact-truncation.

Without this comparison, a practitioner cannot determine whether FastCoT's complexity (ATB management, Strip KV overhead, padding schemes, verification loop) is justified, or whether they could achieve most of the benefit by simply tuning an autoregressive generation length limit—a far simpler intervention requiring no modification to the decoding loop.

What evidence exists in the paper: The FastCoT (w/o by-products) baseline is the closest comparison, but it uses Jacobi-generated exact tokens (which have different properties from autoregressive-generated tokens at the same positions, since Jacobi exact tokens are verified against previous Jacobi states). The missing baseline is autoregressive decoding truncated to the same token count—a comparison that would require running autoregressive CoT up to the same exact-token count as FastCoT's termination point, then extracting the answer. This baseline is not evaluated, and the paper does not discuss why.

The corrupting rationale experiment (Figure 5) provides indirect evidence that partial rationales can work, but it uses oracle rationales (autoregressive-generated and then artificially masked), not autoregressive-truncated rationales. A truncation-at-40% experiment would answer the question directly: does generating exactly 40% of a rationale and stopping produce the same accuracy as generating 100% and masking 60%?

Mitigation status: Not addressed, not acknowledged. The missing baseline is a significant methodological gap because it would clarify whether FastCoT's complexity is necessary. If the answer is "yes, Jacobi approximate tokens provide signal beyond simple truncation," the paper should demonstrate this explicitly. If the answer is "truncation alone gets most of the benefit," then FastCoT's contribution is primarily the multi-token verification speedup (which standard Jacobi decoding or speculative decoding could also provide). Neither case is made.

7. Implications and Future Directions

How This Work Changes the Landscape

FastCoT introduces a conceptual reframing rather than a paradigm shift: it demonstrates that the intermediate by-products of accuracy-lossless decoding methods—tokens previously treated as transient noise to be discarded—can be productively repurposed as partial reasoning context when the final output is not the generated text itself but a downstream answer extracted from that text. This is not a new decoding algorithm, a new model architecture, or a new prompting strategy. It is a re-interpretation of Jacobi decoding's outputs that opens a previously unexamined design dimension in the CoT inference pipeline: the tolerance of answer extraction to token-level inaccuracy in the intermediate rationale.

The significance of this reframing is that it reconciles two previously disconnected optimization goals. The decoding acceleration community has focused exclusively on generating the same output faster—extending speculative decoding (Leviathan et al., 2023), Jacobi decoding (Santilli et al., 2023), and non-autoregressive models (Gu et al., 2017) under the constraint that the output distribution be preserved exactly. The CoT community has focused exclusively on improving answer accuracy through better prompt engineering—Self-Consistency, Tree-of-Thought, Self-Ask, and rationale distillation—with inference speed treated as a secondary, orthogonal concern. FastCoT sits at the intersection and says: for the specific structure of CoT reasoning, these goals are not orthogonal. The rationale is an intermediate representation, not the final deliverable, and its accuracy requirements are more relaxed than the translation or open-ended generation tasks that prior decoding acceleration work targeted. By relaxing the exact-output constraint only on the rationale portion, FastCoT achieves speedups that lossless methods cannot match, since lossless methods must still generate every rationale token.

This reframing makes several research directions newly attractive:

Lossy-by-design decoding for intermediate computation. The paper implicitly asks: what other intermediate model outputs (not just CoT rationales, but tool-use plans, retrieval queries, decomposition steps in multi-hop reasoning) might tolerate approximate rather than exact generation? If Figure 5's corrupting rationale experiment generalizes—if partial, noisy intermediate states can often drive correct downstream decisions—then the standard assumption that intermediate model outputs must be exact is unnecessarily restrictive across a broad class of LLM applications.
Difficulty-adaptive decoding strategies. The paper's dynamic-context-window insight (Section 5.5, Figure 8) shows that the optimal Jacobi window size is iteration-dependent: small windows are better early (when approximate tokens are noisy), larger windows are better later (when they carry useful signal). This mirrors a broader principle that decoding hyperparameters should adapt to the model's internal state, not remain fixed. Research on decoding strategies that condition on uncertainty estimates, attention entropy, or per-token PRM scores could build on this observation.
Understanding the information content of CoT rationales. The corrupting rationale experiment (Figure 5) provides direct evidence that CoT rationales contain substantial redundancy—accuracy saturates with only 40% of tokens revealed on StrategyQA. This finding, while not the paper's main contribution, has implications for the theoretical understanding of why CoT works. If rationales function primarily as structured context that biases the model's next-token distribution toward the correct answer (rather than as logically necessary step-by-step derivations), then research on rationale quality should focus on the properties that make rationales effective biasing signals—keyword density, structural markers, entity coherence—rather than on formal logical validity. This connects to the faithfulness literature (Lanham et al., 2023; Radhakrishnan et al., 2023) that the paper cites to explain the non-monotonic accuracy curve on Llama2-13B with StrategyQA.

Conversely, this work makes certain research directions less attractive within its scope:

Purely lossless decoding acceleration for CoT tasks. If a method insists on exact output matching, it must generate every rationale token. FastCoT shows that the tail of the rationale is partially redundant for answer extraction (Figure 5), so any method that generates the full rationale—however quickly—leaves speedup on the table relative to methods that identify when enough rationale has been generated to confidently extract the answer. Speculative decoding and standard Jacobi decoding are complementary (they can accelerate the generation of the exact-token prefix), but alone, they are fundamentally upper-bounded by the full rationale length.
Rationale distillation without speed consideration. Prior work (Magister et al., 2022; Hsieh et al., 2023; Li et al., 2023) focuses on training smaller student models to produce CoT rationales that improve their own accuracy. FastCoT suggests a different optimization target: train student models to produce shorter rationales that capture only the high-signal tokens—the 40-60% of tokens that the corrupting rationale experiment identifies as sufficient. If a student model could be trained to generate "minimal sufficient rationales" that are 40-60% of the teacher's rationale length while preserving accuracy, the combined speedup from shorter rationales plus FastCoT's Jacobi acceleration could be multiplicative.

A critical caveat: FastCoT's conceptual contribution is bounded by its demonstrated scope. The paper shows that CoT rationales for commonsense and strategy reasoning (CSQA, StrategyQA) and algebraic word problems (AQuA) contain redundancy that FastCoT can exploit. It does not demonstrate—and the authors do not claim—that this property holds for all reasoning types. Tasks requiring precise intermediate calculations (multi-digit arithmetic, symbolic equation solving, multi-step deduction with interacting constraints) may have rationales where every token is load-bearing, and approximate token substitution would cause cascading errors. The paper's contribution is thus best understood as establishing the existence of a class of reasoning tasks where lossy intermediate decoding is viable, and providing a method (the corrupting rationale experiment) for diagnosing whether a given task falls into this class. The classification of which reasoning tasks are "fast-CoT-able" and which are not is left entirely to future work.

Follow-Up Research This Work Enables

Dynamic context window sizing via iterative refinement quality estimation. The paper's own Figure 8 demonstrates that the optimal Jacobi context window size depends on iteration stage: c=0 is best in early iterations, larger c (up to ~20) is best in mid-iterations, and all window sizes converge in late iterations. The paper interprets this as a quality effect—early approximate tokens are noisy, later ones are more accurate. A concrete follow-up would implement a quality-conditioned window scheduler that uses the Jacobi verification module's own output as a quality signal: when the verification step (Equation 5) finds a long prefix match (many consecutive tokens verified), the Jacobi iterative solutions are high-quality, and the window can expand; when prefix matches are short, the approximations are unreliable, and the window should contract. The experiment would compare (a) a fixed window (the paper's current approach), (b) the quality-conditioned scheduler, and (c) an oracle scheduler that uses per-iteration accuracy from Figure 8 as a lookup table, measuring both accuracy-vs-iteration curves and wall-clock time at the convergence point. The paper already provides the necessary diagnostic framework (Figure 8) and a proposed MDP formulation (Section 7); what's missing is the implementation and evaluation against the fixed-window baseline.

Corrupting rationale experiments as a pretest for FastCoT applicability. The paper uses the corrupting rationale experiment (Section 5.2, Figure 5) to motivate FastCoT, but the experiment has independent value as a diagnostic probe for whether a given model-dataset pair will benefit from lossy intermediate decoding. A follow-up study could run the corrupting rationale experiment systematically across 10-15 diverse reasoning datasets (spanning arithmetic, logical deduction, multi-hop QA, code generation, scientific reasoning, planning) and multiple model families (LLaMA, Mistral, Qwen, DeepSeek), measuring the saturation overlap ratio—the fraction of rationale tokens at which accuracy plateaus—for each configuration. The hypothesis is that datasets with low saturation ratios (<0.5, like StrategyQA) will show the largest FastCoT speedups with minimal accuracy loss, while datasets with high saturation ratios (>0.8, possibly including multi-digit arithmetic) will show either no speedup (because early stopping degrades accuracy unacceptably) or require much tighter performance-loss thresholds that reduce the speedup to near zero. Such a study would transform FastCoT from a method demonstrated on three datasets into a framework with predictable applicability boundaries, enabling practitioners to pretest whether their task is suitable before implementing the Jacobi decoding pipeline.

Combining FastCoT with speculative decoding for multiplicative speedup. FastCoT exploits Jacobi decoding to generate c approximate tokens in the same forward pass that produces one exact token (Figure 4). Speculative decoding (Leviathan et al., 2023) uses a separate draft model to propose multiple tokens, which the large model verifies in parallel. These mechanisms are complementary: speculative decoding accelerates the production of exact tokens (the draft model proposes candidates that, if verified, are accepted into the exact token stream), while FastCoT repurposes the unverified tokens as approximate rationale context. A combined system would use a draft model to propose k tokens, run one large-model forward pass to verify them (accepting a tokens into the exact region, where a ≤ k), and simultaneously compute c Jacobi window positions beyond the verified prefix, whose tokens enter the approximate buffer. The experiment would measure whether the speedups compose multiplicatively (a 1.5× speedup from speculative times a 1.2× speedup from FastCoT yielding 1.8× total) or whether they interfere (e.g., the draft model's token distribution differs from the Jacobi iterative solutions', reducing verification rates). The paper's time composition breakdown (Table 2) provides a baseline for attributing wall-clock costs, and the IS/TI metric provides a per-model measure of Jacobi-specific savings, enabling clean isolation of each mechanism's contribution.

Training minimal-sufficient-rationale student models. The corrupting rationale experiment (Figure 5) shows that only 40-60% of an oracle rationale's tokens are needed for saturated accuracy. A natural extension is to train a student model to generate only these high-signal tokens, producing shorter rationales that FastCoT can then accelerate further via Jacobi decoding. The training data would be constructed by: (1) using the corrupting rationale experiment to identify, per question, which tokens are necessary for correct answer extraction (e.g., using a gradient-based attribution method or a leave-one-out perturbation approach), (2) distilling the teacher model on examples where the target output is the minimal sufficient rationale rather than the full rationale. The experiment would compare three approaches: (a) FastCoT on the full teacher (the paper's current method), (b) autoregressive decoding of minimal sufficient rationales (no FastCoT), and (c) FastCoT on minimal sufficient rationales. The hypothesis is that (c) would provide multiplicative speedup—shorter rationales mean fewer forward passes even with lossless decoding, and FastCoT's Jacobi mechanism further reduces the per-token forward pass count—while preserving accuracy close to the full-teacher CoT baseline. This experiment tests whether the token-level redundancy FastCoT exploits can be designed into model outputs rather than merely discovered in existing outputs.

Stress-testing FastCoT on long-horizon and calculation-heavy reasoning. The paper's three datasets (CSQA, StrategyQA, AQuA) involve rationales of modest length (tens to low hundreds of tokens) and reasoning that is primarily qualitative (commonsense, strategy) rather than quantitative. A critical stress test would apply FastCoT to tasks where rationales are both longer and more sequentially interdependent: GSM8K (grade-school math, multi-step arithmetic where each step's numeric output feeds the next), MATH (competition mathematics with symbolic manipulation and multi-paragraph proofs), and BIG-Bench Hard (diverse reasoning tasks including logical deduction, causal reasoning, and temporal sequencing). The key measurements would be: (a) the saturation overlap ratio from the corrupting rationale experiment on each dataset—if rationales for MATH problems show near-linear accuracy decay with masking (saturation ratio >0.9), FastCoT is inapplicable; (b) whether the non-monotonic accuracy dip observed on StrategyQA (Figure 6, top-right) appears on other tasks where model rationales are known to be unfaithful; and (c) whether the Jacobi context window's fixed depth (c ≤ 25 in the paper's experiments) is sufficient for tasks where critical answer-relevant information lies 50+ tokens ahead of the current exact-token position. This stress test would establish the boundary between FastCoT-applicable and FastCoT-inapplicable reasoning types, converting the paper's implicit scope limitation into an explicit, empirically grounded taxonomy.

Offline difficulty estimation for per-question iteration budget allocation. The paper uses a uniform iteration budget calibrated on a subset of the dataset (Section 4.4, Stop Condition 1). Figure 6 shows that convergence speed varies significantly across models and datasets, and it is plausible that it also varies across individual questions within a dataset—easy questions may need fewer iterations than hard ones. A follow-up could train a lightweight question-difficulty classifier that takes only the question text as input (no model inference) and predicts the minimum number of FastCoT iterations needed to reach within, say, 2% of the question's converged accuracy. Training data would come from running FastCoT to convergence on a subset of questions and recording per-question accuracy-vs-iteration curves. At inference time, the classifier predicts a per-question iteration budget, and FastCoT uses the maximum predicted budget across the batch as the stop condition (or pads shorter generations with a no-op mask). This experiment would test whether per-question adaptive allocation can close the 1-3% accuracy gap reported in Table 1 while maintaining or improving the speedup—since easy questions would stop earlier than the uniform budget, saving additional forward passes. The paper already provides the calibration framework and the statistical infrastructure; adding a difficulty classifier is a natural efficiency improvement.

Practical Applications and Downstream Use Cases

Batch inference for reasoning-heavy document processing at scale. Consider a legal tech company processing thousands of contracts per day, using an LLM with CoT prompting to answer structured questions about each contract (e.g., "Does this clause comply with Regulation X? Let's think step by step."). Each query generates a 100-200 token rationale followed by a yes/no answer. With autoregressive decoding on Llama2-13B-class hardware, processing 10,000 contracts might take roughly 10,000 × (365.20s / |test_set|)—substituting the CSQA test set size (1,221 questions) as a rough estimate, that's ~2,990 seconds or ~50 minutes of GPU time for 10,000 queries. FastCoT at 10.47% time reduction (Table 1) saves ~5.2 minutes per 10,000-query batch. Across a year of daily processing, this is ~31.6 GPU-hours saved—modest per-batch but meaningful at annual aggregate for a single application. For higher-throughput settings or larger reasoning models, the absolute savings scale proportionally. The key deployment condition is that the contracts domain must exhibit the redundancy property from Figure 5—if legal reasoning rationales contain individually load-bearing logical steps with no redundancy, early stopping could degrade accuracy unacceptably. A corrupting rationale pretest on a held-out sample of the target domain's queries before deployment would establish whether FastCoT is appropriate.

Self-improvement data generation pipelines with controlled cost. A growing paradigm is using LLMs to generate training data for themselves (or for smaller student models) by producing CoT rationales on unlabeled questions and filtering for correct answers via self-consistency or an external verifier (Zelikman et al., 2022; Singh et al., 2024). In these pipelines, the rationale generation cost is the dominant computational expense—a single training run might require generating rationales for 10,000-100,000 unlabeled questions. FastCoT's 10-20% wall-clock reduction (Table 1) directly translates to 10-20% lower generation cost, or equivalently, 10-20% more training data generated within the same compute budget. The tolerable accuracy loss (1-3%) may be acceptable or even beneficial in this context, since self-improvement pipelines typically include a filtering step that selects correct answers—and if FastCoT's accuracy loss affects a random subset of questions, the filtering step simply discards slightly more generated data, with the cost savings on the remaining generated data still accruing. The key risk is whether FastCoT introduces systematic errors (e.g., consistently failing on questions requiring specific reasoning patterns) that bias the filtered training data toward the reasoning patterns FastCoT happens not to disrupt. Evaluating the distribution of FastCoT errors across reasoning types would be a necessary validation step before deployment in a self-improvement loop.

Latency reduction for interactive reasoning assistants. For real-time tutoring systems, AI-powered search with reasoning, or interactive debugging assistants, the user waits for the rationale to generate before seeing the answer. FastCoT reduces the number of forward passes (IS/TI in Table 1) and terminates generation early, both of which reduce time-to-first-answer-token—the wall-clock delay the user experiences. The magnitude of the reduction depends on the model and dataset, but ranges from 3.20% (Llama-7B on StrategyQA) to 23.71% (Llama2-7B on CSQA) in the paper's measurements. For a query that takes 5 seconds with autoregressive CoT, a 20% reduction is 1 second—noticeable in interactive settings. However, there is a latency-consistency tradeoff: FastCoT's per-iteration latency is dominated by the Strip KV overhead (5.7% of runtime in Table 2), which adds a fixed per-iteration cost regardless of tokens generated. For very short rationales (where autoregressive decoding would take only a few forward passes anyway), the per-iteration overhead might make FastCoT slower than the baseline in terms of wall-clock time—early stopping saves forward passes, but each remaining forward pass costs more due to cache manipulation. The paper does not measure FastCoT's per-query latency distribution, only total test-set wall-clock time. A deployment to latency-sensitive interactive systems should benchmark per-query latency at various rationale lengths to identify the crossover point where FastCoT's overhead exceeds its benefit.

GPU-constrained edge deployment of reasoning models. On edge devices with limited GPU memory and compute (e.g., an NVIDIA Jetson or a laptop GPU running a quantized 7B model), every forward pass is expensive, and generating a 100-token rationale can take tens of seconds. FastCoT's mechanism operates at the generation-loop level and requires no additional model parameters, no draft model, and no change to the model weights—all of which are constrained on edge hardware. The speedup comes purely from reducing the number of forward passes (IS/TI in Table 1) and enabling early stopping. A quantized Llama-7B running FastCoT on an edge device could, in principle, achieve the same 3.64-23.71% speedup range shown in Table 1, with the absolute time savings being much more impactful for user experience than in datacenter deployments where absolute times are already low. The key unknown is whether the Strip KV and padding overheads (which are GPU-memory-bandwidth-dependent) scale differently on edge GPUs—the paper's A100 measurements may not transfer directly. Benchmarking FastCoT on edge-class hardware (Jetson Orin, Apple M-series, Intel integrated GPU with OpenVINO) would be a prerequisite for this use case.