Chain of Draft: Thinking Faster by Writing Less

🎯 Pitch

LLMs can solve multi-step reasoning problems just as accurately while using only 7.6% of the tokens—by jotting down five-word notes instead of verbose explanations. This ‘Chain of Draft’ prompting slashes latency by up to 76% on GPT-4o and Claude 3.5 Sonnet, matching Chain-of-Thought accuracy on several benchmarks.

1. Executive Summary

This paper introduces Chain of Draft (CoD), a prompting strategy that reduces the verbosity of LLM reasoning by encouraging models to produce minimal, draft-style intermediate steps — limited to roughly five words each — instead of the full natural-language explanations characteristic of Chain-of-Thought (CoT). Evaluated on arithmetic reasoning (GSM8k), commonsense reasoning (date understanding and sports understanding from BIG-bench), and symbolic reasoning (coin flip) using GPT-4o and Claude 3.5 Sonnet, CoD maintains competitive or improved accuracy — reaching 91% on GSM8k versus CoT's ~95% — while using as little as 7.6% of the tokens and cutting latency by 48–76%, establishing that effective multi-step reasoning in LLMs does not require verbose intermediate outputs only when few-shot examples of the draft style are provided, as zero-shot and small-model (<3B parameter) settings reveal a substantial accuracy gap between CoD and CoT.

2. Context and Motivation

The Hidden Cost of Reasoning: Why Verbosity Is a Problem Worth Solving

The last three years have witnessed a dramatic shift in how large language models are deployed for complex reasoning tasks. Following Wei et al. (2022)'s Chain-of-Thought (CoT) prompting — which demonstrated that encouraging models to "think step by step" substantially improves accuracy on arithmetic, commonsense, and symbolic reasoning benchmarks — the field has largely converged on a consensus: structured intermediate reasoning is essential for generating correct answers to multi-step problems. This insight has been amplified by a new generation of reasoning-trained models (OpenAI's o1, DeepSeek's R1, Alibaba's QwQ) that internalize chain-of-thought-style processing through specialized training rather than relying solely on prompting, pushing benchmark performance to unprecedented levels.

But this progress has come with an under-scrutinized cost: verbosity. As the paper demonstrates in Table 1, solving a GSM8k math problem with CoT prompting on GPT-4o requires an average of 205 output tokens, translating to 4.2 seconds of latency compared to 0.6 seconds for direct answering. On Claude 3.5 Sonnet, sports understanding tasks balloon to 189 tokens and 3.6 seconds under CoT versus 1.0 seconds for direct answers (Table 3). These numbers may seem modest in isolation, but they compound dramatically in real-world deployments: a customer support system processing thousands of queries per hour, an educational tool providing step-by-step math tutoring, or a coding assistant reasoning about multiple files simultaneously.

This paper identifies a concrete gap that sits at the intersection of accuracy, latency, and cost — a gap that the prior literature has largely ignored:

"The latency issue has often been overlooked in studies of the reasoning capabilities of LLMs. However, it is crucial for lots of real-time applications to have low latency while maintaining high-quality responses." (Section 5)

The key observation motivating this work is that the form of intermediate reasoning — its length, style, and level of detail — is treated as a byproduct of the reasoning method rather than a design variable to be optimized. CoT prompts produce verbose reasoning because the few-shot exemplars are verbose, because the models were trained on verbose explanations, and because no constraint is placed on how much the model should elaborate. The result is effective but inefficient: models generate full sentences with restated problem facts, natural-language transitions, and explanatory framing that consume tokens without contributing to the logical progression toward the answer.

The Human Reasoning Disanalogy

The paper draws on an observation about human cognition that exposes a misalignment in how LLMs are typically prompted. When humans solve multi-step problems — whether mathematical derivations, logical puzzles, or planning tasks — they rarely write out complete, polished sentences. Instead, they use a drafting strategy: jotted equations, abbreviated notes, symbolic manipulations, and minimal annotations that capture only the essential intermediate results. A human working through the lollipop example from Section 3 would write something like:

20 - x = 12; x = 8

Not:

"Let's think through this step by step. 1. Initially, Jason had 20 lollipops. 2. After giving some to Denny, Jason now has 12 lollipops. 3. To find out how many lollipops Jason gave to Denny, we need to calculate the difference between the initial number of lollipops and the remaining number..."

The human draft is roughly 10 tokens. The CoT version is roughly 170 tokens. Both arrive at the same answer.

This is not merely a stylistic difference — it reflects a fundamental efficiency principle in human cognition: externalize only what your working memory cannot reliably hold. Humans use scratch paper to offload intermediate results, not to rehearse problem statements. LLMs, by contrast, have no working memory constraints in the human sense, yet they inherit the verbose explanatory style from training data that emphasizes pedagogical completeness (textbook solutions, tutorial dialogues, "show your work" grading rubrics). The paper's core intuition is that this verbosity is wasteful — it consumes computation that could be redirected toward solving harder problems or serving more users, and it generates tokens that the user often does not need to see.

Where Prior Efficiency Approaches Fall Short

The paper situates its contribution against a landscape of existing methods that have attempted to address LLM inference efficiency, identifying specific limitations of each:

Streaming (partial output display): Serving intermediate tokens as they are generated can reduce perceived latency by showing users progress rather than forcing them to wait for the complete response. However, as the paper notes, streaming "cannot fully mitigate overall latency or computational cost, and it is often unsuitable for chain-of-thought reasoning, as intermediate steps are often not intended to be shown to end users" (Section 2). In many applications — customer-facing chatbots, API services, automated pipelines — the intermediate reasoning is an implementation detail, not content the user needs to see. Streaming doesn't reduce the total tokens generated; it just changes when they appear.

Skeleton-of-Thought (SoT; Ning et al., 2023): This approach guides LLMs to first generate a skeleton outline of the answer and then decode different parts of the skeleton in parallel to reduce latency. While effective at decreasing wall-clock time through parallelization, the paper identifies two limitations: it "does not reduce computational cost" (the total token count remains high because the model still generates complete explanations in parallel) and it is "limited to questions that can be parallelized effectively" — multi-step reasoning chains where step 3 depends on step 2 cannot be parallelized, and many reasoning tasks exhibit precisely this sequential dependency structure.

Speculative decoding approaches (Zhang et al., 2023): These generate draft tokens at lower quality but higher speed by skipping intermediate transformer layers, then validate the drafts in a single forward pass. This is a hardware-level optimization that speeds up token generation without changing what tokens are generated. It is orthogonal to CoD — the paper explicitly notes that CoD "can be combined with these approaches to further reduce the latency" — but does not address the fundamental verbosity problem that CoD targets. A verbose reasoning chain decoded speculatively is faster than the same chain decoded autoregressively, but it still contains redundant content.

Continuous latent reasoning (Coconut; Hao et al., 2024): This trains LLMs to perform reasoning steps in a continuous latent space (using the model's final hidden state as a "thought" representation) rather than generating natural language tokens for intermediate steps. This is perhaps the most radical efficiency approach: eliminate intermediate token generation entirely. The paper identifies three critical limitations that motivate why natural-language-based approaches like CoD remain valuable:

Reduced accuracy on complex tasks: Coconut "suffers from reduced accuracy in complex tasks, such as GSM8k" — the latent reasoning space may lack the precision needed for exact arithmetic.
Loss of interpretability: "it loses the interpretability of natural language reasoning." For debugging, verification, and trust, being able to inspect intermediate steps is valuable. A latent vector is opaque.
Incompatibility with black-box models: Coconut requires access to internal model states, making it "cannot be applied to black-box models like GPT and Claude." For the majority of practitioners who access LLMs through APIs, Coconut is not a viable option.

Token-budget methods (CCoT and TALE): The two approaches closest to CoD are Concise Thoughts (CCoT; Nayab et al., 2024) and Token-Budget-Aware LLM Reasoning (TALE; Han et al., 2024), both of which constrain the total token budget allocated to reasoning. The paper identifies distinct weaknesses in each:

CCoT imposes a fixed global token budget across all problems. But "different tasks may require varying budgets to achieve the optimal balance between performance and cost." A simple arithmetic problem might need 10 reasoning tokens; a complex multi-step logic puzzle might need 100. A fixed budget is either wasteful on easy problems or crippling on hard ones. Moreover, "LLMs may fail to adhere to an impractical budget, often generating far more tokens than intended" — telling the model "use at most 20 words" does not reliably produce 20-word outputs, a finding replicated from Han et al. (2024).
TALE addresses the fixed-budget limitation by dynamically estimating a global token budget for each problem based on its estimated complexity. However, this introduces its own costs: "it requires an additional LLM call to estimate the budget, which increases latency." The overhead of running a budget-estimation model before the actual reasoning model partially offsets the efficiency gains. More fundamentally, TALE's approach "assumes that the model can accurately predict the complexity of requests, limiting its applicability to more complex tasks where reflection, self-correction, or external knowledge retrieval may be necessary during the reasoning process." If a problem turns out to be harder than estimated, the pre-allocated token budget may be insufficient.

The paper positions CoD as addressing these limitations through a fundamentally different mechanism: a per-step word budget rather than a global token budget. By constraining each individual step to roughly five words (a soft guideline, not an enforced hard limit), CoD allows unlimited steps — a complex problem that genuinely requires many reasoning stages can simply take more steps, each one concise. This avoids the rigidity of CCoT's fixed total budget and the estimation overhead of TALE, while retaining the interpretability that Coconut sacrifices and the black-box compatibility that Coconut requires. The paper explicitly frames this as an advantage: "our approach employs a per-step budget, allowing unlimited reasoning steps, which makes it more adaptable to various structured reasoning techniques" (Section 2).

The Overthinking Problem

An additional motivating observation — implicit in the paper's design but aligned with recent findings in the broader literature — is that LLMs exhibit overthinking on simple tasks. Chiang and Lee (2024) and Chen et al. (2024b), both cited in the paper, document that reasoning-trained models often produce excessively long reasoning chains for straightforward problems where the answer is obvious. The paper mentions that "the model's lack of awareness regarding task complexity often leads to overthinking even on simple tasks, resulting in unnecessary resource consumption" (Section 2). CoD can be understood as addressing this from the prompt-engineering side: by instructing the model to keep each step minimal, it nudges the model toward proportional reasoning effort — easy problems get fewer and shorter steps; harder problems can still take more steps, but each remains concise.

A Frame for the Contribution: Reasoning Efficiency as an Independent Research Dimension

Reading across the paper's positioning, the implicit argument is that the reasoning capabilities literature has been one-dimensionally focused on accuracy — chain-of-thought, tree-of-thought, graph-of-thought, self-consistency, ReAct with tool use, and the new generation of reasoning-trained models are all evaluated primarily on whether they get the right answer. Efficiency — measured in output tokens, latency, and compute cost — has been treated as a downstream engineering concern rather than a property of the reasoning method itself.

The paper positions CoD as carving out a distinct point in the design space: a prompting strategy optimized for the accuracy-per-token frontier. The key claim is not that CoD is more accurate than CoT — it generally matches or slightly underperforms CoT on accuracy (91% vs. 95% on GSM8k) — but rather that it achieves comparable accuracy at a fraction of the cost, and that this tradeoff is the right one for a large class of real-world applications. Figure 1 visually anchors this claim by plotting accuracy against token usage across tasks for Claude 3.5 Sonnet, showing that CoD sits in a region of the Pareto frontier that CoT cannot reach: high accuracy with dramatically lower token consumption.

Why This Paper Exists Now

The timing of this work is not accidental. As of early 2025, the LLM ecosystem has bifurcated: reasoning-specialized models (o1, R1, QwQ) deliver state-of-the-art accuracy on complex benchmarks but do so through internal chain-of-thought processes that are often hidden from the user but still consume significant inference compute. Simultaneously, LLM deployment has moved from research demos to production systems where latency directly affects user experience and token costs determine economic viability. This creates a tension: the methods that maximize benchmark scores may not be practical for applications that are cost-sensitive or latency-sensitive.

CoD enters this landscape as a prompt-engineering intervention that does not require retraining models, modifying architectures, or accessing internal representations. It works with any model that responds to few-shot prompting — including the black-box API models (GPT-4o, Claude 3.5 Sonnet) that the paper primarily evaluates. This makes it immediately deployable while research on more fundamental solutions (training with compact reasoning data, latent-space reasoning) progresses. The paper's contribution is thus simultaneously a practical tool and a conceptual demonstration: effective reasoning does not require verbosity, and treating efficiency as a first-class objective in prompt design opens up a region of the accuracy-efficiency trade-off that the field had not systematically explored — with the important caveat, revealed in the limitations analysis (Section 4.5), that this region is only accessible when the model has been exposed to draft-style reasoning patterns through few-shot exemplars, suggesting that the training data of current models lacks the necessary priors.

3. Technical Approach

3.1 Reader Orientation

Chain of Draft (CoD) is a prompting strategy — a specific way of writing instructions and few-shot examples that shapes how an LLM structures its reasoning — not a new model architecture, training procedure, or decoding algorithm. The system being built is simply a prompt template combined with hand-crafted exemplars that together teach the model to externalise its intermediate reasoning as a sequence of ultra-concise, information-dense "drafts" rather than as verbose natural-language explanations. The problem it solves is the verbosity-accuracy tension in multi-step reasoning: Chain-of-Thought prompting achieves high accuracy by having the model produce detailed step-by-step reasoning, but this reasoning consumes hundreds of output tokens and adds seconds of latency per query. CoD reshapes the reasoning output to capture only the essential logical or computational content at each step — typically as short equations, variable assignments, or symbolic manipulations — preserving the structured, step-by-step structure that makes CoT effective while discarding the explanatory and discursive framing that makes CoT expensive.

3.2 Big-Picture Architecture (Diagram in Words)

The CoD system has three major components, though two of them are "prompt design" rather than "code":

System-level instruction (system prompt): A short directive prepended to the model's context that defines the output format constraints: think step by step, keep each step to roughly five words, return the final answer after ####. This sets the global behaviour expectation.
Few-shot exemplars (demonstrations): Hand-crafted input-output pairs showing the model what CoD-style reasoning looks like for the target task. Each exemplar contains a question and a CoD-formatted reasoning chain ending with the answer. These teach the model the desired reasoning style (concise, symbolic, step-structured) through imitation.
The LLM itself (black-box inference engine): GPT-4o or Claude 3.5 Sonnet, accessed via API. The model receives the concatenated system prompt + exemplars + test question and generates the reasoning chain and answer autoregressively. CoD imposes no architectural modifications, no fine-tuning, and no access to internal states — it operates entirely through the model's in-context learning capability.

Information flows linearly: a test question is appended to the prompt template (system instruction + few-shot CoD examples) → the LLM generates a sequence of concise reasoning steps (each approximating the 5-word-per-step guideline) followed by a separator (####) and the final answer → the answer is extracted by parsing everything after ####.

3.3 Roadmap for the Deep Dive

First, the core mechanism: how CoD transforms the reasoning output compared to CoT — what "concise" means operationally and why the per-step structure matters.
Second, the prompt engineering: the exact system prompts and few-shot exemplar design, including the critical "5 words at most" guideline and why it is a soft constraint rather than an enforced limit.
Third, the comparison to alternative efficiency methods: a granular walk-through of why per-step budgeting is fundamentally different from global token budgeting (CCoT, TALE) and latent reasoning (Coconut), explaining the design-space trade-offs the paper makes explicit.
Fourth, the answer extraction protocol: how the #### separator standardises output parsing across all three prompting strategies (Standard, CoT, CoD) and why this matters for fair comparison.

3.4 Detailed, Sentence-Based Technical Breakdown

CoD is primarily an empirical design paper whose core idea is that the verbosity of chain-of-thought reasoning is not a necessary condition for its accuracy benefits — a well-structured sequence of ultra-concise intermediate steps can preserve the logical scaffolding while dramatically reducing token consumption. The technical approach is a prompt-engineering intervention that encodes this idea into two artifacts: a system-level instruction and a set of few-shot exemplars.

The Core Mechanism: Compressing Reasoning Steps While Preserving Step Structure

The fundamental insight differentiating CoD from simpler approaches (like "be concise" prompts) is that it preserves the sequential step structure of Chain-of-Thought while compressing each step's content. CoT's effectiveness is widely attributed to two properties: (1) it decomposes a multi-step problem into sub-problems that the model can solve incrementally, and (2) it externalises intermediate results, allowing the model to condition its subsequent reasoning on explicitly stated prior conclusions. CoD retains both properties — it still produces a sequence of reasoning steps, each building on previous ones — but the form of each step shifts from explanatory prose to minimal symbolic representation.

The paper illustrates this with a concrete side-by-side example in Section 3. For the lollipop problem ("Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny?"), the CoT version produces roughly 170 tokens spanning six enumerated steps with natural-language framing:

"Let's think through this step by step: 1. Initially, Jason had 20 lollipops. 2. After giving some to Denny, Jason now has 12 lollipops. 3. To find out how many lollipops Jason gave to Denny, we need to calculate the difference between the initial number of lollipops and the remaining number. 4. We can set up a simple subtraction problem: Initial number of lollipops - Remaining number of lollipops = Lollipops given to Denny. 5. Putting in the numbers: 20 - 12 = Lollipops given to Denny. 6. Solving the subtraction: 20 - 12 = 8. Therefore, Jason gave 8 lollipops to Denny."

The CoD version compresses this into a single line of symbolic manipulation:

"20 - x = 12; x = 20 - 12 = 8. #### 8"

This is approximately 15 tokens — a reduction of over 90%. The compression works through three mechanisms:

Elimination of problem restatement: CoT step 1 repeats "Initially, Jason had 20 lollipops" — a fact already stated in the question. CoD omits this entirely because the model can reference the question context directly.
Symbolic encoding of operations: Instead of explaining "we need to calculate the difference... we can set up a subtraction problem," CoD writes the equation 20 - x = 12. The mathematical structure itself encodes both the operation (subtraction) and the unknown (what was given away).
Elimination of transitional and concluding discourse: Phrases like "Let's think through this step by step," "Therefore," and "Putting in the numbers" are removed entirely. The reasoning content — the equation and its solution — remains.

Critically, CoD is not simply asking the model to "be brief." The per-step constraint (five words at most) is designed to break the model's learned tendency toward explanatory prose at the structural level. By instructing the model to limit each individual step, rather than the total output, CoD creates a different reasoning rhythm: one where each line is a self-contained logical or computational unit — essentially a note-to-self — rather than a sentence in an expository paragraph. This aligns with the human drafting behaviour the paper cites as inspiration.

Prompt Engineering: The System Instruction and Few-Shot Exemplar Design

CoD's mechanism operates through two prompt components, both detailed in Section 4.1.

The system prompt. This is the instruction prepended to the model's input before any examples or test questions. For CoD, the exact text is:

"Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Return the answer at the end of the response after a separator ####."

Three design choices embedded in this short instruction warrant examination:

"Think step by step": This phrase is directly inherited from the CoT literature (Kojima et al., 2022), where it was shown to be the minimal trigger for eliciting structured intermediate reasoning. CoD retains it because the core mechanism still relies on sequential decomposition of the problem. Without this phrase, the model might default to a single-step direct answer without any intermediate computation.
"only keep a minimum draft for each thinking step, with 5 words at most": This is the novel constraint. The number five is not derived from any theoretical analysis — it is a heuristic guideline chosen to be small enough to force significant compression but not so small as to make coherent reasoning impossible. The paper explicitly states: "Note that we do not enforce such limitation in any way, it is just a general guideline to promo[te] short reasoning steps." This is a crucial design decision. A hard enforcement mechanism (e.g., truncating each step to exactly five words) would risk cutting off essential information mid-reasoning, potentially degrading accuracy more than the guideline approach. The soft constraint works through the model's instruction-following behaviour: models trained with RLHF are incentivised to comply with explicit instructions, so stating the five-word guideline biases the generation distribution toward shorter steps without introducing hard cutoffs that could break the reasoning chain.
"Return the answer at the end of the response after a separator ####.": This standardises answer extraction across all three prompting strategies (Standard, CoT, CoD) evaluated in the paper, ensuring that any measured accuracy differences are attributable to the reasoning strategy rather than to differences in how the final answer is identified and parsed.

The few-shot exemplars. These are manually written by the authors and included in the prompt before the test question. The paper states: "For each few-shot example, we also include the Chain of Draft written manually by the authors." The exemplars serve as in-context demonstrations that teach the model what CoD-style reasoning looks like in practice. This is essential because, as the paper hypothesises in Section 4.5, "CoD-style reasoning patterns" are likely "scarce or absent" in the pretraining data of current LLMs — the models have been trained on verbose explanations, textbook solutions, and tutorial dialogues, not on terse symbolic drafts. The few-shot exemplars bridge this distribution gap by providing the model with a template to imitate.

The paper does not reproduce the exact few-shot exemplars in the main text (they are presumably in an appendix or the linked repository), but their design can be inferred from the CoD example in Section 3 and the evaluation results. For arithmetic reasoning (GSM8k), each exemplar would show a word problem followed by a sequence of short equations and the final answer. For commonsense reasoning (date understanding, sports understanding), the drafts would capture the logical deductions in minimal form. For symbolic reasoning (coin flip), the drafts would track the state changes without narrative framing.

The decision to use author-written rather than model-generated or algorithmically compressed exemplars is a practical one: it guarantees that the exemplars are high-quality demonstrations of exactly the style the authors intend, and it avoids the circularity of using the model to generate its own training examples (which could reinforce verbose patterns if the model hasn't yet learned the concise style).

For comparison, the paper also specifies the Standard and CoT system prompts:

Standard: "Answer the question directly. Do not return any preamble, explanation, or reasoning."
CoT: "Think step by step to answer the following question. Return the answer at the end of the response after a separator ####."

Both lack the per-step word constraint that defines CoD. The CoT prompt is essentially the same as CoD minus the "five words at most" clause, isolating the effect of the draft-style constraint from the effect of step-by-step reasoning itself.

Per-Step Budgeting vs. Global Token Budgeting: The Design-Space Distinction

The paper explicitly contrasts CoD's per-step word budget against the global token budget approaches of CCoT (Nayab et al., 2024) and TALE (Han et al., 2024). Understanding this distinction is central to grasping why CoD works differently.

Global token budgeting (CCoT/TALE): These methods impose a limit on the total number of tokens the model can use for reasoning across all steps. CCoT uses a fixed budget (e.g., "use at most 100 tokens for reasoning"). TALE dynamically estimates a problem-specific budget via a separate LLM call. The model must distribute this total budget across however many reasoning steps it chooses to take.

The paper identifies two failure modes:

Rigidity on heterogeneous problems: A problem that genuinely requires more reasoning steps than the budget allows will be truncated before reaching a correct answer. Conversely, a simple problem that could be solved in fewer tokens than the budget allows may still use the full budget (the model has no incentive to stop early, and LLMs often fill allocated space with redundant content).
Compliance failure: "LLMs may fail to adhere to an impractical budget, often generating far more tokens than intended." This is an empirical observation from Han et al. (2024) that the paper cites: telling a model "use at most 20 tokens" does not reliably produce 20-token outputs because the model's generation process is not token-count-aware in a precise way.

Per-step word budgeting (CoD): By constraining each individual step rather than the global total, CoD avoids both failure modes. If a problem requires 10 reasoning steps, the model takes 10 steps, each ~5 words — the total token count scales naturally with problem complexity. There is no risk of truncation mid-reasoning because no step limit is imposed, and there is no risk of the model padding a simple problem to fill a budget because there is no global budget to fill. The per-step guideline simply shapes the density of each reasoning unit.

This design choice also makes CoD "more adaptable to various structured reasoning techniques" (Section 2). Because CoD doesn't constrain the number of steps, it composes naturally with methods that might require varying step counts — reflection loops where the model backtracks and revises, tool-using workflows where a step involves an API call and result parsing, or self-consistency approaches where multiple reasoning chains are generated in parallel. A global token budget would need to be pre-allocated for all of these, guessing at the required length; a per-step budget adapts dynamically.

The cost of this flexibility is that CoD provides no absolute guarantee on total output length — a model could theoretically take 100 five-word steps and produce 500 tokens of reasoning. In practice, the paper's results show that this doesn't happen: the average output token counts in Tables 1-4 are dramatically lower than CoT, suggesting that the per-step density constraint, combined with the few-shot exemplars showing short chains, implicitly encourages the model to be efficient in both dimensions (steps taken and words per step).

Relationship to Latent Reasoning (Coconut)

The paper also contrasts CoD against Coconut (Hao et al., 2024), which takes a fundamentally different approach: instead of generating natural-language intermediate tokens at all, Coconut trains the model to perform reasoning steps in the continuous latent space using the final hidden state as a "thought" vector. The comparison illuminates what CoD deliberately preserves that Coconut sacrifices:

Interpretability: CoD's intermediate steps are human-readable (if terse) natural language or symbolic expressions. A user or developer can inspect the reasoning chain to verify correctness, identify errors, or understand the model's logic. Coconut's latent vectors are opaque — there is no way to inspect what the model is "thinking" at intermediate stages. For applications where trust, debuggability, or audit trails matter (education, legal reasoning, medical decision support), this is a significant practical consideration.

Black-box compatibility: CoD works with any model accessible through a standard text-completion API. Coconut requires access to internal hidden states, which is unavailable for API-only models like GPT-4o and Claude 3.5 Sonnet — the two primary models evaluated in this paper. For the majority of practitioners who do not host their own models, CoD is deployable today; Coconut is not.

Accuracy on complex tasks: The paper notes that Coconut "suffers from reduced accuracy in complex tasks, such as GSM8k." This is a critical empirical finding: latent-space reasoning may lose precision that natural-language (even terse natural-language) reasoning retains. The hypothesis (not stated in the paper but consistent with the broader literature) is that natural language provides a discretisation and symbol-grounding mechanism that helps with exact operations like arithmetic, where a latent vector representing "the number 8" might drift or blend with nearby numbers in a way that the token "8" does not.

CoD thus occupies a middle ground in the design space: it reduces the verbosity of natural-language reasoning without abandoning natural language (or symbolic notation) entirely, retaining interpretability and API compatibility while achieving substantial efficiency gains.

Answer Extraction Protocol

All three prompting strategies (Standard, CoT, CoD) share a common answer extraction mechanism to ensure fair comparison. The system prompts for CoT and CoD both specify: "Return the answer at the end of the response after a separator ####." The model's complete output is parsed by splitting on the string #### and taking everything after it as the predicted answer. For Standard prompting, where no reasoning is generated, the output is expected to be the answer directly (with the instruction "Do not return any preamble, explanation, or reasoning").

This design choice addresses a common challenge in evaluating reasoning prompts: how to reliably extract the final answer from a verbose output that may contain the answer in multiple places, parenthetical remarks, or restatements. The #### separator acts as an explicit structural marker that the model is trained (through the few-shot exemplars, which all follow this format) to place immediately before the final answer. Using the same separator convention for both CoT and CoD ensures that any differences in extraction reliability (e.g., the model forgetting to include the separator, or placing it in the wrong location) affect both strategies equally, so accuracy comparisons are not confounded by parsing artefacts.

The paper notes that this convention is "with the exception of having the final answer after four hashtags (####) for a more stable answer extraction" compared to the original CoT paper, which used different answer formatting. This standardisation is a minor but methodologically important detail: it makes the evaluation pipeline uniform across strategies, strengthening the claim that accuracy differences reflect reasoning quality rather than parsing noise.

Summary of Design Choices and Their Justifications

Per-step word budget ("5 words at most") over global token budget: Allows the number of reasoning steps to scale with problem complexity without risking mid-chain truncation or encouraging padding; avoids the compliance failures of global budgets; composes naturally with variable-step reasoning methods.
Soft guideline rather than hard enforcement: Prevents information loss from mid-step truncation while still biasing the generation distribution toward concision through instruction-following behaviour; practical for black-box API models where output token-level control is unavailable.
Author-written few-shot exemplars: Guarantees high-quality demonstrations of the target style; bridges the training-data distribution gap where concise drafts are underrepresented; avoids circularity of model-generated exemplars.
Retention of "Think step by step": Preserves the sequential decomposition mechanism shown to be effective in CoT; isolates the effect of per-step compression from the effect of having structured intermediate reasoning at all.
Shared #### separator across Standard, CoT, and CoD: Standardises answer extraction to ensure fair accuracy comparisons not confounded by parsing reliability.
No model modification, no internal state access, no fine-tuning: Maximises deployability — works with any instruction-following LLM accessible via text-completion API, including the black-box models (GPT-4o, Claude 3.5 Sonnet) on which the paper primarily evaluates.

4. Key Insights and Innovations

Innovation 1: Reasoning Verbosity as a Design Variable, Not an Inevitable Byproduct

The field's working assumption since Wei et al. (2022) has been that structured intermediate reasoning — the kind that produces accuracy gains on multi-step tasks — necessarily comes packaged in verbose, natural-language form. This wasn't stated as an explicit claim; it was an implicit consequence of how reasoning was elicited. CoT prompts produce verbose outputs because the few-shot exemplars are verbose, the training data is verbose, and no one had asked whether the verbosity was causal to the accuracy or merely correlated with it. The unexamined default was: step-by-step reasoning equals sentence-by-sentence explanation.

CoD's central conceptual move is to decouple the structural property of sequential intermediate reasoning from the stylistic property of explanatory prose. The paper demonstrates that you can preserve the former — decomposing a problem into sub-steps, externalising intermediate results so the model conditions on its own prior conclusions — while radically altering the latter. The reasoning scaffolding remains intact; what changes is that each step carries only the minimal information payload needed to advance the computation, discarding restatements, transitions, and pedagogical framing.

This is not a "be concise" patch. It's a reframing of what an intermediate reasoning step is. In CoT, a step is a miniature essay: it orients the reader, states the operation, performs it, and often interprets the result. In CoD, a step is a cognitive note-to-self — an equation, a variable update, a symbolic state change — that captures the logical content with zero expository overhead. The paper's lollipop example makes this concrete: the CoT version spends ~170 tokens explaining what subtraction is and why it applies; the CoD version writes 20 - x = 12; x = 8 and moves on. Both externalise the same mathematical operation, but only one treats the output as a public tutorial rather than a private scratchpad.

The significance of this reframing extends beyond token-count optimization. It opens a conceptual space that the field hadn't mapped: the accuracy-per-token Pareto frontier for reasoning strategies. Before CoD, the tradeoff was implicit — you either got high accuracy with verbose CoT or low accuracy with fast direct answering. CoD identifies a third region: high accuracy with low verbosity, achieved not by sacrificing reasoning depth but by changing reasoning format. The paper's Figure 1 visualises this by plotting both dimensions simultaneously, showing CoD occupying a region that neither CoT nor Standard prompting reaches. This reframing is fundamental, not incremental, because it changes what researchers should optimise for when designing reasoning strategies — not just "does it get the right answer?" but "how much output does it cost to get the right answer?"

The evidence anchoring this claim is distributed across Tables 1–4. On GSM8k with GPT-4o (Table 1), CoD achieves 91.1% accuracy using 43.9 tokens versus CoT's 95.4% using 205.1 tokens — an 80% token reduction for a 4.3 percentage point accuracy tradeoff. On coin flip (Table 4), both CoT and CoD hit 100% accuracy, but CoD does so with 16.8 tokens versus CoT's 52.4 — identical accuracy at one-third the cost. This pattern — competitive accuracy at dramatically lower token counts — is the empirical signature of a Pareto improvement in the reasoning design space.

Innovation 2: Per-Step Budgeting as a Structural Alternative to Global Token Constraints

The closest prior work — CCoT (Nayab et al., 2024) and TALE (Han et al., 2024) — shared CoD's motivation of reducing reasoning verbosity but approached the problem through global token budgets: impose a total token limit on the entire reasoning process, either fixed (CCoT) or dynamically estimated per problem (TALE). The paper identifies this as a fundamental architectural mismatch with how multi-step reasoning works, and in doing so introduces a distinct design principle: constrain the density of each reasoning unit, not the total length of the reasoning chain.

The conceptual distinction matters because problems vary in how many reasoning steps they require. A global token budget forces a zero-sum allocation: a problem that genuinely needs 10 steps to solve must either compress each step below a usable threshold (risking incoherence) or truncate the chain before reaching the answer. A problem that could be solved in 2 steps has unused budget that the model may fill with redundant content — the "overthinking" problem documented by Chiang and Lee (2024) and Chen et al. (2024b), which the paper cites. The global-budget approach treats all problems as if they have the same reasoning depth, when manifestly they do not.

Per-step budgeting inverts this logic. By limiting each step to roughly five words (a soft guideline) and imposing no limit on the number of steps, CoD lets problem complexity determine total reasoning length organically. A simple subtraction problem takes one or two steps; a multi-step arithmetic word problem takes more. The constraint operates orthogonally to problem difficulty — it controls the format of reasoning, not its extent. This is not an incremental tweak to CCoT/TALE; it's a different design philosophy that avoids the two failure modes the paper identifies in global-budget approaches: rigidity (truncation on complex problems) and compliance failure (models ignoring impractical budgets to generate verbose output anyway).

The innovation also carries a subtle but important composability property. Because CoD doesn't constrain step count, it integrates naturally with reasoning methods that require variable numbers of steps — reflection loops where the model backtracks, tool-using workflows where a step involves an API call, or self-consistency ensembles where multiple reasoning chains are generated in parallel. A global budget would need to pre-allocate tokens for these dynamic behaviours; a per-step budget adapts automatically. The paper doesn't demonstrate these compositions, but the architectural compatibility is a direct consequence of the design choice.

Evidence for the effectiveness of this approach is indirect but clear: across all four benchmark tasks and both models (Tables 1–4), CoD's output token counts are dramatically lower than CoT without any evidence of truncation-related accuracy collapse. If the per-step constraint were causing models to cut off essential reasoning, we would expect accuracy to degrade sharply — but it doesn't, except in the specific failure cases the paper documents and analyses in Section 4.5 (zero-shot setting, small models), which are attributed to training data distribution gaps rather than fundamental flaws in the per-step mechanism.

Innovation 3: The Diagnose-and-Reveal Move — Why CoD Fails in Zero-Shot and Small-Model Settings

Perhaps the paper's most intellectually interesting contribution is not the success case but the pattern of failure documented in Section 4.5. When CoD is applied zero-shot (no few-shot exemplars) to GSM8k with Claude 3.5 Sonnet, accuracy collapses from 91.4% (few-shot CoD) to 65.5% — only 3.6 percentage points above Standard prompting's 61.9% (Table 5). On small models (<3B parameters), the CoD-to-CoT accuracy gap widens substantially: Qwen2.5-3B-Instruct achieves 59.1% with CoT versus 43.1% with CoD; Llama3.2-3B-Instruct achieves 70.7% versus 52.5% (Table 6).

This failure pattern is revealing because it diagnoses a property of the pretraining data distribution through a purely behavioural experiment. CoD works when the model has in-context demonstrations of the draft style to imitate. It fails when the model must generate that style from its parametric knowledge alone. This implies — and the paper explicitly hypothesises — that concise, draft-style reasoning patterns are underrepresented in the pretraining corpora of current LLMs. The models have been trained on verbose explanations (textbook solutions, StackExchange answers, tutorial transcripts) and do not possess a strong internal prior for generating 20 - x = 12; x = 8 as a natural reasoning format.

This is a diagnostic contribution rather than a performance contribution. It uses CoD as a probe to reveal something about the training data that wasn't previously measured: the degree to which models associate "step-by-step reasoning" with "verbose natural language explanation" is not a necessary cognitive property but a learned statistical association that can be overridden with sufficient in-context exemplars. The implication is that reasoning verbosity is in substantial part an artefact of training data, not an intrinsic requirement of effective computation. If models were trained on data that included concise symbolic reasoning alongside verbose explanations — the way human educational materials include both expanded textbook derivations and terse margin notes — the zero-shot gap might close or disappear entirely.

The small-model results extend this diagnosis. Smaller models have less capacity to bridge the distribution gap through in-context learning alone; the few-shot exemplars provide a template, but the model's weaker instruction-following and pattern-matching capabilities mean the template is less faithfully reproduced. The paper treats this as evidence that fine-tuning on CoD-formatted data could substantially close the gap, a hypothesis that remains untested but follows naturally from the diagnostic logic.

This is a fundamental insight — not an incremental finding — because it shifts the conversation from "how do we prompt models to be concise?" to "how were models trained to associate reasoning with verbosity in the first place?" It suggests that the efficiency problem CoD addresses is not primarily a prompting problem but a training data curation problem, and that the long-term solution is likely to involve including concise reasoning exemplars in pretraining and instruction-tuning corpora rather than relying entirely on prompt engineering to override the training distribution at inference time.

Innovation 4: Accuracy-Per-Token as a First-Class Evaluation Metric for Reasoning Strategies

Prior work on LLM reasoning — from CoT (Wei et al., 2022) through Tree-of-Thoughts (Yao et al., 2024), Graph-of-Thoughts (Besta et al., 2024), self-consistency (Wang et al., 2022), and the reasoning-trained model family (o1, R1, QwQ) — evaluates reasoning strategies almost exclusively on accuracy. Efficiency, when acknowledged at all, is treated as an implementation detail or reported as a secondary observation. The field's implicit evaluation framework is: a reasoning method is better if it achieves higher benchmark scores, regardless of computational cost.

CoD doesn't just achieve a favourable accuracy-efficiency tradeoff — it argues, through its experimental design and presentation, that efficiency should be a co-equal dimension of evaluation for reasoning strategies. The paper's Figure 1 is the clearest manifestation of this: it plots accuracy against token usage as a two-dimensional evaluation space, showing that CoD occupies a region that CoT cannot reach. This is not a claim about CoD's superiority on either axis individually — CoT wins on accuracy in most settings, Standard wins on token count — but about the existence of a Pareto frontier that any reasoning strategy can be positioned on.

This reframes what "better" means for reasoning methods. Under an accuracy-only evaluation, CoT strictly dominates CoD on several tasks (95.4% vs. 91.1% on GSM8k with GPT-4o). Under a token-only evaluation, Standard prompting dominates both (1.1 tokens vs. 43.9 for CoD vs. 205.1 for CoT). But under a joint accuracy-per-token evaluation, CoD is non-dominated — there is no single strategy that is simultaneously more accurate and more token-efficient. This is the language of multi-objective optimisation, and importing it into reasoning evaluation is a conceptual contribution independent of CoD's specific performance numbers.

The practical significance is that this framework legitimises tradeoffs that practitioners were already making implicitly. A production chatbot serving millions of users may prefer 91% accuracy at 1.0s latency over 95% accuracy at 4.2s latency; a batch evaluation pipeline processing a fixed dataset may have the opposite preference. By making the tradeoff explicit and measurable, the paper provides a vocabulary for reasoning about these decisions that the field previously lacked.

This is an incremental contribution to evaluation methodology rather than a fundamental theoretical advance, but it is significant because it addresses a gap between how reasoning research is evaluated and how reasoning systems are deployed. The metric — output tokens per correct answer, or some combined efficiency-accuracy score — is not formally defined in the paper, leaving room for future work to standardise it. But the conceptual move of treating efficiency as a first-class dimension is the key insight.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three categories of reasoning tasks drawn from established benchmarks, mirroring the evaluation setup of the original CoT paper (Wei et al., 2022). For arithmetic reasoning, GSM8k (Cobbe et al., 2021) provides 8,500 diverse grade-school-level math word problems, each with a detailed step-by-step solution. For commonsense reasoning, the paper uses the date understanding and sports understanding tasks from BIG-bench (bench authors, 2023). For symbolic reasoning, the coin flip task introduced in the CoT paper is used; since the exact dataset was not published, the authors synthesize a test set of 250 examples following the same design — randomly selecting 4 out of the top 1,000 US first names from NameDataset (Remy, 2021) and randomly deciding whether each person flips the coin or not. The exact sizes of the BIG-bench task splits are not specified in the paper, but they are standard subsets used in prior CoT evaluations.
Base model(s). All primary experiments use two flagship API-accessible models: GPT-4o (gpt-4o-2024-08-06) from OpenAI and Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) from Anthropic. These are chosen because they represent state-of-the-art instruction-following LLMs accessible to practitioners via black-box APIs — a deliberate choice that demonstrates CoD works without requiring model internals, fine-tuning, or architectural modifications. The limitation experiments (Section 4.5, Table 6) additionally test four small open-weight models: Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct (Yang et al., 2024), Llama3.2-3B-Instruct (Dubey et al., 2024), and Zoom-SLM-2.3B (Zoom, 2025), all with fewer than 3B parameters, to probe whether CoD's effectiveness scales down to models with weaker in-context learning capabilities.
Metrics. Three metrics are reported for each experiment: accuracy (the fraction of test questions for which the extracted final answer matches the ground truth — extraction uses everything after the #### separator), average output token count (the mean number of tokens generated per response, including both reasoning steps and the final answer), and average latency (wall-clock time per response, reported in seconds). The combination of accuracy and token count is the paper's operationalization of the accuracy-per-token efficiency tradeoff that it positions as a first-class evaluation dimension (see Innovation 4 in Section 4). Latency is reported for practical intuition but is acknowledged to be deployment-environment-dependent. The paper does not report confidence intervals or statistical significance tests for any of these metrics.
Baselines. Two primary baselines are compared against CoD:
- Standard prompting: Few-shot prompting where the model is instructed to "Answer the question directly. Do not return any preamble, explanation, or reasoning." The model receives input-output exemplars and is asked to return only the final answer. This represents the efficiency ceiling (minimal tokens) but accuracy floor for multi-step reasoning tasks.
- Chain-of-Thought (CoT; Wei et al., 2022): Few-shot prompting with the instruction "Think step by step to answer the following question. Return the answer at the end of the response after a separator ####." The model receives the same few-shot exemplars as in the original CoT paper (with the answer format modified to use #### for consistent extraction). This represents the accuracy ceiling, with verbose reasoning as the cost.
Both baselines use the same #### separator convention for answer extraction, ensuring that parsing reliability does not confound comparisons (Section 4.1). The paper does not evaluate against the closest prior efficiency methods — CCoT (Nayab et al., 2024), TALE (Han et al., 2024), Coconut (Hao et al., 2024), or Skeleton-of-Thought (Ning et al., 2023) — as baselines in the experimental section; these comparisons exist only in the conceptual positioning of Section 2.
Generation budget / compute accounting. There is no explicit compute budget control or FLOPs accounting in this paper. All comparisons are at the level of naturally occurring output token counts — the model generates however many tokens it produces under each prompting strategy, and efficiency is measured post hoc by comparing average token counts and latencies. This is a fundamentally different approach from the "fixed generation budget" framework used in compute-optimal test-time scaling work: CoD does not allocate a budget and then optimize within it; rather, it reshapes the model's output distribution so that the naturally generated responses are more concise. The implicit assumption is that token count is the relevant cost metric for API-based deployments where pricing is per-token and latency scales with output length. No attempt is made to equalize total FLOPs or token budgets across strategies for a controlled comparison — the paper's claim is precisely that CoD achieves comparable accuracy without needing the same token budget.
Cross-validation / statistical protocol. The paper does not employ cross-validation, statistical significance testing, or multiple random seeds. Each experimental configuration (model × prompt strategy × task) is evaluated once on the standard test set for that task. The coin flip task, being synthetically generated, uses a single random seed for test set construction (random name selection and flip assignment) with no replication across different random draws. The paper does not report variance estimates for accuracy, token counts, or latency. For zero-shot experiments, no few-shot exemplars are provided, making each test question an independent evaluation but with no cross-validation over exemplar selection (since there are no exemplars to select).

Main Quantitative Results

The experimental results are organized by reasoning category, with each of the three categories (arithmetic, commonsense, symbolic) evaluated on both GPT-4o and Claude 3.5 Sonnet under all three prompting strategies. The paper's headline finding is visible across all categories: CoD achieves accuracy competitive with CoT — sometimes matching it exactly — while reducing output token counts by 68–92% and latency by 48–76%.

Arithmetic Reasoning (GSM8k)

Table 1 presents the GSM8k results. The headline numbers:

GPT-4o: Standard prompting achieves 53.3% accuracy with 1.1 tokens and 0.6s latency. CoT achieves 95.4% accuracy with 205.1 tokens and 4.2s latency. CoD achieves 91.1% accuracy with 43.9 tokens and 1.0s latency — a 78.6% reduction in tokens relative to CoT (from 205.1 to 43.9) and a 76.2% reduction in latency (from 4.2s to 1.0s), at a cost of 4.3 percentage points of accuracy (95.4% → 91.1%).
Claude 3.5 Sonnet: Standard achieves 64.6% accuracy (1.1 tokens, 0.9s). CoT achieves 95.8% accuracy (190.0 tokens, 3.1s). CoD achieves 91.4% accuracy (39.8 tokens, 1.6s) — a 79.1% token reduction and 48.4% latency reduction, for a 4.4 percentage point accuracy tradeoff.

Notably, CoD's token count (43.9 and 39.8) is roughly 40× the Standard prompting token count (1.1), confirming that CoD does generate intermediate reasoning — it is not simply reverting to direct answering. The accuracy gap between CoD and Standard (91.1% vs. 53.3% for GPT-4o; 91.4% vs. 64.6% for Claude) is large, demonstrating that the concise reasoning steps are doing meaningful cognitive work despite their brevity.

The paper does not report accuracy broken down by problem difficulty within GSM8k, so it is impossible to assess from these numbers whether the 4-point accuracy gap between CoD and CoT is concentrated on the hardest problems (where verbose reasoning might be most necessary) or distributed uniformly.

Commonsense Reasoning

The commonsense reasoning evaluation covers two BIG-bench tasks: date understanding (Table 2) and sports understanding (Table 3). Unlike GSM8k, where CoD slightly underperforms CoT, these tasks reveal cases where CoD matches or exceeds CoT accuracy while maintaining large token reductions.

Date understanding (Table 2):

GPT-4o: Standard: 72.6% (5.2 tokens, 0.6s). CoT: 90.2% (75.7 tokens, 1.7s). CoD: 88.1% (30.2 tokens, 1.3s) — a 60.1% token reduction relative to CoT, with a 2.1-point accuracy gap.
Claude 3.5 Sonnet: Standard: 84.3% (5.2 tokens, 1.0s). CoT: 87.0% (172.5 tokens, 3.2s). CoD: 89.7% (31.3 tokens, 1.4s) — CoD outperforms CoT by 2.7 percentage points while reducing tokens by 81.9% (from 172.5 to 31.3). This is the paper's clearest case of CoD simultaneously improving both accuracy and efficiency — a strict Pareto improvement over CoT on this task-model combination.

The Claude 3.5 Sonnet result on date understanding is particularly striking because CoT's token count (172.5) is anomalously high relative to GPT-4o's CoT (75.7) — Claude produces substantially more verbose reasoning by default — yet CoD brings it down to 31.3, slightly below GPT-4o's CoD token count of 30.2, while achieving higher accuracy. This suggests that Claude's default verbosity on this task is especially wasteful, and CoD's compression is particularly beneficial.

Sports understanding (Table 3):

GPT-4o: Standard: 90.0% (1.0 tokens, 0.4s). CoT: 95.9% (28.7 tokens, 0.9s). CoD: 98.3% (15.0 tokens, 0.7s) — CoD outperforms CoT by 2.4 points while reducing tokens by 47.7%.
Claude 3.5 Sonnet: Standard: 90.6% (1.0 tokens, 0.9s). CoT: 93.2% (189.4 tokens, 3.6s). CoD: 97.3% (14.3 tokens, 1.0s) — CoD outperforms CoT by 4.1 points while reducing tokens by 92.4% (from 189.4 to 14.3) and latency by 72.2% (3.6s to 1.0s). This 92.4% token reduction is the paper's most dramatic efficiency gain — representing the "as little as only 7.6% of the tokens" claim from the paper's abstract and title.

The sports understanding results are the strongest evidence in the paper for CoD's central claim that "effective reasoning in LLMs does not necessarily require lengthy outputs" (Section 5). On this task, CoD achieves both the highest accuracy and the lowest reasoning token count (among the three strategies that produce intermediate reasoning) for both models — a strict dominance that cannot be dismissed as "slightly worse accuracy for much lower cost." The accuracy improvement over CoT, while modest in absolute points, is in the direction opposite to what one would expect if concision traded off against reasoning quality.

The paper does not analyze why CoD sometimes outperforms CoT — whether because verbose reasoning introduces confusion or contradiction, because the draft format forces more precise thinking, or because of some other mechanism. This is a notable analytical gap.

Symbolic Reasoning (Coin Flip)

Table 4 presents the coin flip results. This is the only task where CoD achieves perfect accuracy (100%) — matching CoT exactly — for both models:

GPT-4o: Standard: 73.2% (1.0 tokens, 0.4s). CoT: 100.0% (52.4 tokens, 1.4s). CoD: 100.0% (16.8 tokens, 0.8s) — a 67.9% token reduction with zero accuracy cost.
Claude 3.5 Sonnet: Standard: 85.2% (1.0 tokens, 1.2s). CoT: 100.0% (135.3 tokens, 3.1s). CoD: 100.0% (18.9 tokens, 1.6s) — an 86.0% token reduction with zero accuracy cost.

The coin flip task is the cleanest demonstration of CoD's core thesis: when the reasoning structure is simple and repetitive (tracking a binary state through a sequence of named agents, each either flipping or not flipping), verbose natural-language explanation adds no decision-relevant information. The model needs only to track the state (H → T → H → H or equivalent) — exactly the kind of minimal symbolic manipulation that CoD encourages. The fact that CoT uses 52–135 tokens for a task whose logical content can be captured in 5–10 symbols underscores the paper's argument about wasteful verbosity.

A subtle observation: the Standard prompting baseline achieves non-trivial accuracy on coin flip (73.2% GPT-4o, 85.2% Claude), substantially higher than Standard on GSM8k (53.3%, 64.6%). This suggests that coin flip is an easier task where the model can sometimes track state internally without externalised reasoning. CoT and CoD both push this to 100%, indicating that the remaining gap for Standard prompting comes from state-tracking failures that even minimal externalisation (CoD's ~17-19 tokens) resolves.

Cross-Cutting Observations

Several patterns emerge when examining the results across all four tasks and both models:

1. CoD's accuracy relative to CoT is task-dependent. On GSM8k, CoD trails CoT by ~4 points for both models. On date understanding, CoD roughly matches CoT (within 2 points, and outperforms for Claude). On sports understanding, CoD outperforms CoT by 2–4 points. On coin flip, both achieve 100%. This variation suggests that the accuracy cost of concision depends on problem structure — tasks with simpler logical structures (coin flip, sports understanding) may benefit from draft-style compression because it reduces opportunities for the model to confuse itself with verbose elaboration, while tasks requiring precise multi-step arithmetic (GSM8k) may benefit from the additional "working memory" that verbose intermediate steps provide. The paper does not analyze this task-dependence, but the data pattern is consistent with this interpretation.

2. Claude 3.5 Sonnet exhibits more dramatic token reduction under CoD than GPT-4o. Across all four tasks, Claude's default CoT outputs are substantially more verbose than GPT-4o's (e.g., 189.4 vs. 28.7 tokens on sports understanding CoT; 172.5 vs. 75.7 on date understanding), yet CoD brings both models to similar token counts (~14-44 tokens depending on the task). This means CoD's relative efficiency gain is larger for Claude — the 92.4% token reduction on sports understanding — because Claude's baseline verbosity is higher. This is consistent with the interpretation that CoD acts as a verbosity "equalizer," compressing outputs toward a task-determined minimum regardless of model-specific tendencies toward elaboration.

3. Latency reductions track token reductions but imperfectly. On GSM8k with GPT-4o, tokens drop 78.6% while latency drops 76.2% — roughly proportional. On GSM8k with Claude, tokens drop 79.1% but latency drops only 48.4% (from 3.1s to 1.6s). This asymmetry reflects that latency is not purely proportional to output token count — it includes input processing time, model architecture factors, and API overhead — but the paper does not analyze these components. The latency numbers should be interpreted as illustrative of practical deployment impact rather than as precisely controlled measurements.

4. The "as little as only 7.6% of the tokens" claim. This headline number from the abstract corresponds to the sports understanding task with Claude 3.5 Sonnet: CoD uses 14.3 tokens versus CoT's 189.4 tokens, a ratio of 14.3/189.4 ≈ 7.6%. This is the most extreme case in the paper's results. Across all tasks and models, CoD's token usage ranges from ~8% to ~40% of CoT's, depending on the combination. The 7.6% figure is accurate as a lower bound but should not be interpreted as typical; the average reduction is closer to 75-85% for Claude and 50-80% for GPT-4o across the tasks evaluated.

Ablation Studies and Robustness Checks

The paper's ablation studies are concentrated in Section 4.5 ("Limitations of CoD") and take the form of testing CoD under two degraded conditions: zero-shot prompting (no few-shot exemplars) and deployment on small models (<3B parameters). These are not traditional ablations that isolate specific design choices (e.g., "5 words vs. 10 words," "soft vs. hard constraint," "author-written vs. model-generated exemplars") — rather, they probe the boundary conditions under which CoD ceases to be effective.

Zero-shot prompting (Table 5): Removing few-shot exemplars degrades CoD substantially on GSM8k for both models:

GPT-4o: Standard: 56.9% (2.2 tokens). CoT: 94.8% (278.4 tokens). CoD: 84.4% (76.4 tokens). The CoD-to-CoT accuracy gap widens from 4.3 points (few-shot) to 10.4 points (zero-shot). Token reduction is still significant relative to CoT (72.6% fewer tokens), but accuracy drops noticeably.
Claude 3.5 Sonnet: Standard: 61.9% (5.2 tokens). CoT: 90.4% (248.8 tokens). CoD: 65.5% (73.7 tokens). The CoD-to-CoT gap is 24.9 points — CoD is only 3.6 points above Standard prompting. Token reduction relative to CoT is 70.4%, but the accuracy collapse makes this largely irrelevant.

The authors hypothesize that this failure "arises due to the scarcity or absence of CoD-style reasoning patterns in the training data of large language models, making it a challenging task to generate concise and insightful 'drafts' without guidance from few-shot examples." This is a plausible interpretation but is not directly tested — the paper does not, for example, train a model on CoD-formatted data and compare zero-shot performance before and after such training. The zero-shot result is a diagnostic observation rather than a controlled ablation establishing causality.

An important nuance: zero-shot CoT also degrades relative to few-shot CoT (the paper does not report few-shot GSM8k CoT numbers directly in Table 5 for easy comparison, but the few-shot results from Table 1 are 95.4% GPT-4o, 95.8% Claude — comparing these to zero-shot CoT in Table 5 shows drops to 94.8% and 90.4% respectively). The CoT degradation is modest, while the CoD degradation is severe, suggesting that CoD is more dependent on few-shot exemplars than CoT — consistent with the hypothesis that the draft style is out-of-distribution for the model's pretraining.

Small model evaluation (Table 6): Testing on four models with fewer than 3B parameters (Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Llama3.2-3B-Instruct, Zoom-SLM-2.3B) on GSM8k reveals a consistent pattern: CoD improves over Standard prompting but substantially underperforms CoT:

Qwen2.5-1.5B-Instruct: CoT 32.5% vs. CoD 24.2% (8.3-point gap)
Qwen2.5-3B-Instruct: CoT 59.1% vs. CoD 43.1% (16.0-point gap)
Llama3.2-3B-Instruct: CoT 70.7% vs. CoD 52.5% (18.2-point gap)
Zoom-SLM-2.3B: CoT 77.7% vs. CoD 50.9% (26.8-point gap)

The gap varies by model — from 8.3 to 26.8 points — with no clear relationship to model size within this narrow range. CoD does consistently reduce token counts (e.g., Zoom-SLM-2.3B: 129.0 CoT → 55.6 CoD), but the accuracy cost is substantial. The paper hypothesizes that fine-tuning these models with CoD-formatted data "could significantly enhance their reasoning accuracy with CoD," but this experiment is not conducted.

There are no ablations reported for the core design parameters of CoD itself:

The "5 words at most" threshold: What happens with 3 words? 10 words? Is there a smooth tradeoff or a threshold effect?
Soft vs. hard constraint: What if the constraint were enforced by post-processing (truncating outputs after 5 words per line)? Would accuracy drop?
Author-written vs. model-generated exemplars: Could the model generate its own CoD exemplars for a given task, enabling zero-shot deployment?
Per-step vs. per-line vs. per-sentence constraint granularity: Does the word limit applied to "each thinking step" behave differently than a limit on each line, each equation, or each sentence?
Interaction with model scale: The paper tests only two large API models and four small models. What happens at intermediate scales (7B, 13B, 70B)? Is CoD's effectiveness monotonic with model size?

The paper also does not report ablations that would clarify why CoD sometimes outperforms CoT in accuracy (sports understanding, date understanding with Claude, coin flip). Potential mechanisms — reduced self-contradiction, less distraction from irrelevant problem details, forced precision in step formulation — are not isolated or tested.

Critical Assessment

The experimental results provide clear evidence for CoD's practical value proposition: substantial token and latency reduction with modest accuracy tradeoffs on large API-accessible models when few-shot exemplars are provided. The paper's central claim — that effective multi-step reasoning does not require verbose intermediate outputs — is supported by the fact that CoD achieves 91-100% of CoT's accuracy (or higher) across all tested tasks while using 8-40% of the tokens. However, several important limitations constrain the generality and strength of this evidence.

The claim holds conditionally, and the conditions are narrow. CoD works well only when (a) few-shot exemplars demonstrating the draft style are provided, and (b) the underlying model is large enough to reliably follow the per-step constraint through in-context learning. The zero-shot results (Table 5) and small-model results (Table 6) demonstrate that removing either condition causes CoD to substantially underperform CoT. This is not a fatal weakness — few-shot prompting with large API models is a common deployment pattern — but it means the paper's efficiency claims do not generalize to zero-shot settings or to the growing ecosystem of smaller, locally-deployed models without additional intervention (fine-tuning on CoD-formatted data, which the paper hypothesizes but does not test).

The evaluation is limited to two models from a single generation of API providers. GPT-4o and Claude 3.5 Sonnet are both state-of-the-art instruction-tuned models from mid-2024. The paper does not evaluate on any open-weight models at comparable scale (e.g., Llama 3.1 70B, Mixtral 8x22B), on reasoning-specialized models (o1, DeepSeek-R1, QwQ), or on older models where in-context learning capabilities might be weaker. This means we cannot assess whether CoD's effectiveness is specific to the training recipes of OpenAI and Anthropic models or whether it generalizes across model families. The absence of reasoning-specialized models is particularly notable given the paper's positioning in the introduction: o1 and R1 are mentioned as motivating the latency problem, but they are not evaluated, leaving open the question of whether CoD-like concision is already internalized in their reasoning processes or whether prompting them with CoD yields further gains.

The evaluation is limited to three task categories, all from the original CoT paper's evaluation suite. GSM8k, BIG-bench date/sports understanding, and coin flip are all relatively simple reasoning tasks by 2025 standards. GSM8k, in particular, saturates at ~95% for large models with CoT, leaving limited headroom to measure accuracy differences. The paper does not evaluate on more challenging reasoning benchmarks (MATH, MMLU, GPQA, HumanEval for code, multi-step tool-use scenarios, or long-context reasoning tasks) where the relationship between reasoning verbosity and accuracy might be qualitatively different. On harder problems, the extra tokens spent by CoT on detailed derivations, error-checking, and alternative-approach exploration might be essential rather than wasteful, and CoD's compression could impose a larger accuracy cost than the ~4-point gap observed on GSM8k.

The metrics lack statistical rigor. No confidence intervals, standard deviations, or significance tests are reported for any accuracy, token count, or latency measurement. For the GSM8k test set (presumably the standard 1,319-example split, though the paper does not specify), a 4-percentage-point difference represents approximately 53 examples — a difference that may or may not be statistically significant depending on the variance structure, which is unreported. For the BIG-bench tasks, the test set sizes are unspecified. For the coin flip task, the synthetic test set has only 250 examples, making the 100% accuracy claims somewhat fragile — a single additional failure would not change the qualitative conclusion but would mean the accuracy is not truly perfect. The absence of any statistical reporting makes it impossible to assess whether the observed accuracy differences between CoD and CoT are reliable or could be due to sampling variation.

The "no compute budget control" design is both a feature and a bug. The paper measures what naturally happens under each prompting strategy rather than equalizing total compute. This is appropriate for the paper's practical argument — "here's what you get if you use CoD instead of CoT in your API calls" — but it means the comparison is not a controlled experiment in the traditional sense. We cannot distinguish whether CoD's accuracy is slightly lower than CoT's on GSM8k because concision inherently trades off against reasoning quality, or because the model is simply doing less total computation (fewer FLOPs) and would match CoT's accuracy if allowed comparable compute. An experiment that gave CoD a larger number of parallel reasoning chains (self-consistency over CoD drafts) to match CoT's token budget would address this — if CoD with budget-matched self-consistency matched or exceeded CoT accuracy, the case for draft-style reasoning would be stronger. This experiment is not reported.

The cases where CoD outperforms CoT are not analyzed. On sports understanding (both models) and date understanding (Claude), CoD achieves higher accuracy than CoT. This is potentially the most interesting finding in the paper — it suggests that verbosity can actively harm reasoning in some contexts — but the paper provides no analysis of why. Do verbose outputs introduce contradictions? Does the model lose track of the core logical thread when it elaborates? Are there specific failure modes in CoT that CoD avoids? Without error analysis, the accuracy advantage remains an intriguing observation rather than an explained phenomenon that could guide further method development.

Missing experiments that would substantially strengthen the paper:

Difficulty-stratified accuracy analysis: Does CoD's accuracy gap relative to CoT widen on harder problems within each task? If so, the tradeoff is not uniform, and deployment guidance should account for this.
Self-consistency over CoD: If CoD generates 5× fewer tokens per chain, running 5 CoD chains in parallel and majority-voting would cost roughly the same as one CoT chain. Would this match or exceed CoT accuracy? This is the natural experiment for testing whether token efficiency translates to compute efficiency.
Intermediate-scale models: Does CoD work on Llama-3.1-8B or 70B? Qwen2.5-7B or 32B? The small-model results show a gap, but there is a large uncharted territory between 3B and GPT-4o scale.
Harder reasoning benchmarks: MATH, GPQA, MMLU, BBH — does the 4-point accuracy gap hold, widen, or narrow?
Reasoning-trained models: o1, o1-mini, DeepSeek-R1 — these models are explicitly optimized for chain-of-thought reasoning. Does CoD prompting help, hurt, or have no effect on them? Is their internal reasoning already draft-like?
Ablation of the "5 words" threshold: Is there a smooth accuracy-efficiency curve as the per-step word limit varies from 3 to 15? Is 5 a sweet spot or an arbitrary choice?
Error analysis: For questions where CoD gets the wrong answer but CoT gets the right one, what goes wrong? Is the reasoning truncated, or is a step simply wrong, or does the compressed format cause a parsing error?
Human evaluation of CoD drafts: Are the generated drafts interpretable to humans? The paper claims interpretability as an advantage over latent reasoning (Coconut), but does not verify that CoD's terse equations and symbolic manipulations are actually readable.

In summary, the experiments demonstrate that CoD is a practically useful prompting strategy for reducing cost and latency on multi-step reasoning tasks when using large API-accessible models with few-shot exemplars. The core insight — that reasoning verbosity is not necessary for accuracy — is supported on the tasks and models tested. However, the evaluation is narrow in model coverage, task diversity, and statistical rigor. The boundary conditions (zero-shot, small models) are well-documented, but the mechanisms underlying both the successes (accuracy parity or improvement) and failures (accuracy gap on GSM8k, zero-shot collapse) are not empirically investigated. The paper successfully establishes CoD as a point on the accuracy-efficiency Pareto frontier but leaves open the question of how broadly that frontier extends and what design parameters control its shape.

6. Limitations and Trade-offs

6.1 The Concision Constraint Is Ineffective Without Few-Shot Exemplars

The assumption or constraint. CoD assumes that the model can generate concise, draft-style reasoning steps in response to the system prompt instruction alone. The "5 words at most" guideline is presented as a general-purpose prompting strategy that reshapes reasoning output through instruction-following. However, Section 4.5 reveals a sharp boundary condition: the approach depends critically on the presence of hand-crafted few-shot exemplars that demonstrate the draft format. The paper explicitly hypothesizes why:

"this limitation arises due to the scarcity or absence of CoD-style reasoning patterns in the training data of large language models, making it a challenging task to generate concise and insightful 'drafts' without guidance from few-shot examples."

The consequence. In zero-shot settings — which represent a substantial fraction of real-world LLM usage, including many chatbot deployments, API integrations where prompt length is constrained, and agentic workflows where few-shot exemplars are impractical to maintain per task — CoD fails to deliver its promised accuracy-efficiency tradeoff. On GSM8k with Claude 3.5 Sonnet (Table 5), zero-shot CoD achieves only 65.5% accuracy — a mere 3.6 percentage points above Standard prompting (61.9%) and 24.9 points below CoT. Even with GPT-4o, the gap between CoD and CoT widens from 4.3 points (few-shot) to 10.4 points (zero-shot). The token reductions persist (70-73% fewer tokens than CoT in zero-shot), but the accuracy collapse means these savings come at an unacceptable cost for most applications. This means CoD is not a replacement for CoT as a general-purpose reasoning strategy — it is a specific few-shot prompt template that must be manually constructed for each task domain and included in every API call, consuming input token budget that partially offsets the output token savings.

What evidence exists in the paper. Table 5 provides the zero-shot GSM8k comparison across both models and all three prompting strategies. The evidence is clear: CoD's accuracy advantage over Standard prompting nearly vanishes for Claude (3.6 points) and substantially degrades for GPT-4o (from a 37.8-point few-shot advantage over Standard to a 27.5-point zero-shot advantage). The paper does not report zero-shot results for the other three tasks (date understanding, sports understanding, coin flip), so we cannot assess whether this degradation is task-dependent or universal.

Mitigation status. The paper does not attempt to address this limitation. It acknowledges the hypothesis (training data distribution gap) and suggests that fine-tuning on CoD-formatted data could help, but conducts no fine-tuning experiments. The limitation is presented as an observation in Section 4.5 without a proposed solution beyond the forward-looking hypothesis. A practitioner currently cannot deploy CoD zero-shot and expect reliable results; they must invest in writing task-specific few-shot exemplars.

6.2 The Accuracy Gap on Arithmetic Reasoning Is Persistent and Unexplained

The assumption or constraint. The paper's central claim is that CoD "matches or surpasses CoT in accuracy" (Abstract) and "maintains or even improves accuracy compared with standard Chain of Thought" (Introduction). However, on the most practically significant and widely benchmarked reasoning task — GSM8k arithmetic reasoning — CoD consistently underperforms CoT by approximately 4 percentage points for both models (Table 1: 91.1% vs. 95.4% for GPT-4o; 91.4% vs. 95.8% for Claude). This is not a marginal difference: on the GSM8k test set, it represents roughly 50-60 additional incorrect answers that CoT would have gotten right.

The consequence. The 4-point gap creates a genuine tradeoff that the paper's framing obscures. A practitioner deciding between CoT and CoD for arithmetic reasoning faces a real accuracy cost — not a trivial one — in exchange for the ~80% token reduction. Whether this tradeoff is acceptable depends on the application: for a tutoring system where correctness is paramount, a 4-point drop might be unacceptable regardless of cost savings; for a high-volume batch processing pipeline where 91% accuracy is sufficient and costs dominate, CoD is clearly preferable. The paper's rhetorical emphasis on cases where CoD matches or exceeds CoT (sports understanding, coin flip, date understanding with Claude) draws attention away from the task — arithmetic reasoning — that is simultaneously the most benchmarked, the most commercially relevant (math tutoring, financial calculation, data analysis), and the one where CoD's accuracy tradeoff is clearest. The abstract's claim that CoD "matches or surpasses CoT in accuracy" is technically true for some tasks but misleading as a general characterization given the GSM8k results.

What evidence exists in the paper. Table 1 provides the GSM8k accuracy numbers. The paper does not analyze why the gap exists — e.g., whether errors are concentrated on harder problems within GSM8k, whether the compressed format leads to calculation errors that verbose reasoning would catch, whether certain problem types (multi-step, problems requiring unit conversion, problems with irrelevant information) are disproportionately affected. No error analysis or difficulty-stratified breakdown is provided. The absence of this analysis means we cannot assess whether the gap is fundamental (concision inherently impairs arithmetic precision) or fixable (specific prompt refinements could close it).

Mitigation status. The paper does not acknowledge this as a systematic limitation or attempt to analyze its causes. The GSM8k results are presented as a success case ("91.1%... thereby reducing the average output token count by 80%") without discussing whether the 4-point gap could be narrowed through prompt engineering, combining CoD with self-consistency (majority voting over multiple CoD chains), or other interventions. The tradeoff is left for the reader to evaluate ad hoc.

6.3 The Headline Token Reduction Numbers Are an Overestimate for Practical Deployments

The assumption or constraint. The paper reports token reduction as the ratio of CoD's average output tokens to CoT's average output tokens, with the most dramatic figure — CoD uses "as little as only 7.6% of the tokens" (Abstract) — corresponding to the Claude 3.5 Sonnet sports understanding result (14.3 vs. 189.4 tokens; Table 3). This calculation accounts only for output tokens — the tokens the model generates in response. It does not account for the input tokens consumed by the few-shot exemplars that CoD requires to function effectively.

The consequence. The few-shot exemplars that make CoD work (Section 4.5 demonstrates they are essential) consume input tokens that are billed and processed on every API call. If the CoD exemplars are comparable in length to the CoT exemplars (both are manually written by the authors and include reasoning chains, just in different styles), then the input token cost is similar across strategies. However, the relative importance of input tokens depends on the ratio of input to output tokens. For short queries with long exemplar sets, input tokens dominate total cost, and CoD's output token savings become less significant as a fraction of total spend. For long, multi-turn conversations where exemplars are amortized across many queries, output savings dominate. The paper does not report exemplar token counts, so a practitioner cannot compute the total token cost (input + output) of CoD versus CoT for their specific deployment pattern.

For a concrete illustration: if the few-shot exemplars for GSM8k consume 500 input tokens per query (a typical number for 5-8 exemplars of word problems with solutions), then the total token cost per query is roughly 500 + 205 (input + output) for CoT and 500 + 44 for CoD. The total token reduction is from 705 to 544 — a 23% reduction, not the 79% reduction reported in Table 1 based on output tokens alone. The input token costs partially eat into the efficiency gains, and the magnitude of this effect depends on exemplar length relative to output length — information the paper does not provide.

What evidence exists in the paper. None. The paper does not report input token counts for any experiment, does not factor exemplar length into efficiency calculations, and does not discuss the input-vs-output token tradeoff. The exemplars themselves are not reproduced in the main text (they are presumably available in the linked code repository), making it impossible for a reader to estimate the input token burden from the paper alone.

Mitigation status. The paper does not address input token costs. A full accounting would report total tokens (input + output) per query and compute the efficiency ratio on that basis. Future work could explore whether CoD enables shorter exemplars (since the drafts are themselves more concise) — if CoD exemplars are substantially shorter than CoT exemplars, the total-token efficiency gains might be even larger than the output-only numbers suggest. But this is not measured or discussed.

6.4 Evaluation Scope Is Too Narrow to Support General Claims About Reasoning Efficiency

The assumption or constraint. The paper evaluates CoD on three task categories — arithmetic reasoning (GSM8k), commonsense reasoning (two BIG-bench subsets), and symbolic reasoning (coin flip) — using two API models (GPT-4o and Claude 3.5 Sonnet). All four tasks are drawn from the original CoT paper's evaluation suite (Wei et al., 2022). The paper positions CoD as a general reasoning strategy and makes claims like "effective reasoning in LLMs does not necessarily require lengthy outputs" (Section 5) and "this minimalist approach maintains or even improves accuracy compared with standard Chain of Thought" (Introduction) without qualifying the limited scope of the evidence base.

The consequence. Several important deployment regimes are completely untested, and existing evidence suggests they may be problematic:

Harder reasoning benchmarks: GSM8k is a grade-school math benchmark that saturates at ~95% for large models with CoT. On substantially harder tasks — MATH (competition-level mathematics), GPQA (graduate-level science), MMLU (multi-domain knowledge), BBH (hard BIG-bench tasks), HumanEval (code generation) — the relationship between reasoning verbosity and accuracy may be qualitatively different. Complex derivations, error-checking, and exploration of alternative approaches may genuinely require more tokens, and CoD's compression could impose a larger accuracy penalty than the ~4-point gap observed on GSM8k. The paper provides no evidence either way.
Reasoning-specialized models: The introduction (Section 1) explicitly names OpenAI o1 and DeepSeek R1 as motivating the latency problem — these models "demand substantially more computational resources at inference time, leading to verbose outputs and higher latency." Yet neither model is evaluated. These models are trained specifically to internalize chain-of-thought reasoning, and it is unknown whether CoD prompting would help (reducing their output verbosity), hurt (conflicting with their training), or have no effect. Given that these are the models most associated with the latency problem the paper aims to solve, their absence is a significant gap.
Open-weight models at scale: The small-model experiments (Table 6) test models up to 3B parameters. There is no evaluation of intermediate-scale open models (7B, 13B, 70B) where many practitioners operate. The paper provides no evidence about where on the scale spectrum CoD transitions from "substantially worse than CoT" (3B and below) to "competitive with CoT" (GPT-4o scale).
Tasks requiring external knowledge or tool use: All evaluated tasks are self-contained reasoning problems where all necessary information is in the prompt. CoD's draft format may be unsuitable for tasks requiring the model to articulate retrieval queries, tool calls, or explanations that reference external facts — the concision constraint could interfere with the specificity needed for API calls or knowledge-grounded reasoning.

What evidence exists in the paper. The limitation in task scope is partially acknowledged insofar as the paper evaluates "3 categories of tasks... following the original CoT paper" (Section 4), but it is not presented as a limitation. The small-model results (Table 6) and zero-shot results (Table 5) provide direct evidence that CoD's effectiveness is not universal — it degrades substantially outside the specific regime of large models with few-shot exemplars. These failure modes suggest that the claims should be circumscribed more carefully than the paper's abstract and introduction imply.

Mitigation status. Not addressed. The paper does not suggest that results might not generalize to harder tasks, reasoning-trained models, or intermediate-scale open models. Future work sections (Section 5) discuss "combining CoD with other latency-reducing methods" and "training with compact reasoning data" but do not prioritize expanding the evaluation scope to test generalization boundaries.

6.5 CoD Transfers the Reasoning Burden to the Prompt Designer Without Documenting the Cost

The assumption or constraint. CoD relies on manually written few-shot exemplars that demonstrate the draft reasoning style for each target task. The paper states: "For each few-shot example, we also include the Chain of Draft written manually by the authors." These exemplars are task-specific: the exemplars for GSM8k (arithmetic word problems) differ from those for sports understanding (logical deduction about game rules) and coin flip (binary state tracking). The paper provides no guidance on how to construct CoD exemplars for a new task, what makes an exemplar effective, or how many exemplars are needed.

The consequence. Deploying CoD on a novel task or domain requires a human to (a) understand the task's reasoning structure well enough to write correct, concise draft solutions, (b) determine what constitutes an appropriate "step" and how to compress it to roughly five words, and (c) select representative exemplars that cover the reasoning patterns the model will encounter. This is a non-trivial prompt engineering burden that the paper's evaluation model — where the authors wrote exemplars for tasks they deeply understand — does not capture. For a practitioner deploying CoD on a custom domain (e.g., legal reasoning, medical diagnosis, financial analysis), the cost of developing and validating CoD exemplars could be substantial, and the paper provides no estimate of this cost or methodology for minimizing it.

This limitation interacts with the scope limitation (Section 6.4): because CoD has only been tested on four simple, well-studied tasks for which the authors could write high-quality exemplars based on their own understanding, we have no evidence about how well the approach transfers to tasks where the prompt designer has less intuition about the optimal draft format. On tasks with ambiguous reasoning structure (e.g., ethical dilemmas, creative planning, open-ended analysis), defining what counts as a correct "draft step" may be inherently difficult.

What evidence exists in the paper. The zero-shot results (Table 5) provide indirect evidence of the dependence on exemplar quality: when no exemplars are provided, CoD's accuracy collapses. This demonstrates that the exemplars are load-bearing — the model is not generalizing from the system prompt alone. However, the paper provides no ablation varying exemplar count, exemplar quality, or exemplar format to characterize how sensitive CoD is to these design choices. The exemplars themselves are not printed in the main text, making it impossible for a reader to assess their construction or replicate them for a new task.

Mitigation status. The paper acknowledges that CoD-style reasoning patterns are "scarce or absent" in pretraining data (Section 4.5) and hypothesizes that fine-tuning on CoD-formatted data could help, but does not explore automated exemplar generation, methods for adapting exemplars across tasks, or guidelines for practitioners. The exemplar construction cost is an unmodeled deployment expense.

6.6 The Metrics Lack Statistical Rigor, Undermining Confidence in the Specific Numerical Claims

The assumption or constraint. The paper reports accuracy, token count, and latency as point estimates without confidence intervals, standard deviations, statistical significance tests, or error bars on any figure or table. The test set sizes are not specified for the BIG-bench tasks, and the coin flip test set is a single synthetic draw of 250 examples with no replication across random seeds. The comparison between CoD and CoT accuracy on GSM8k — a 4-percentage-point gap — is presented as a reliable finding without any quantification of uncertainty.

The consequence. A practitioner cannot assess whether the observed accuracy differences between CoD and CoT are statistically reliable or could be due to sampling variation. On GSM8k (presumably the standard 1,319-example test split), a 4-percentage-point difference corresponds to roughly 53 examples. Without standard deviations, we do not know whether this difference exceeds the expected variance from resampling the test set. On the coin flip task (250 examples), the 100% accuracy reported for both CoT and CoD means that zero errors were observed in 250 trials — the 95% binomial confidence interval extends from roughly 98.5% to 100%, meaning the true accuracy could be as low as 98.5%. This does not undermine the qualitative conclusion (both strategies are very effective on this task), but it means the specific "100%" numbers should not be taken literally as evidence of perfect reasoning. On the BIG-bench tasks (unspecified test set sizes), we simply cannot assess reliability at all.

The latency measurements (reported to one decimal place, e.g., 4.2s, 1.0s) are presented without any information about measurement methodology — were these measured on a single API call, averaged over multiple calls, measured concurrently or sequentially, on what date/time (API latency varies with load)? Latency is the most deployment-environment-dependent metric reported, yet the paper provides the least methodological detail about it.

The absence of statistical rigor is particularly consequential for the paper's central comparative claims: "CoD matches or surpasses CoT in accuracy" on sports understanding and date understanding (with Claude). These are claims about small accuracy advantages (2-4 percentage points) on tasks with unspecified test set sizes. Without statistical testing, we cannot distinguish "CoD genuinely outperforms CoT on this task" from "CoD and CoT have equivalent accuracy, and the observed difference is noise."

What evidence exists in the paper. The absence of statistical reporting is systematic across all tables (Tables 1-6). No table includes ± values, confidence intervals, or significance indicators. The paper does not discuss this as a limitation.

Mitigation status. Not addressed. The addition of confidence intervals on accuracy (via binomial proportion confidence intervals or bootstrap resampling), standard deviations on token counts and latency, and basic significance tests (e.g., McNemar's test for paired accuracy comparisons on the same test set) would substantially strengthen the reliability of the numerical claims without requiring additional experiments.

7. Implications and Future Directions

How This Work Changes the Landscape

Chain of Draft does not introduce a new model architecture, training paradigm, or decoding algorithm. It is, in the strictest sense, a prompt-engineering contribution — a specific way of writing system instructions and few-shot exemplars that reshapes how an LLM structures its intermediate reasoning. This might suggest that its impact on the field should be correspondingly modest. That conclusion would miss the point. CoD's significance lies not in the technical complexity of its mechanism but in the conceptual reframing it forces on how the field thinks about reasoning in language models.

The dominant assumption since Wei et al. (2022) has been that structured intermediate reasoning — the kind that produces accuracy gains on multi-step tasks — comes packaged in verbose, natural-language form. This was not stated as a claim to be defended; it was an unexamined default inherited from the training data. Textbook solutions are verbose. StackExchange answers are verbose. Tutorial transcripts are verbose. Models trained on this data learn to associate "step-by-step reasoning" with "sentence-by-sentence explanation," and the field's reasoning methods — CoT, self-consistency, Tree-of-Thoughts, Graph-of-Thoughts — all inherited this association without questioning it.

CoD breaks this association empirically. The paper demonstrates that you can preserve the structural scaffolding of sequential intermediate reasoning — decomposing a problem into sub-steps, externalizing intermediate results so the model conditions on its own prior conclusions — while discarding the expository scaffolding that surrounds it. On sports understanding with Claude 3.5 Sonnet (Table 3), CoD achieves 97.3% accuracy — 4.1 points higher than CoT's 93.2% — while using 14.3 tokens versus CoT's 189.4. On coin flip (Table 4), both reach 100% accuracy, but CoD does so with 16.8 tokens versus CoT's 52.4 on GPT-4o. These are not marginal efficiency wins. They are demonstrations that the verbosity of CoT reasoning is largely decorative — it adds tokens that restate the problem, explain operations that are already obvious, and frame conclusions that the model has already reached, all without contributing to the logical progression toward the answer.

This reframing has two consequences for the research landscape:

First, it opens a new dimension on the reasoning-methods design frontier. Before CoD, researchers developing reasoning strategies — whether prompt-based (CoT, Tree-of-Thoughts) or training-based (o1, R1, QwQ) — optimized almost exclusively for accuracy. Efficiency was an afterthought, a deployment concern to be addressed by engineers after the research paper was published. CoD argues, through its experimental design, that efficiency should be a co-equal design objective. The paper's Figure 1 visualizes this by plotting accuracy against token usage as a two-dimensional evaluation space, showing that CoD occupies a Pareto region that CoT cannot reach — not because CoD is more accurate, but because it achieves competitive accuracy at dramatically lower cost. This is a methodological contribution to how reasoning strategies should be evaluated and compared, and it applies regardless of whether one uses CoD specifically.

Second, it performs a diagnostic on pretraining data that the field had not conducted. When CoD fails — in zero-shot settings (Table 5) and on small models (Table 6) — the failure is informative. It reveals that current LLMs do not possess a robust internal prior for concise, draft-style reasoning. The models can produce such reasoning when given in-context exemplars to imitate, but they cannot generate it from the system prompt alone, and smaller models struggle even with exemplars. The paper's hypothesis — that "CoD-style reasoning patterns" are "scarce or absent in the training data of large language models" (Section 4.5) — is not directly proven but is the most parsimonious explanation for the pattern of results. If correct, this implies that the field's verbosity problem is not primarily a prompting problem but a training data curation problem. The long-term solution is likely to involve including concise reasoning exemplars — equations, symbolic manipulations, terse state updates — in pretraining and instruction-tuning corpora, not just relying on prompt engineering to override the training distribution at inference time.

This diagnostic function is arguably CoD's most lasting contribution. It transforms the paper from a "here's a useful prompt template" into a probe that reveals something about current model capabilities and their origins. Future work on training more efficient reasoners — whether through data curation, fine-tuning, or architectural changes — now has a behavioral assay (zero-shot CoD accuracy vs. zero-shot CoT accuracy) for measuring whether those interventions are closing the concision gap or merely working around it.

The paper also resolves a latent tension in the reasoning literature. Prior work on reasoning efficiency split into two camps: methods that reduce verbosity by compressing or eliminating natural-language reasoning (CCoT, TALE, Coconut) and methods that reduce latency without reducing verbosity (streaming, speculative decoding, Skeleton-of-Thought). The former camp sacrificed either accuracy (Coconut on GSM8k) or flexibility (CCoT's rigid budgets). CoD demonstrates that these tradeoffs are not inherent — a properly designed per-step constraint can achieve substantial verbosity reduction while preserving accuracy and interpretability, and without restricting the number of reasoning steps. This establishes a new point in the design space: efficiency through format density rather than through length limits or language elimination.

The research direction that becomes more attractive as a result of this work is training data curation for reasoning efficiency. If the zero-shot gap is genuinely attributable to training data distribution, then the highest-leverage intervention is not better prompting but better pretraining — including concise reasoning traces alongside verbose ones so that models develop an internal capability for draft-style reasoning that can be activated with minimal prompting.

The research direction that becomes less attractive is global token budgeting as a primary mechanism for efficiency. CoD's results demonstrate that per-step budgets avoid the rigidity and compliance-failure problems that CCoT and TALE encounter, and the paper's analysis (Section 2) provides a clear argument for why constraining density rather than total length is architecturally more compatible with how multi-step reasoning works. Future work on token-efficient reasoning should start from per-step constraints and treat global budgets as a coarse approximation at best.

This is not a paradigm shift — it does not overturn the consensus that structured intermediate reasoning is valuable. It is a reframing with diagnostic force: it changes what properties of reasoning we consider designable, reveals a gap in current training practices, and provides a concrete behavioral benchmark for measuring progress toward closing that gap.

Follow-Up Research This Work Enables

Training models on CoD-formatted data to close the zero-shot concision gap. The most direct and high-impact follow-up is suggested by the paper's own hypothesis: if zero-shot CoD fails because draft-style reasoning is underrepresented in pretraining data (Tables 5 and 6), then including such data during training should close the gap. A strong experiment would take a base model (e.g., Llama-3.1-8B), curate a dataset of 10,000–50,000 reasoning problems with paired CoD-style solutions (generated by a large model like GPT-4o with few-shot CoD prompting, then filtered for correctness), and fine-tune the model on this data. The evaluation would compare zero-shot CoD accuracy before and after fine-tuning on GSM8k, MATH, and BIG-bench reasoning tasks. The prediction from the paper's hypothesis is that post-fine-tuning, zero-shot CoD should approach few-shot CoD accuracy, and the zero-shot CoD-to-CoT gap should substantially narrow. A negative result — if fine-tuning on CoD data does not enable zero-shot CoD generalization — would indicate that the barrier is not training data distribution but a more fundamental limitation of how current architectures represent compressed reasoning steps, which would redirect research toward architectural or training-objective changes.

Self-consistency over CoD drafts to test whether token efficiency translates to compute efficiency. CoD's central claim is that concise drafts carry the same logical content as verbose CoT reasoning. A direct test of this claim would equalize the total token budget between CoD and CoT by running multiple parallel CoD chains and aggregating via majority voting. On GSM8k, CoD uses ~44 tokens per chain versus CoT's ~205 — roughly a 1:4.7 ratio. Running 5 parallel CoD chains (~220 total tokens) and majority-voting would cost approximately the same as one CoT chain (~205 tokens). If this self-consistency CoD configuration matches or exceeds single-chain CoT accuracy, it would demonstrate that CoD's per-token reasoning is genuinely more information-dense, not just shorter. If it underperforms, it would suggest that the 4-point accuracy gap on GSM8k (Table 1) reflects a genuine loss of reasoning fidelity that cannot be compensated by sampling diversity. This experiment would also clarify whether CoD's efficiency gains are primarily about reducing serial latency (fewer tokens per chain) or about reducing total FLOPs (fewer tokens period). The paper does not run this experiment, but it is the most natural stress test of the claim that "effective reasoning in LLMs does not necessarily require lengthy outputs."

Difficulty-stratified error analysis to characterize where concision fails. The paper reports aggregate accuracy on GSM8k (Table 1) but does not analyze whether the 4-point gap between CoD and CoT is uniform across problem difficulty or concentrated on specific problem types. A targeted follow-up would take the 1,319-example GSM8k test set and categorize errors by problem characteristics: number of reasoning steps required, presence of irrelevant numerical information, need for unit conversion, multi-step arithmetic versus single-step calculation, and estimated difficulty (using a baseline model's pass@1 rate as a continuous difficulty measure, following the approach of Snell et al., 2024). The analysis would compare error patterns between CoD and CoT on the same questions: when CoT gets a question right and CoD gets it wrong, what goes wrong in the CoD draft — a missing step, an arithmetic error in a compressed equation, a misinterpretation of the problem that verbose framing would have prevented? This error taxonomy would transform the aggregate accuracy gap from an opaque number into an actionable map of where draft-style reasoning is safe versus risky, and would guide refinement of the prompt template (e.g., "for problems requiring unit conversion, allow an extra step to state the conversion factor explicitly").

Evaluation on reasoning-specialized models (o1, o1-mini, DeepSeek-R1) to probe interaction between training-based reasoning and prompting-based concision. The paper's introduction cites o1 and R1 as motivation — these models "demand substantially more computational resources at inference time" — but never evaluates them. A critical follow-up would test CoD prompting on o1, o1-mini, and DeepSeek-R1 across GSM8k, MATH, and GPQA, comparing against both CoT prompting and default (no explicit reasoning prompt) performance. The key question is whether these models, which are trained to internalize chain-of-thought reasoning, respond to CoD prompting by (a) generating more concise external outputs while preserving accuracy — which would extend CoD's applicability to the model class most associated with the latency problem — or (b) ignoring or being disrupted by the CoD format, since their reasoning process is already optimized during training. A null or negative result would bound the scope of CoD to models that rely on prompting rather than training for structured reasoning. A positive result — especially if CoD reduces o1's notoriously long reasoning traces without accuracy degradation — would be practically significant for deployment cost.

Cross-domain stress testing on tasks where verbosity may be functionally necessary. The paper evaluates on self-contained reasoning problems where all necessary information is in the prompt. Several important reasoning domains may resist CoD-style compression: multi-hop question answering requiring explicit retrieval articulation, code generation where variable names and comments carry semantic weight, and tasks requiring explanation rather than computation (e.g., ethical reasoning, medical diagnosis justification). A systematic evaluation across these domains — using datasets like HotpotQA (multi-hop QA), HumanEval and MBPP (code), and TruthfulQA or a medical reasoning benchmark — would map the boundary of CoD's applicability. The hypothesis, suggested by the paper's results but untested, is that CoD works when the reasoning content is primarily symbolic or logical (arithmetic, state tracking, deductive logic) and may fail when the reasoning content is primarily semantic or knowledge-grounded, where compression risks losing necessary context. Such a stress test would convert CoD from a task-specific prompt template into a understood tool with known operating conditions.

Human evaluation of CoD draft interpretability to validate the claimed advantage over latent reasoning. The paper positions interpretability as a key advantage of CoD over Coconut-style latent reasoning: "it loses the interpretability of natural language reasoning... cannot be applied to black-box models" (Section 2). This claim assumes that CoD's terse drafts — sequences like 20 - x = 12; x = 20 - 12 = 8 — are interpretable to human readers. A human study would test this by presenting participants with CoD reasoning traces (without the final answer) and asking them to (a) determine whether the reasoning is correct, (b) identify where an error occurred in deliberately flawed traces, and (c) rate the interpretability of the draft compared to the corresponding CoT trace. If CoD drafts are rated as highly interpretable and users can reliably detect errors, the interpretability claim is validated. If users find the compressed format confusing or ambiguous — especially for domains like date understanding where the draft might use abbreviations or implicit references — then CoD's advantage over latent methods is weaker than the paper implies, and the prompt template might need to be adjusted to include slightly more context per step.

Practical Applications and Downstream Use Cases

High-volume API-based reasoning pipelines where per-token costs dominate. Consider a financial analysis service that processes thousands of earnings report summaries per hour, extracting key figures and performing arithmetic comparisons. Using GPT-4o at current API pricing, a CoT-based pipeline generating ~205 tokens per query (Table 1, GSM8k) would consume roughly 205,000 output tokens per 1,000 queries. A CoD-based pipeline generating ~44 tokens per query would consume ~44,000 output tokens — a 78.6% reduction. If the task's accuracy requirement tolerates 91% accuracy (CoD on GSM8k) versus 95% (CoT), the cost savings are immediate and substantial. The deployment decision reduces to a business calculation: is the 4-point accuracy gap worth a ~5× reduction in per-query output token cost? For many batch processing applications where occasional errors can be caught by downstream validation, the answer is clearly yes. CoD provides a documented, off-the-shelf prompt template for achieving this tradeoff without investing in model fine-tuning or infrastructure changes.

Real-time interactive tutoring systems where latency directly impacts user experience. In educational applications — math tutoring, logic puzzle practice, coding interview preparation — the system must engage in multi-step reasoning while maintaining conversational responsiveness. A 4.2-second latency per response (GPT-4o CoT on GSM8k; Table 1) creates a stilted interaction where the student waits noticeably for each explanation. A 1.0-second latency (CoD) approaches the threshold for perceived real-time interaction (~0.5-1.0 seconds). The tradeoff here is not just cost but user engagement: students are less likely to persist with a tutoring system that feels slow. CoD's 76.2% latency reduction on GSM8k with GPT-4o directly enables a more fluid user experience while maintaining 91% accuracy, which for a tutoring context where the system can also detect and flag uncertain answers for review may be entirely acceptable. The interpretability of CoD drafts (terse equations) also aligns with how math tutoring content is typically presented — students learn by seeing worked equations, not paragraph-long explanations of subtraction.

Edge deployment or on-device scenarios where output token generation is the bottleneck. The small-model results in Table 6, while showing a CoD-to-CoT accuracy gap, still demonstrate meaningful token reductions: Zoom-SLM-2.3B reduces from 129.0 tokens (CoT) to 55.6 (CoD). For on-device models running on consumer hardware — smartphones, laptops, embedded systems — where token generation speed is limited by memory bandwidth and compute, a 55% reduction in output tokens translates directly to faster responses and lower battery drain. In this setting, CoD is not competing with CoT on accuracy (where CoT dominates) but with Standard prompting and other lightweight strategies. The accuracy improvement of CoD over Standard prompting on GSM8k with small models is substantial (e.g., 50.9% vs. 5.9% for Zoom-SLM-2.3B; Table 6), making CoD a viable middle ground: substantially more accurate than direct answering, substantially faster than CoT. The paper's hypothesis that fine-tuning these models on CoD-formatted data would close the CoT accuracy gap, if validated, would make CoD the default reasoning format for on-device deployment.

When to Prefer This Method

The paper's experimental results and limitation analysis (Section 4.5) jointly define the conditions under which CoD is preferable. The decision rule is not hypothetical — it is directly grounded in the paper's data:

Prefer Chain of Draft when all three conditions hold:

You are deploying few-shot prompting. The paper demonstrates that CoD without few-shot exemplars (zero-shot) suffers a severe accuracy collapse — from 91.4% to 65.5% on GSM8k with Claude 3.5 Sonnet (Tables 1 and 5). If your deployment cannot include task-specific CoD exemplars in the prompt — whether due to input token budget constraints, prompt length limits, or the need for a single general-purpose system prompt — CoD is not a reliable substitute for CoT.
You are using a large instruction-following model. The small-model results (Table 6) show CoD-to-CoT accuracy gaps of 8-27 points on GSM8k for models under 3B parameters. While the paper tests only two large API models (GPT-4o and Claude 3.5 Sonnet), the pattern suggests that effective in-context learning of the draft style requires model capacity. Practitioners using models below ~7B parameters should benchmark CoD against CoT on their specific task rather than assuming the paper's results will transfer.
Your task's reasoning structure is primarily symbolic or logical rather than semantically nuanced. The paper evaluates on arithmetic (GSM8k), symbolic state tracking (coin flip), and constrained commonsense deduction (date understanding, sports understanding). These tasks share a property: the reasoning content can be expressed as equations, state updates, or short logical deductions without loss of fidelity. The paper provides no evidence about tasks requiring explanation (rather than computation), external knowledge articulation, or creative reasoning. In those domains, CoD's compression may discard necessary content, and CoT's verbosity may be functional rather than decorative.

Under these conditions, the specific tradeoff is: a ~4-point accuracy reduction on arithmetic tasks (GSM8k; Table 1) and accuracy parity or slight improvement on symbolic and constrained commonsense tasks (Tables 2-4), in exchange for 68–92% fewer output tokens and 48–76% lower latency. The tradeoff tilts more favorably toward CoD on tasks where CoT is especially verbose (Claude on sports understanding: 189.4 CoT tokens vs. 14.3 CoD; Table 3) and in deployments where latency or per-token cost dominates accuracy considerations.