Large Language Models are Zero-Shot Reasoners

ArXiv: 2205.11916

🎯 Pitch

Simply adding "Let's think step by step" to a prompt unlocks complex, multi-step reasoning in large language models—no examples needed. On arithmetic benchmarks, this zero-shot trick boosted accuracy from 17.7% to 78.7%, revealing a hidden cognitive capability that only emerges at massive scale.

1. Executive Summary

This paper proposes Zero-shot-CoT, a simple prompting method that elicits step-by-step reasoning from large language models without requiring any hand-crafted few-shot examples, by prepending the single phrase "Let's think step by step" before each answer. Evaluating on 12 diverse reasoning benchmarks—including arithmetic (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), commonsense QA, and logical reasoning tasks from BIG-bench—the method produces enormous accuracy gains over standard zero-shot prompting with InstructGPT (text-davinci-002), increasing MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%, while requiring a two-stage prompting pipeline (reasoning extraction followed by answer extraction from the generated chain of thought) rather than the single-pass approach of prior zero-shot baselines. The method's effectiveness obeys a scaling law—substantial improvements emerge only with sufficiently large models (~100B+ parameters), and smaller models show negligible benefit—establishing that the ability to leverage chain-of-thought reasoning in a zero-shot manner is an emergent property of scale, with the paper also demonstrating that the same single prompt generalizes across all task types tested, from arithmetic to symbolic to commonsense reasoning, without any per-task template modification.

2. Context and Motivation

The Central Puzzle: Why Don't LLMs Reason Well Out of the Box?

The core problem this paper tackles is a puzzling disconnect in the behavior of large language models. By 2022, it was well-established that scaling up language models—from millions to hundreds of billions of parameters—yielded impressive improvements across a wide range of NLP tasks. Models like GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022) demonstrated remarkable few-shot learning abilities: given just a handful of examples in the prompt, they could perform translation, summarization, question answering, and many other tasks at competitive levels. This was the "pre-train and prompt" paradigm that had largely displaced the older "pre-train and fine-tune" approach for many applications.

Yet these same models, even at the 100B+ parameter scale, consistently failed on tasks requiring multi-step reasoning. The paper cites Rae et al. (2021), which showed that "even language models at the scale of 100B or more parameters had struggled on system-2 tasks requiring slow and multi-step reasoning." These are not obscure edge cases—they include grade-school math word problems (GSM8K), multi-step arithmetic (MultiArith), symbolic manipulation (concatenating last letters of words, tracking coin flip sequences), and logical deduction. What makes this failure particularly striking is that it violates the standard scaling laws: on most NLP benchmarks, performance improves smoothly and predictably with model size, but on reasoning tasks, the scaling curves were notoriously flat—bigger models didn't help much, if at all.

This creates a fundamental tension. The same model that can write a coherent essay, translate between languages, or answer factual questions with high accuracy cannot reliably solve a problem like "If Roger has 5 tennis balls and buys 2 more cans of 3 tennis balls each, how many tennis balls does he have now?" The knowledge is present—the model knows what addition is, what "can" and "tennis ball" mean—but something about the reasoning process itself is missing when the model is prompted in the standard way.

Why This Gap Matters

The inability to perform multi-step reasoning is not merely an academic curiosity. It has direct practical and theoretical implications:

Practical importance. Many real-world applications require reasoning chains. A customer service bot needs to compute refunds involving multiple discount tiers. A code assistant needs to trace through nested function calls to identify bugs. A medical QA system needs to combine multiple pieces of evidence from a patient history to reach a diagnostic conclusion. If LLMs cannot perform such reasoning, their usefulness in these domains is severely limited, regardless of how much factual knowledge they possess.

Theoretical importance. The flat scaling curves on reasoning tasks challenge the dominant narrative of the time—that simply scaling up models and data would eventually solve all NLP problems. The fact that reasoning performance did not improve with scale suggested that there was something qualitatively different about reasoning ability compared to pattern recognition or knowledge retrieval. Understanding what that difference is, and whether it can be overcome without fundamentally changing the model architecture or training procedure, is a deep scientific question about the nature of the capabilities that emerge from large-scale language modeling.

Benchmarking importance. The reasoning benchmarks themselves (GSM8K, MultiArith, AQUA-RAT, etc.) had become standard evaluation tools for LLMs, and the community needed to know: are these benchmarks measuring something that LLMs fundamentally cannot do, or are we simply not accessing a capability that is already there? The answer to this question shapes whether researchers should invest in new architectures, new training objectives, or simply better ways of interacting with existing models.

The Prior Solution: Few-Shot Chain-of-Thought Prompting

The dominant approach to addressing the reasoning gap, introduced by Wei et al. (2022), was Few-shot Chain-of-Thought (Few-shot-CoT) prompting. The idea was elegantly simple: rather than providing few-shot examples where the answer is just the final number or choice (as in Figure 1a), provide examples where the answer includes the full step-by-step reasoning that leads to that answer (Figure 1b). For instance, instead of:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: The answer is 11.

You provide:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.

When the model sees several such reasoning examples in its context, it learns—purely through in-context learning, without any weight updates—to produce its own step-by-step reasoning before giving the final answer. The results were dramatic: with PaLM 540B, Few-shot-CoT improved GSM8K accuracy from 17.9% to 58.1% (Wei et al., 2022), a more than 3× improvement. The scaling behavior also changed qualitatively: reasoning performance now obeyed scaling laws, improving substantially with model size, whereas standard few-shot performance had remained flat. This was widely interpreted as evidence that the ability to reason was latent in large language models, but required a specific type of prompting to be elicited.

Where Few-Shot-CoT Falls Short

Despite its success, Few-shot-CoT has several significant limitations that this paper identifies and addresses:

1. It requires per-task manual engineering of few-shot examples. For each new reasoning task—whether it's arithmetic, coin flipping, date understanding, or commonsense QA—a human must carefully craft a set of step-by-step reasoning examples that demonstrate the correct reasoning pattern for that specific task. These examples are not interchangeable: the paper shows (Table 5) that using CommonsenseQA examples for a MultiArith task yields only a 27.0% accuracy, compared to 78.7% with Zero-shot-CoT and 88.2% with properly matched Few-shot-CoT examples. The effectiveness of the approach depends heavily on the quality and task-relevance of the provided examples.

2. It relies on the few-shot learning assumption. The original CoT paper framed its contribution as an instance of few-shot learning—the model learns to reason from the provided examples. This framing implies that the reasoning capability is not intrinsically present in the model but must be "taught" through examples at inference time. If true, this would mean that LLMs are not fundamentally capable of reasoning on their own; they can only mimic reasoning patterns they've seen in the prompt. This has important implications for what we believe about the nature of the capabilities that emerge from scale.

3. Zero-shot baselines were not even reported. Perhaps most tellingly, Wei et al. (2022) did not report zero-shot baselines for reasoning tasks. The paper explicitly notes this: "few-shot learning was taken as a given for tackling such difficult tasks, and the zero-shot baseline performances were not even reported in the original work." This omission reflects a field-wide assumption: complex multi-step reasoning was believed to be beyond the reach of zero-shot prompting. The few-shot examples were considered necessary, not merely helpful.

4. Performance is sensitive to example design. The paper demonstrates (Table 5) that when few-shot CoT examples have answer formats that mismatch the target task, performance degrades substantially. Even within the same task, the specific choice of examples matters—the paper reports two different 4-shot-CoT results on MultiArith (89.2% vs. 90.5%) depending on which subset of examples was used (Table 2). This sensitivity means that deploying Few-shot-CoT in practice requires careful example curation per task, which is labor-intensive and brittle.

The Overlooked Possibility: Zero-Shot Reasoning

The paper identifies a striking gap in the literature: no one had seriously investigated whether LLMs can produce chain-of-thought reasoning in a zero-shot manner. The prevailing narrative—reinforced by the flat scaling curves on reasoning tasks and the dramatic improvements from Few-shot-CoT—was that reasoning requires the model to see examples of reasoning first. The model, on this view, needs to be shown how to reason before it can do so itself.

But this narrative has a hole: it conflates the elicitation of reasoning with the capability for reasoning. It's possible that LLMs already possess the capability for multi-step reasoning as an emergent property of their scale and training, but the standard zero-shot prompt format ("Q: [question] A: The answer is") doesn't provide the right signal for the model to deploy that capability. The model might need to be told, in some generic way, to engage in step-by-step thinking—not shown specific examples of what that thinking looks like for a particular task.

The paper points to suggestive but incomplete prior work. Reynolds and McDonell (2021) had demonstrated that a prompt like "Let's solve this problem by splitting it into steps" could facilitate multi-step reasoning in a simple arithmetic problem, but they treated it as a task-specific example and "did not evaluate quantitatively on diverse reasoning tasks against baselines." Shwartz et al. (2020) proposed decomposing commonsense questions into sub-questions—a form of structured reasoning—but this required "substantial manual prompt engineering per each reasoning task." Neither work asked the radical question: can a single, task-agnostic prompt unlock zero-shot reasoning across a wide variety of tasks?

How This Paper Positions Itself

This paper positions itself as filling this exact gap. It does not propose a new model architecture, a new training procedure, or even a new prompting paradigm in the traditional sense. Instead, it makes a conceptual reframing: chain-of-thought reasoning is not something that needs to be taught to LLMs through few-shot examples; it is a latent capability that can be elicited through a simple, generic trigger phrase appended to any question.

The paper explicitly positions its contribution along three axes:

First, against the few-shot assumption. The paper's central claim is that "LLMs are decent zero-shot reasoners" (abstract, Section 1). This directly challenges the framing of Wei et al. (2022), which attributed CoT's success to LLMs' few-shot learning ability. If a single zero-shot prompt can achieve a large fraction of Few-shot-CoT's gains—and the paper shows it can, e.g., 40.7% vs. 48.7% on GSM8K with 8-shot CoT—then the few-shot examples are not the source of the reasoning capability; they are merely one way (and not the only way) to activate it.

Second, against task-specific prompt engineering. Most prior prompting work, whether few-shot or zero-shot, involved crafting prompts that were specific to each task (Liu et al., 2021b). The paper emphasizes that Zero-shot-CoT uses "the same single prompt template" across all 12 diverse reasoning tasks without modification (Section 1). The paper borrows terminology from Chollet (2019) to frame this distinction: prior prompts elicit "narrow generalization" or task-specific skills, while Zero-shot-CoT elicits "broad generalization" or "broad cognitive abilities" such as logical reasoning itself (Section 5). This shifts the research question from "how do we design the best per-task prompt?" to "what multi-task prompts can unlock general cognitive capabilities?"

Third, as the strongest zero-shot baseline. The paper explicitly positions Zero-shot-CoT as "the minimal strongest zero-shot baseline for the challenging reasoning benchmarks" (Section 6). By establishing that a simple, reproducible zero-shot prompt achieves substantial reasoning performance, the paper provides a reference point against which future methods—whether few-shot, fine-tuning, or architecture-based—should be compared. This is methodologically important because the original CoT paper's omission of zero-shot baselines made it impossible to know how much of the improvement came from the reasoning format versus the few-shot examples themselves.

The Broader Implications the Paper Hints At

Beyond the specific method, the paper gestures toward a deeper implication that it describes as "untapped and understudied zero-shot fundamental capabilities of LLMs" (Section 1). If a single phrase—"Let's think step by step"—can unlock reasoning across arithmetic, symbolic manipulation, commonsense inference, and logical deduction, then what other broad cognitive capabilities might be elicited by similarly simple prompts? The paper speculates about "higher-level broad cognitive capabilities like generic logical reasoning" (Section 1) that may be latent in large models but inaccessible through standard prompting formats.

This framing connects to a broader debate in the field about what LLMs "know" versus what they can "express." A model might have learned, through its training on diverse text, the abstract pattern of step-by-step reasoning—the structure of breaking down complex problems, maintaining intermediate states, and building toward a conclusion. But when prompted with "Q: [question] A: The answer is," the model defaults to a direct-answer mode that bypasses this structured reasoning capability. The trigger phrase "Let's think step by step" might serve as a key that unlocks access to this latent reasoning mode, in the same way that instruction-tuned models (Sanh et al., 2022; Ouyang et al., 2022) learn to access their capabilities through natural language instructions. This is related to but distinct from instruction tuning: the paper shows that Zero-shot-CoT works on both instruction-tuned models (InstructGPT) and base models (vanilla GPT-3, PaLM), suggesting the capability is present in the pretraining itself, not implanted by instruction fine-tuning.

The paper thus sets up a research agenda that goes beyond the specific technique: systematically exploring what other broad cognitive capabilities can be extracted from LLMs through minimal, task-agnostic prompting, without resorting to task-specific examples or fine-tuning.

3. Technical Approach

3.1 Reader Orientation

This paper proposes Zero-shot-CoT, a prompting-only method that makes large language models produce step-by-step reasoning without any hand-crafted examples, by inserting the phrase "Let's think step by step" into the prompt and then using a two-stage process—first generating the reasoning, then extracting the final answer from it. The core problem it solves is that standard zero-shot prompting ("Q: [question]\nA: The answer is") causes LLMs to output a direct answer without any intermediate reasoning, which works poorly for multi-step arithmetic and logic tasks, while the existing solution (Few-shot-CoT) requires carefully crafting per-task examples with reasoning demonstrations. The "shape" of the solution is a two-stage prompting pipeline that separates the task of generating a reasoning chain from the task of reading a final answer off that chain, using a single task-agnostic trigger phrase that works across arithmetic, symbolic, commonsense, and logical reasoning benchmarks without modification.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components:

1. The base language model (InstructGPT text-davinci-002, vanilla GPT-3, or PaLM) — an autoregressive transformer that takes a text prompt as input and generates a continuation token-by-token using greedy decoding. It is not fine-tuned or modified; the entire "method" is prompt engineering.

2. The two-stage prompting pipeline — a procedure that calls the language model twice:

   • Stage 1 (reasoning extraction): the original question is reformatted into a prompt ending with the trigger phrase Let's think step by step. The model generates a multi-sentence reasoning chain. This text is saved as intermediate output.
   • Stage 2 (answer extraction): the original prompt, the generated reasoning chain, and an answer-format-specific extraction phrase (e.g., Therefore, the answer (arabic numerals) is) are concatenated into a second prompt. The model generates a short continuation from which the final answer is parsed.

3. The answer parser — a deterministic post-processing function that scans the Stage 2 output for the first token or number matching the expected answer format (number, multiple-choice letter, yes/no, or free-form string) and returns it as the prediction.

Information flows as follows: a question enters the system → the question text is wrapped in a Q: [X]. A: [T] template where [T] is the trigger phrase → the first LLM call produces reasoning text z → z is concatenated with the first prompt and an answer extraction phrase → the second LLM call produces answer text → the parser extracts the final prediction from that text.

3.3 Roadmap for the Deep Dive

• First, the two-stage prompting pipeline itself—exactly what text goes into each LLM call, how the outputs are concatenated, and why two calls are needed rather than one.
• Second, the trigger phrase design and the robustness study that justifies "Let's think step by step" as the specific trigger choice among 16 alternatives tested.
• Third, the answer extraction phrases and answer cleansing procedure, since the raw model output is unreliable and must be parsed into a standardized format per task.
• Fourth, the decoding strategy and model configurations, since the method's deterministic greedy decoding interacts with the two-stage design in important ways.
• Fifth, the experimental design choices—why greedy decoding, why single-run evaluation, and what comparison baselines are used—that frame how the method is evaluated.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a prompting methods paper whose core idea is that multi-step reasoning is a latent capability in large language models that can be elicited by a single generic trigger phrase appended to any question, without requiring task-specific few-shot examples, and that a two-stage pipeline (reasoning generation followed by answer extraction) is the minimal machinery needed to make this work reliably.

The Two-Stage Prompting Pipeline

The central technical mechanism is a two-stage prompting procedure, illustrated in Figure 2 of the paper. Unlike standard zero-shot prompting—which wraps a question in a single template and expects the model to output the answer directly—Zero-shot-CoT splits the process into separate reasoning and answer phases. This design is necessary because the model, when asked to both reason and answer in a single generation, tends to either skip the reasoning or produce the answer in an unparseable format.

Stage 1: Reasoning Extraction. The input question x is transformed into a prompt x' using the template:

Q: [X]. A: [T]

where [X] is the original question text and [T] is a hand-crafted trigger sentence. In the canonical configuration, [T] is the string "Let's think step by step.". So a concrete prompt looks like:

Q: On average Joe throws 25 punches per minute. A fight lasts 5 rounds of 3 minutes. How many punches did he throw?
A: Let's think step by step.

This prompted text x' is fed into the language model, which generates a continuation z—the chain-of-thought reasoning—using greedy decoding (temperature = 0 for OpenAI models, TopK=1 for PaLM). The model sees the question followed by the trigger phrase and is thus conditioned to produce a step-by-step reasoning path rather than a direct answer. The paper uses a custom stop sequence "Q:" for all models except InstructGPT, to prevent the model from hallucinating new question-answer pairs and continuing indefinitely.

Why greedy decoding? The paper uses greedy decoding throughout all experiments, making the zero-shot results deterministic. The authors note in the Checklist: "Our paper mainly used GPT-3 API with greedy decoding, and there are no randomness for the experiments." This is a deliberate choice: it eliminates variance from sampling, making the results exactly reproducible, and it isolates the effect of the prompt design from any stochastic search strategy. However, it also means that the reported numbers are a lower bound on what could be achieved with sampling and majority voting—a point the paper later explores with the self-consistency experiments on PaLM (Table 25 in Appendix D).

Stage 2: Answer Extraction. The reasoning text z generated in Stage 1 contains the step-by-step logic but often ends with the reasoning itself rather than a cleanly formatted final answer. The model might conclude with "So the total is 375 punches" or "Therefore, the answer is 375" or simply "375"—the format is inconsistent. To extract a parseable answer, the paper concatenates three elements into a second prompt:

[X'] [Z] [A]

where:

• [X'] is the entire Stage 1 prompt ("Q: [X]. A: Let's think step by step."),
• [Z] is the full reasoning text generated by the model in Stage 1,
• [A] is an answer-format-specific extraction trigger.

The extraction trigger [A] varies by task depending on the expected answer format. The paper provides the complete list in Appendix A.5 (Table 10). Examples include:

• For arithmetic tasks requiring a numerical answer (SingleEq, AddSub, MultiArith, GSM8K, SVAMP): "Therefore, the answer (arabic numerals) is"
• For multiple-choice tasks (AQUA-RAT, CommonsenseQA, Date Understanding, Tracking Shuffled Objects): "Therefore, among A through E, the answer is" (with the letter range adjusted per task)
• For yes/no tasks (StrategyQA, Coin Flip): "Therefore, the answer (Yes or No) is"
• For free-form tasks (Last Letters): "Therefore, the answer is"

A concrete full Stage 2 prompt would be:

Q: On average Joe throws 25 punches per minute. A fight lasts 5 rounds of 3 minutes. How many punches did he throw?
A: Let's think step by step.
In one minute, Joe throws 25 punches. In three minutes, Joe throws 3 * 25 = 75 punches. In five rounds, Joe throws 5 * 75 = 375 punches.
Therefore, the answer (arabic numerals) is

The model then generates a short continuation (e.g., "375" or "375 punches"), and the answer parser extracts the first number from this text.

Why two stages instead of one? The paper does not provide an explicit ablation comparing one-stage versus two-stage prompting, but the design choice can be understood by examining the failure modes. In a single-stage approach where the prompt is "Q: [X]. A: Let's think step by step. The answer is"—forcing both reasoning and answer extraction in one generation—the model would need to produce the reasoning and then seamlessly transition to the answer format. In practice, the model's generated reasoning often ends with a natural-language conclusion rather than a machine-parseable answer token, or it includes multiple candidate answers, or it continues reasoning past the point of reaching an answer. The two-stage design decouples these concerns: Stage 1 optimizes for generating coherent reasoning without worrying about answer format, and Stage 2 optimizes for answer extraction by conditioning on both the question and the complete reasoning. The Stage 2 prompt is "self-augmented"—it contains text generated by the same model in Stage 1, giving the model the full context of its own reasoning when asked to extract the answer.

The zero-shot baseline uses a single stage. The standard zero-shot approach (Figure 1c) uses a single prompt like "Q: [X]. A: The answer (arabic numerals) is" and expects the model to output the answer directly. The paper reports both the standard zero-shot baseline with this single-stage approach and the Zero-shot-CoT method with two stages. Critically, the zero-shot baseline already uses an answer extraction prompt ("The answer (arabic numerals) is") similar to Stage 2 of Zero-shot-CoT—the difference is that Zero-shot-CoT interposes a reasoning generation step before answer extraction.

The few-shot methods avoid two stages by design. In Few-shot and Few-shot-CoT, the in-context examples are explicitly formatted so that the answer appears in a predictable position. In Few-shot-CoT (Figure 1b), each example ends with "The answer is [number]" after the reasoning chain. The model, having seen several such examples, learns to produce output with the same structure, and the paper's parser simply scans for "The answer is " and extracts whatever follows. This is why few-shot methods need only a single model call: the format is taught through examples, and the answer position is standardized by the demonstration format.

The Trigger Phrase and Robustness Study

The paper's central claim is that a specific short text string, when appended to a question, is sufficient to elicit step-by-step reasoning. But which string? The paper tests 16 different templates on the MultiArith dataset with text-davinci-002, categorized into three groups (Table 4):

Instructive templates (encourage reasoning):

1. "Let's think step by step." — 78.7% accuracy (best)
2. "First," — 77.3% (used in Ahn et al., 2022 for robotic action planning)
3. "Let's think about this logically." — 74.5%
4. "Let's solve this problem by splitting it into steps." — 72.2% (used in Reynolds and McDonell, 2021)
5. "Let's be realistic and think step by step." — 70.8%
6. "Let's think like a detective step by step." — 70.3%
7. "Let's think" — 57.5%
8. "Before we dive into the answer," — 55.7%
9. "The answer is after the proof." — 45.7%

Misleading templates (discourage reasoning or encourage wrong reasoning): 10. "Don't think. Just feel." — 18.8% 11. "Let's think step by step but reach an incorrect answer." — 18.7% 12. "Let's count the number of 'a' in the question." — 16.7% 13. "By using the fact that the earth is round," — 9.3%

Irrelevant templates (nothing to do with reasoning): 14. "By the way, I found a good restaurant nearby." — 17.5% 15. "Abrakadabra!" — 15.5% 16. "It's a beautiful day." — 13.1%

For reference, the standard zero-shot baseline achieves 17.7% on this dataset.

Key findings from the template study:

The gap is large between best and worst instructive templates. Even within the "instructive" category, accuracy ranges from 45.7% to 78.7%. This means the exact wording matters substantially—not just the semantic intent of "please reason." The paper notes that "Let's think step by step." achieves the best results but does not provide a principled explanation for why this specific phrasing outperforms semantically similar alternatives like "Let's solve this problem by splitting it into steps." (72.2%) or "Let's think about this logically." (74.5%).

Misleading and irrelevant templates produce accuracy near or below the zero-shot baseline. Templates like "Don't think. Just feel." (18.8%) and "By the way, I found a good restaurant nearby." (17.5%) yield accuracy indistinguishable from the zero-shot baseline (17.7%). Template 11—"Let's think step by step but reach an incorrect answer."—is particularly revealing: it uses the same key phrase "Let's think step by step" but appends "but reach an incorrect answer.", and accuracy drops from 78.7% to 18.7%. This demonstrates that the model is not simply pattern-matching the substring "Let's think step by step" to trigger a reasoning mode—the semantic content of the full instruction matters.

The trigger phrase affects not just whether reasoning occurs but how it is expressed. Appendix B (Table 13) shows example outputs for each template on the same question. Template 9 ("The answer is after the proof.") produces a structured mathematical format ("11 + 5 = 16\n16 ÷ 8 = 2\n2 days") that is genuinely different in style from template 1's prose reasoning, even though both produce correct answers. Template 7 ("Let's think") produces a garbled one-sentence non-answer: "Let's think about this in terms of days. Megan would receive candy for 16 days. Therefore, the answer (arabic numerals) is 16." (incorrect). This suggests that longer, more specific trigger phrases provide stronger conditioning for the model to engage in genuine multi-step decomposition.

Design choice: why "Let's think step by step" over alternatives? The paper selects this phrase pragmatically—it achieves the highest accuracy on MultiArith—but the theoretical significance is that a single phrase works across all 12 diverse tasks. The paper does not claim this phrase is optimal; it explicitly states, "It remains an open question how to automatically create better templates for Zero-shot-CoT" (Section 4.1). The robustness study's purpose is to establish that the phenomenon (zero-shot chain-of-thought elicitation) is real and robust, not that the specific trigger is uniquely special.

Answer Extraction Phrases and Answer Cleansing

Why task-specific answer extraction is needed. After Stage 2, the model outputs a text string that typically contains the answer embedded in natural language—e.g., "375 punches. Therefore, the answer (arabic numerals) is 375." The answer parser must extract just the relevant part and convert it to a standardized format for evaluation. The extraction phrase used in Stage 2 ("Therefore, the answer (arabic numerals) is") helps push the model toward a parseable format, but the raw output is still not guaranteed to be machine-readable. The answer cleansing step handles the remaining parsing.

Answer extraction phrase design. The paper provides a complete mapping of answer extraction phrases to tasks in Appendix A.5, Tables 9 and 10. There are two variants for each method:

• Zero-shot (left): the answer extraction phrase used as the default. These are format-specific: "The answer (arabic numerals) is" for numerical tasks, "Among A through E, the answer is" for multiple-choice, "The answer (Yes or No) is" for binary tasks.

• Zero-shot (right): a simpler, format-agnostic variant using just "The answer is" for all tasks. This variant is tested in Table 1 as a robustness check—it shows that the format-specific variants provide moderate improvements (e.g., 78.7% vs. 79.3% on MultiArith with the generic prompt), but the method works even without task-specific answer formatting.

• Zero-shot-CoT (left): the Stage 2 extraction phrase, prepended with "Therefore, " to create a natural transition from the reasoning text to the answer. The format specificity is maintained: "Therefore, the answer (arabic numerals) is" for numerical tasks, etc.

• Zero-shot-CoT (right): the generic variant "Therefore, the answer is" used as a robustness check.

The design principle is that the extraction phrase should match the expected answer format to minimize parsing ambiguity, but the method is not brittle to this choice—the generic variant still works.

Answer cleansing procedure (Appendix A.6, Table 11). The paper defines four cleansing approaches, one per answer format:

1. Number format: Scan the output text for the first occurrence of a number (possibly with commas, decimals, or a leading minus sign). The regex used is -?\d+\.?\d* after removing commas. The first match is returned as the prediction. Example: if the output is "probably 375 and 376", the parser extracts "375".

2. Multiple-choice format: Scan for the first occurrence of a capital letter among the valid choices (A|B|C|D|E|F depending on the task). The first match is returned. If the model outputs "Therefore, among A through E, the answer is A, B, C, or D"—a known failure mode where the model cannot narrow to one choice—the parser would return "A" (the first large letter encountered).

3. Yes/No format: Convert the output to lowercase, remove punctuation and whitespace, split into tokens, and scan for the first occurrence of "yes" or "no". This handles outputs like "Therefore, the answer is: Yes." or "So the coin is still heads up. The answer is Yes.".

4. Free format (Last Letters only): Remove punctuation and whitespace characters (", ', \n, ., space) and return the cleaned string. The model is expected to output the concatenated letters directly.

For few-shot methods, answer extraction follows a different procedure. The paper first searches for the substring "The answer is " in the model output, and applies the same cleansing logic to whatever follows it. If "The answer is " is not found, the parser searches from the end of the text backward and applies the cleansing logic to the first text that matches the answer format. This fallback is necessary because few-shot models occasionally deviate from the expected format.

Critical interaction with the two-stage design. The answer cleansing procedure explains why Stage 2 is needed even though the reasoning in Stage 1 often already contains the answer. The reasoning text z might say "So the total is 375 punches" or "That means she has 28 pieces of candy in total." But this text is embedded in a longer narrative and may include multiple numbers, intermediate calculations, and natural language. Simply parsing the Stage 1 output directly would require a much more sophisticated parser that understands which number is the final answer. The Stage 2 prompt, by providing an explicit answer extraction format ("Therefore, the answer (arabic numerals) is"), positions the model to isolate the answer in a parseable position, making the cleansing step robust with simple regex-based extraction.

Decoding Strategy and Model Configurations

Greedy decoding as the default. The paper uses greedy decoding (temperature = 0 for OpenAI API calls, TopK=1 for PaLM) across all experiments. This makes the zero-shot results fully deterministic—given the same prompt, the model always produces the same output. The authors justify this in the Checklist: "there are no randomness for the experiments." For few-shot experiments, a fixed random seed is used for example ordering, and each experiment is run only once, following Wei et al. (2022)'s finding that "the order of examples did not cause large variance in CoT experiments."

Implications of greedy decoding. Greedy decoding is a conservative choice. It means the reported numbers represent the model's single most likely generation, not the best among multiple samples. When the paper later experiments with self-consistency on PaLM (Table 25 in Appendix D)—generating 40 reasoning paths with non-zero temperature and taking a majority vote—accuracy on GSM8K jumps from 43.0% (greedy Zero-shot-CoT) to 70.1% (Zero-shot-CoT + self-consistency). This suggests the greedy decoding results are a lower bound. The paper does not explore self-consistency with InstructGPT models, which may have shown even larger gains.

Maximum token limits. For OpenAI models, max_tokens = 128 is used. For PaLM, max_tokens = 256. These limits constrain the length of generated reasoning in Stage 1. In practice, some reasoning chains hit the token limit (the paper notes this in the error analysis: Table 24 includes an example categorized as "OTHERS; MAXIMUM LIMIT OF TEXT LENGTH" where the model's reasoning is cut off mid-sentence). This is a practical limitation: complex multi-step problems may require longer reasoning chains than the token budget allows.

Stop sequences. The paper sets "Q:" as a custom stop sequence for all models except InstructGPT. Without this, the model sometimes continues generating beyond the answer, hallucinating new question-answer pairs in a loop. This is a known failure mode of autoregressive models when prompted with a structured Q&A format—they can enter a repetitive generation pattern. The stop sequence prevents this by terminating generation as soon as the model tries to start a new question.

Model versions and API details. The paper experiments with multiple model families:

• Original GPT-3: ada (0.3B), babbage (1.3B), curie (6.7B), davinci (175B) — accessed via OpenAI API between April–May 2022.
• InstructGPT: text-ada-001, text-babbage-001, text-curie-001, text-davinci-001, text-davinci-002 — the -002 model uses fine-tuning data up to June 2021, while -001 models use data up to October 2019. Unless otherwise stated, text-davinci-002 is the primary model.
• PaLM: 8B, 62B, and 540B parameter variants.
• Additional models for scaling study: GPT-2 (1.5B), GPT-Neo (2.7B), GPT-J (6B), T0 (11B), OPT (13B) — accessed via Hugging Face Transformers library.

Implementation details. The paper uses PyTorch with Hugging Face Transformers for non-API models. Experiments run on cloud V100 instances (no GPU needed for GPT-3 API calls), cloud A100×8 GPU (60GB) instances for T0 and OPT, and cloud A100×1 GPU (60GB) instances for GPT-J, GPT-Neo, and GPT-2.

Experimental Design Choices

Why single-run evaluation rather than multiple runs with error bars? The paper explicitly addresses this in the Checklist: "Our paper mainly used GPT-3 API with greedy decoding, and there are no randomness for the experiments." This is a deliberate methodological choice—because greedy decoding is deterministic, multiple runs would produce identical results. For the few-shot experiments where example ordering could introduce variance (Lu et al., 2022), the paper follows Wei et al. (2022)'s protocol of fixed-seed single runs and cites their finding that ordering variance is small for CoT experiments.

Baseline comparison design. The paper compares Zero-shot-CoT against four categories:

1. Zero-shot (standard): Single-stage prompting with format-specific answer extraction. This is the direct baseline that isolates the effect of adding chain-of-thought reasoning.

2. Few-shot (standard): In-context learning with 2 or 8 question-answer pairs where the answer is just the final result. This tests whether few-shot pattern matching (without reasoning) can match zero-shot reasoning.

3. Few-shot-CoT: The Wei et al. (2022) method with 2, 4, or 8 examples containing step-by-step reasoning. This is the upper bound—the best known prompting method at the time.

4. Zero-Plus-Few-Shot-CoT: A hybrid where "Let's think step by step." is inserted at the beginning of each few-shot CoT example's answer. This tests whether the zero-shot trigger adds value on top of few-shot demonstrations.

The paper uses the same 8 examples from Wei et al. (2022) for all few-shot baselines on MultiArith and GSM8K, ensuring fair comparison. For the 4-shot variants, the 8 examples are split into two non-overlapping groups of 4, and results are reported for both groups to measure sensitivity to example choice.

Task coverage and dataset selection. The 12 datasets span four reasoning categories:

• Arithmetic: SingleEq, AddSub, MultiArith, GSM8K, AQUA-RAT, SVAMP (6 datasets)
• Commonsense: CommonsenseQA, StrategyQA (2 datasets)
• Symbolic: Last Letter Concatenation, Coin Flip (2 datasets)
• Other logical: Date Understanding, Tracking Shuffled Objects (2 datasets, from BIG-bench)

The arithmetic tasks are further stratified by difficulty: SingleEq and AddSub are single-step problems that don't require multi-step reasoning (and thus serve as a sanity check—Zero-shot-CoT should not help much here, and indeed it doesn't), while MultiArith, GSM8K, AQUA-RAT, and SVAMP require multi-step reasoning.

Answer cleansing for evaluation consistency. The cleansing procedure is applied identically to all methods (zero-shot, few-shot, and their CoT variants) to ensure fair comparison. The only difference is that few-shot methods first search for "The answer is " as a delimiter before applying cleansing, while zero-shot methods apply cleansing directly to the Stage 2 output (or the single-stage output for standard zero-shot).

Why the two-stage design is not ablated directly. The paper does not include an experiment where Stage 2 is omitted and the answer is extracted directly from Stage 1's output. This means we cannot quantify exactly how much the second stage contributes over a hypothetical one-stage CoT prompt. However, the paper implicitly provides evidence: the Standard Zero-shot baseline (Table 1) can be viewed as a one-stage approach with an answer extraction prompt but no reasoning trigger. The gap between Zero-shot and Zero-shot-CoT (e.g., 17.7% → 78.7% on MultiArith) measures the combined effect of the trigger phrase AND the two-stage design, but doesn't separate them. The template robustness study (Table 4) shows that the trigger phrase alone (in a two-stage setup) accounts for most of the gain, but the marginal contribution of the second stage is not isolated.

4. Key Insights and Innovations

Innovation 1: Reasoning is a Latent Zero-Shot Capability, Not a Learned Few-Shot Skill

The paper's most fundamental conceptual move is reframing chain-of-thought reasoning from a capability that must be taught to LLMs via in-context examples to one that is already present and merely needs the right trigger to be elicited. This is not a small refinement of Few-shot-CoT—it's a direct challenge to the core assumption underlying Wei et al. (2022) and the broader few-shot learning paradigm for reasoning tasks.

What the dominant assumption was. Prior to this work, the field treated multi-step reasoning as a capability that LLMs could acquire through few-shot demonstrations. The narrative was: the model sees examples of step-by-step reasoning in the prompt, learns the pattern in-context, and then applies it to the target question. This framing was reinforced by three pieces of evidence. First, reasoning performance obeyed scaling laws only when CoT prompting was used—standard prompting showed flat scaling curves, suggesting the capability emerged from the combination of scale and exemplars. Second, Wei et al. (2022) did not report zero-shot baselines, implicitly treating them as trivially poor. Third, the dramatic gap between standard few-shot (17.9% on GSM8K with PaLM 540B) and Few-shot-CoT (58.1%) made the few-shot examples appear essential.

How this paper reframes the evidence. Zero-shot-CoT with PaLM 540B achieves 43.0% on GSM8K—a score that captures 70% of the gap between standard few-shot and Few-shot-CoT, using no task-specific examples whatsoever (Table 2, bottom block). With self-consistency (sampling 40 reasoning paths and taking a majority vote), the zero-shot approach reaches 70.1%, which actually exceeds Few-shot-CoT's 56.9% without self-consistency. The few-shot examples are not the source of the reasoning capability—they are one way (among others) to activate it, and they add roughly 10–15 percentage points on top of what zero-shot already achieves.

The implications of this reframing go beyond prompting technique. If reasoning can be elicited zero-shot, then the capability is genuinely an emergent property of large-scale pretraining—not something the model "learns" on the fly from examples. The model has internalized the abstract pattern of step-by-step decomposition from its training data, and the prompt serves only as a gate that determines whether the model accesses this pattern or falls back to a shallower direct-answer mode. This is a stronger claim about what pretraining produces than the Few-shot-CoT framing allows, and it shifts the burden of explanation from "how does in-context learning teach reasoning?" to "why does standard prompting fail to elicit reasoning that is already there?"

Evidence anchoring the claim. Table 2 is the lynchpin: Zero-shot-CoT (40.7% on GSM8K) substantially outperforms standard 8-shot few-shot prompting (15.6%), which uses task-specific examples without reasoning chains. This means the zero-shot reasoning trigger provides more benefit than seeing eight task-relevant answer examples. Figure 3 shows that this holds across model scales and families: Zero-shot-CoT dramatically improves over Zero-shot on MultiArith and GSM8K for Original GPT-3, InstructGPT, and PaLM, with the gap widening at larger scales.

Innovation 2: The Two-Stage Prompt as a General Design Pattern for Eliciting Structured Output

The two-stage prompting pipeline (reasoning extraction followed by answer extraction from the generated reasoning) is more than an implementation detail—it's a design pattern that addresses a fundamental limitation of single-pass prompting for tasks requiring structured intermediate computation. The paper doesn't name this as a general principle, but the experimental evidence makes the case implicitly: the two-stage design is what makes zero-shot reasoning reliable enough to work across diverse tasks without per-task engineering.

What prior work did. Standard zero-shot prompting wraps a question in a single template ("Q: [question] A: The answer is") and expects the model to output the answer directly. This conflates two distinct sub-tasks that LLMs handle poorly when combined: (1) producing coherent multi-step reasoning and (2) formatting the final answer in a machine-parseable way. When asked to do both in one generation, the model often skips the reasoning entirely (the zero-shot failure mode) or produces reasoning with an unparseable answer. Few-shot-CoT solved this problem differently—by embedding the answer format into every few-shot example, the model learns through in-context repetition to terminate its reasoning with "The answer is [number]".

What the two-stage design contributes. By separating reasoning generation and answer extraction into two distinct model calls, the paper decouples the quality of the reasoning from the parseability of the answer. Stage 1 can focus exclusively on producing a coherent reasoning chain—the model doesn't need to worry about answer format, and the reasoning can end naturally with a prose conclusion. Stage 2 then takes that reasoning as context and conditions on an explicit extraction format ("Therefore, the answer (arabic numerals) is"), which provides strong local conditioning for outputting just the answer in the expected format. The model in Stage 2 sees its own reasoning as part of the prompt, so it doesn't need to regenerate the logic—it only needs to read it off and format the conclusion.

Why this matters beyond the specific trigger phrase. The template robustness study (Table 4) shows that 7 of the 9 instructive templates achieve accuracy above 55% on MultiArith, even though their exact wording varies substantially. This suggests that the two-stage architecture, not the specific trigger text, is the essential enabler—the trigger phrase matters for activating the reasoning mode in Stage 1, but the Stage 2 extraction makes the output reliably parseable regardless of which trigger was used. The paper demonstrates this implicitly: the answer cleansing procedure (Appendix A.6) is simple and regex-based precisely because Stage 2 pushes the model toward a consistent answer format. Without Stage 2, the answer would need to be extracted from the middle of a prose reasoning chain where it might appear as "so she has 28 pieces in total" rather than the parseable "28" that Stage 2 produces after the extraction trigger.

Evidence of generality. Tables 9 and 10 (Appendix A.5) list the answer extraction prompts for all 12 tasks. The pattern is identical across every task type: a "Therefore, " prefix plus a format-specific extraction cue. The fact that this single design pattern works for arithmetic (numbers), multiple-choice (letters), yes/no (binary), and free-form (string) outputs—with only the format cue changing—indicates the two-stage architecture is the general solution, not a task-specific hack.

Innovation 3: A Single Multi-Task Prompt as a Probe for Broad Cognitive Capabilities

The paper introduces a new methodological concept: using a fixed, task-agnostic prompt not as a practical tool but as a diagnostic instrument for probing what broad cognitive capabilities are latent in a language model. This is a significant departure from the dominant prompt engineering paradigm of 2022, which treated prompt design as inherently task-specific.

The narrow-vs-broad distinction. The paper borrows from Chollet (2019) to frame this as the difference between "narrow generalization" (task-specific skills) and "broad generalization" (general cognitive abilities). Prior prompting work overwhelmingly targeted narrow generalization: even zero-shot templates were designed per task (Liu et al., 2021b), with researchers hand-crafting a different prompt format for arithmetic, commonsense QA, translation, etc. The implicit assumption was that what works for one task won't work for another—that effective prompts must be tailored to the specific reasoning patterns of each domain.

What this paper demonstrates instead. Zero-shot-CoT uses exactly one trigger phrase—"Let's think step by step."—across 12 datasets spanning arithmetic, symbolic manipulation, commonsense inference, and logical deduction. The prompt is not modified per task; only the answer extraction phrase changes to match the answer format (and even that is shown to be non-essential, as the generic "The answer is" variant still works—see the right-side numbers in Table 1). This is evidence that the prompt is not eliciting a task-specific skill ("do arithmetic") but rather a domain-general capability ("decompose problems into steps and reason through them").

Why this is a methodological innovation, not just a practical convenience. The single-prompt property means Zero-shot-CoT can serve as a universal probe for whether a model has acquired general reasoning capabilities. If a new model achieves high Zero-shot-CoT accuracy across diverse reasoning tasks, that's evidence of broad cognitive capability emerging from pretraining, independent of any task-specific fine-tuning or example engineering. Conversely, if Zero-shot-CoT works on some tasks but not others, that tells us something about the boundaries of the model's reasoning abilities—which types of reasoning are genuinely general and which remain domain-bound. The paper doesn't fully develop this methodological implication, but it's implicit in the experimental design: by testing on 12 diverse benchmarks with zero prompt modification, the paper is conducting exactly this kind of capability audit.

The distinction from instruction tuning. Sanh et al. (2022) and Ouyang et al. (2022) showed that fine-tuning models to follow instructions improves zero-shot performance. But those methods require explicit training to implant the instruction-following capability, and the instructions themselves are often task-specific. Zero-shot-CoT works on base models (vanilla GPT-3, PaLM) that were never fine-tuned to follow instructions, and it works with a single instruction across all tasks. This suggests the reasoning capability is present in pretraining alone and is orthogonal to instruction tuning—the paper shows gains on both instruction-tuned and base models (Figure 3).

Evidence of breadth. Table 1 shows Zero-shot-CoT improves over standard zero-shot on 10 of 12 datasets. The two exceptions—SingleEq and AddSub—are the easiest arithmetic tasks, which don't require multi-step reasoning, making the null result exactly what you'd expect if the trigger specifically elicits multi-step decomposition. The commonsense tasks (CommonsenseQA, StrategyQA) show mixed results: accuracy doesn't improve on InstructGPT, but the paper's error analysis (Appendix C.1, Table 21) reveals that 78% of correct predictions and 62% of incorrect predictions on CommonsenseQA involve the model producing a "reasonable and understandable chain of thought"—the reasoning is being generated, even when the final answer is wrong due to commonsense knowledge gaps rather than reasoning failures. This is further evidence that the trigger elicits a general reasoning procedure that operates independently of domain-specific knowledge.

Innovation 4: The Error Analysis as a Window into the Nature of Emergent Reasoning

The paper's manual error analysis (Section 4.1, Appendix C) is more than a standard results-section appendix—it's a substantive contribution that reveals how zero-shot reasoning succeeds and fails, which in turn illuminates the nature of the capability itself. The paper systematically categorizes reasoning errors on two datasets (CommonsenseQA and MultiArith), comparing Zero-shot-CoT against Few-shot-CoT, and the differences are diagnostically important.

CommonsenseQA errors: reasoning works, knowledge lags behind. Table 21 shows that on CommonsenseQA, 78% of Zero-shot-CoT's correct predictions are accompanied by correct reasoning chains—the model genuinely reasons its way to the right answer. But on incorrect predictions, 62% of errors are "commonsense mistakes"—the reasoning itself is logically valid, but it's built on a flawed factual premise. For example, in Table 22, the model reasons: "A toy car is likely to be found in a child's room. A child's room is likely to be found in a house. Therefore, a toy car is likely to be found in a house" (answer E, correct answer A). The logical chain is perfectly valid; the error is in the commonsense judgment that "house" is more specific/appropriate than "child's room" for a well-used toy car. This is a knowledge error, not a reasoning error—the model reasons correctly but knows imprecisely.

This distinction is important because it suggests reasoning capability and factual knowledge are partially decoupled in LLMs. The zero-shot trigger successfully elicits the reasoning procedure, but the quality of the final answer is bounded by the model's factual knowledge. Improving reasoning (through better prompts, chain-of-thought style) won't fix knowledge gaps—that requires better pretraining or retrieval augmentation.

MultiArith errors: Zero-shot over-reasons, few-shot makes calculation mistakes. Table 23 reveals that Zero-shot-CoT and Few-shot-CoT have qualitatively different error profiles on arithmetic. Zero-shot-CoT's most common failure on incorrect answers is "logical mistake" (68%), but the specific sub-types differ from Few-shot-CoT. Zero-shot-CoT frequently makes "one unnecessary step" errors (10% vs. 2.4% for Few-shot-CoT)—after reaching the correct answer, the model continues reasoning and changes its answer to an incorrect one. Table 24 shows an example: the model correctly computes intermediate steps but then adds an extraneous subtraction step, producing a wrong final answer from a correct reasoning chain. This is a "runaway reasoning" failure mode unique to the zero-shot setting—the model has no examples showing where reasoning should stop. Few-shot-CoT, in contrast, makes proportionally more "calculator errors" (26.2% vs. 8%)—arithmetic mistakes within otherwise correct reasoning chains.

The significance: different failure modes reveal different underlying mechanisms. The contrast between Zero-shot-CoT's over-reasoning and Few-shot-CoT's calculation errors suggests they're not simply "worse" and "better" versions of the same process—they're engaging different aspects of the model's capabilities. Few-shot-CoT's examples provide implicit guidance on when to stop reasoning (the example chains terminate cleanly after the final answer), which Zero-shot-CoT lacks. But Few-shot-CoT's examples don't fix calculation accuracy—the model still makes arithmetic errors. This suggests that chain-of-thought demonstrations primarily teach format and stopping criteria, not computational accuracy. The reasoning procedure is already present zero-shot; what few-shot adds is better control over the procedure's execution (when to start, how to structure, when to stop).

This error analysis is methodologically innovative because it uses prompting not just to improve performance but as a cognitive probe—by comparing error patterns across prompting conditions, we learn something about what the model knows, what it can compute, and what it needs external structure to execute correctly. This is a research methodology that extends beyond the specific method proposed in the paper.

Innovation 5: The Robustness Study Establishes a Framework for Prompt Sensitivity Analysis

The paper's template robustness study (Table 4) is a methodological contribution that goes beyond validating a specific trigger phrase. It establishes a systematic framework for analyzing why and how prompts work by categorizing templates into instructive, misleading, and irrelevant types, and measuring not just accuracy but also the qualitative style of the generated reasoning.

What distinguishes this from standard ablation. A typical robustness study might test 3–4 alternative phrasings and report which works best. This paper tests 16 templates spanning a deliberately wide range—from genuinely helpful ("Let's solve this problem by splitting it into steps.") to actively harmful ("Don't think. Just feel.") to semantically vacuous ("It's a beautiful day."). The inclusion of actively misleading templates is the key methodological innovation: it tests whether the model is simply pattern-matching a surface form ("think step by step") or genuinely responding to semantic content. Template 11 is the crucial test case: "Let's think step by step but reach an incorrect answer." contains the exact same reasoning-trigger substring but appends a contradictory instruction, and accuracy crashes from 78.7% to 18.7%. This demonstrates semantic sensitivity—the model integrates the full instruction, not just a keyword match.

The categorization also enables causal inference about what makes a prompt effective. The "instructive" templates all encourage reasoning in some way, and all improve over the zero-shot baseline, but with substantial variation (45.7% to 78.7%). The "misleading" and "irrelevant" templates don't improve over baseline at all. This pattern—reasoning-improvement requires semantic encouragement of reasoning, and the specific wording modulates the magnitude—rules out several alternative explanations: that any additional text helps (irrelevant templates don't), that just the word "think" is sufficient ("Let's think" at 57.5% is much worse than the full phrase), or that the model is insensitive to instruction semantics (the contradictory template kills performance).

The qualitative dimension. Appendix B (Table 13) shows the content of reasoning produced by different templates on the same question. Template 9 ("The answer is after the proof.") produces a structured, equation-style chain ("11 + 5 = 16\n16 ÷ 8 = 2\n2 days"), while Template 1 produces prose ("Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister. That means she has a total of 16 pieces of candy..."). These are genuinely different reasoning presentations—not just the same reasoning with different wrappers. This means the template doesn't just control whether reasoning occurs; it shapes how the reasoning is expressed, which in turn may affect accuracy. The paper doesn't fully explore this dimension, but the data in Table 13 makes it visible as an open research question.

Significance beyond this paper. The template study provides a blueprint for future prompt sensitivity research: test a wide range of semantically varied templates, include actively misleading controls to verify semantic sensitivity, categorize by intent (helpful/misleading/irrelevant), and examine both quantitative accuracy and qualitative output style. The paper's finding that "Let's think step by step." happens to work best is less important than the framework for discovering and validating such prompts. As the paper notes, "It remains an open question how to automatically create better templates for Zero-shot-CoT"—and the template categorization framework is the starting point for addressing that question systematically.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 12 datasets spanning four reasoning categories. The six arithmetic datasets are SingleEq (508 questions), AddSub (395 questions), MultiArith (600 questions), AQUA-RAT (254 questions, multiple-choice), GSM8K (1,319 questions), and SVAMP (1,000 questions). SingleEq and AddSub are simpler single-step problems from the classic Math World Problem Repository; MultiArith, AQUA-RAT, GSM8K, and SVAMP require multi-step reasoning. The two commonsense datasets are CommonsenseQA (1,221 questions, multiple-choice) and StrategyQA (2,290 questions, yes/no). The two symbolic reasoning datasets—Last Letter Concatenation (500 samples) and Coin Flip (500 samples)—were created by the authors following Wei et al. (2022) using randomized name lists and coin flip sequences, since the original datasets were not publicly available. The two BIG-bench logical reasoning datasets are Date Understanding (369 samples, multiple-choice) and Tracking Shuffled Objects (750 samples, three-object variant, multiple-choice). Dataset licenses, average question lengths, and answer formats are detailed in Appendix A.2, Table 7.

• Base model(s). The primary model is InstructGPT text-davinci-002 (OpenAI API), an instruction-tuned variant of GPT-3.5 whose parameter count is not publicly disclosed. Additional models for scaling studies include original GPT-3 (ada ~0.3B, babbage ~1.3B, curie ~6.7B, davinci ~175B), InstructGPT variants (text-ada-001, text-babbage-001, text-curie-001, text-davinci-001), PaLM (8B, 62B, 540B), and five open-source models tested via Hugging Face: GPT-2 (1.5B), GPT-Neo (2.7B), GPT-J (6B), T0 (11B), and OPT (13B). The scale range from 0.3B to 540B is chosen deliberately to test whether zero-shot chain-of-thought reasoning is an emergent property of scale, following the finding from Wei et al. (2022) that Few-shot-CoT only improves performance substantially with sufficiently large models.

• Metrics. The primary metric is accuracy—the fraction of test questions for which the extracted answer matches the ground truth. Answers are parsed via deterministic regex-based cleansing (Appendix A.6, Table 11): for numerical tasks, the first number encountered is extracted; for multiple-choice, the first capital letter among valid options; for yes/no tasks, the first occurrence of "yes" or "no" after lowercasing; for Last Letters, punctuation and whitespace are stripped and the cleaned string is checked. The grading protocol follows the standard answer formats for each dataset as established by prior work.

• Baselines. The paper compares Zero-shot-CoT against four prompting baselines: (1) Zero-shot (standard): a single-stage prompt with format-specific answer extraction (e.g., "The answer (arabic numerals) is" for arithmetic tasks), representing the standard zero-shot paradigm from Brown et al. (2020); (2) Few-shot (standard): 2 or 8 in-context examples where each answer is just the final result without reasoning, representing the classic few-shot learning approach; (3) Few-shot-CoT: the Wei et al. (2022) method using 2, 4, or 8 examples with step-by-step reasoning chains, representing the best known prompting approach at the time; (4) Zero-Plus-Few-Shot-CoT: a hybrid where "Let's think step by step." is inserted at the beginning of each few-shot CoT example's answer, testing whether the zero-shot trigger adds value on top of demonstrations. For Few-shot and Few-shot-CoT, the paper uses the same 8 examples from Wei et al. (2022) for MultiArith and GSM8K, with the 4-shot variants created by splitting these 8 into two non-overlapping groups of 4. Additional non-prompting baselines from prior work are cited in Table 2 (e.g., finetuned GPT-3 175B at 33% on GSM8K, and finetuned GPT-3 175B + verifier at 55%).

• Generation budget / compute accounting. The paper's primary "compute" unit is a single model call (one prompt → one generation). Under this accounting: Zero-shot-CoT uses 2 calls per question (Stage 1 reasoning + Stage 2 answer extraction); standard Zero-shot uses 1 call; Few-shot and Few-shot-CoT use 1 call each (the in-context examples are part of the prompt, not separate calls). The paper does not report FLOPs or token counts, making the comparison between methods approximate—the two-stage design of Zero-shot-CoT costs twice as many API calls as standard zero-shot, but the prompts in few-shot methods are substantially longer due to in-context examples. All OpenAI experiments use max_tokens = 128 per call; PaLM experiments use max_tokens = 256. Greedy decoding (temperature = 0 for OpenAI, TopK=1 for PaLM) is used throughout, making the zero-shot results deterministic and exactly reproducible.

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing. The paper explicitly states in the Checklist: "Our paper mainly used GPT-3 API with greedy decoding, and there are no randomness for the experiments." For zero-shot methods, the deterministic decoding means multiple runs would produce identical results. For few-shot methods, the paper uses a fixed random seed and runs each experiment once, citing Wei et al. (2022)'s finding that "the order of examples did not cause large variance in CoT experiments." The 4-shot-CoT variants in Table 2 report two results (89.2% and 90.5% on MultiArith) using different example splits to informally assess example sensitivity, but no formal variance analysis is conducted.

Main Quantitative Results

Zero-Shot-CoT vs. Standard Zero-Shot Across All Tasks

The headline result from Table 1: across 10 of 12 diverse reasoning benchmarks, Zero-shot-CoT with text-davinci-002 produces substantial accuracy gains over standard zero-shot prompting, with the largest improvements on multi-step arithmetic and symbolic reasoning tasks. The complete per-dataset comparison (using format-specific answer extraction, left-side values in Table 1):

The critical pattern: Zero-shot-CoT helps massively on tasks requiring genuine multi-step decomposition (MultiArith, GSM8K, Last Letter, Coin Flip, StrategyQA) but provides minimal or no benefit on simpler single-step tasks (SingleEq, AddSub) and actually hurts marginally on CommonsenseQA (−4.2 percentage points). This makes sense: SingleEq and AddSub don't require reasoning chains to solve, so adding step-by-step thinking doesn't help and may introduce new failure modes. CommonsenseQA's negative result is more complex—the paper's error analysis (Appendix C.1, Table 21) reveals that the model does generate reasonable reasoning chains (78% of correct predictions have correct reasoning, and even 62% of incorrect predictions involve commonsense knowledge errors rather than reasoning flaws), but the accuracy metric doesn't capture this because the commonsense knowledge itself is sometimes wrong. The right-side values in Table 1, using the generic answer trigger "The answer is" instead of format-specific extraction, show similar patterns with slightly different numbers (e.g., MultiArith improves from 22.7% to 79.3%, StrategyQA from 54.3% to 52.3%), confirming the method is not brittle to answer extraction format.

On the two symbolic reasoning tasks created by the authors—Last Letter (concatenating last letters of 4 random names) and Coin Flip (tracking heads/tails through 4 flip/don't-flip events)—Zero-shot-CoT produces the most dramatic gains: from near-zero (0.2%) to 57.6% on Last Letter, and from 12.8% to 91.4% on Coin Flip. These tasks are designed to be trivially easy for humans but nearly impossible for LLMs under standard prompting, because they require maintaining and manipulating state through multiple steps—exactly the capability that chain-of-thought reasoning provides. The 91.4% on Coin Flip is particularly notable: it approaches ceiling performance, suggesting that with step-by-step reasoning, the model can perfectly execute the sequential state-tracking that the task requires.

Comparison Against Few-Shot and Few-Shot-CoT Baselines

Table 2 provides the head-to-head comparison on MultiArith and GSM8K with text-davinci-002. The key comparison tiers:

Zero-shot-CoT vs. standard Few-shot. On MultiArith, Zero-shot-CoT (78.7%) dramatically outperforms both 2-shot (33.7%) and 8-shot (33.8%) standard prompting—a 45-point gap. On GSM8K, Zero-shot-CoT (40.7%) similarly crushes 2-shot (15.6%) and 8-shot (15.6%) by 25 points. This demonstrates that providing a generic reasoning trigger is far more effective than providing 8 task-specific examples without reasoning chains. The model benefits more from being told how to think (step by step) than from seeing additional task demonstrations.

Zero-shot-CoT vs. Few-shot-CoT. On MultiArith, Zero-shot-CoT (78.7%) trails 2-shot-CoT (84.8%) by ~6 points, 4-shot-CoT (89.2–90.5%) by ~11 points, and 8-shot-CoT (93.0%) by ~14 points. On GSM8K, Zero-shot-CoT (40.7%) is competitive with 2-shot-CoT (41.3%) but trails 8-shot-CoT (48.7%) by ~8 points. The consistent pattern is that few-shot examples add roughly 10–15 percentage points on top of zero-shot reasoning, but zero-shot alone captures the majority of the gain—on GSM8K, Zero-shot-CoT achieves 40.7% compared to 48.7% for 8-shot-CoT, meaning zero-shot recovers ~84% of the few-shot benefit.

Zero-Plus-Few-Shot-CoT. Inserting "Let's think step by step." into each few-shot example's answer yields 92.8% on MultiArith (vs. 93.0% for 8-shot-CoT without the trigger) and 51.5% on GSM8K (vs. 48.7% for 8-shot-CoT). The trigger phrase provides a small additional boost on GSM8K (+2.8 points) but is essentially neutral on MultiArith, suggesting that once high-quality reasoning examples are provided, the generic trigger adds little marginal value—the examples already teach the model to reason step-by-step.

Zero-shot-CoT vs. fine-tuning baselines on GSM8K. The paper reports that finetuned GPT-3 175B achieves 33% on GSM8K (from Wei et al., 2022), and finetuned GPT-3 175B + verifier achieves 55%. Zero-shot-CoT with text-davinci-002 (40.7%) already surpasses the plain fine-tuning baseline, though it falls short of the verifier-augmented version. With PaLM 540B, Zero-shot-CoT reaches 43.0%, and adding self-consistency (40 sampled reasoning paths with majority voting, from Wang et al., 2022) boosts it to 70.1%—which exceeds both the fine-tuning + verifier result (55%) and Few-shot-CoT with PaLM 540B (56.9%, from Wei et al., 2022). This is a striking result: zero-shot prompting with a generic trigger, combined with simple majority voting over multiple samples, outperforms carefully crafted few-shot examples and even task-specific fine-tuning.

PaLM 540B results (Table 2, bottom block; Table 25 in Appendix D). The PaLM experiments replicate the core findings at the largest scale tested (540B parameters):

• Zero-shot-CoT (43.0% on GSM8K) substantially exceeds standard few-shot with PaLM 540B (17.9%, from Wei et al., 2022) and standard zero-shot (12.5%).
• With self-consistency (40 paths), Zero-shot-CoT reaches 70.1% on GSM8K, compared to Few-shot-CoT + self-consistency at 74.4% (from Wang et al., 2022)—only a 4.3-point gap.
• On MultiArith: Zero-shot-CoT (66.1%) vs. Zero-shot (25.5%), a 40.6-point gain. With self-consistency: 89.0%.
• On AQUA-RAT: Zero-shot-CoT (36.1%) vs. Zero-shot (23.4%), with self-consistency reaching 46.5% (vs. Few-shot-CoT + self-consistency at 48.3%).
• On SVAMP: Zero-shot-CoT (63.1%) interestingly shows no improvement over standard zero-shot (63.1%), but self-consistency boosts it to 80.5% (vs. Few-shot-CoT + self-consistency at 86.6%).

The SVAMP result is a notable negative finding: on this dataset, the reasoning trigger alone doesn't improve greedy-decoded accuracy, even though the same model benefits enormously on GSM8K and MultiArith. The paper doesn't analyze this discrepancy in detail, but it suggests that SVAMP's difficulty profile or question structure may interact differently with zero-shot reasoning elicitation.

Model Scale Study: Zero-Shot Reasoning as an Emergent Property

Figure 3 and the detailed results in Appendix E (Tables 26 and 27) establish that Zero-shot-CoT's effectiveness is strongly scale-dependent—a finding that mirrors Wei et al. (2022)'s observation for Few-shot-CoT. The key results:

Original GPT-3 family (MultiArith, Table 26, Figure 3a): Zero-shot-CoT accuracy by model size: ada 0.3B (1.7%), babbage 1.3B (2.2%), curie 6.7B (2.3%), davinci 175B (19.0%). Standard Zero-shot: 2.0%, 1.3%, 1.5%, 3.3% respectively. The crucial jump occurs at 175B parameters—below that threshold, Zero-shot-CoT is indistinguishable from standard zero-shot (both near random). At 175B, Zero-shot-CoT achieves 19.0%, a ~6× improvement over the 3.3% zero-shot baseline, but still substantially below Few-shot-CoT at the same scale (44.3%).

InstructGPT family (MultiArith, Table 26, Figure 3b): The instruction-tuned models show even sharper scaling behavior: S (~0.3B equivalent) 2.0%, M 3.7%, L 3.3%, XL-1 (text-davinci-001) 47.8%, XL-2 (text-davinci-002) 78.7%. The jump from L to XL-1 is dramatic—a 44.5-point leap—and XL-2 adds another 30.9 points. Standard Zero-shot for these models: 3.7%, 3.8%, 4.3%, 8.0%, 17.7%—a flat curve until XL-2. The instruction tuning in XL-1 appears to unlock substantial zero-shot reasoning capability that was present but inaccessible in the base model at similar scale (davinci 175B: 19.0% Zero-shot-CoT vs. 3.3% Zero-shot).

PaLM family (GSM8K, Table 27, Figure 3c): Zero-shot-CoT accuracy: 8B (2.4%), 62B (10.5%), 540B (43.0%). The improvement from 62B to 540B is a 32.5-point jump, while standard zero-shot only improves from 7.0% to 12.5% over the same scale range—a flat 5.5-point improvement vs. a 32.5-point improvement with CoT. The scaling curve for Zero-shot-CoT is steep and accelerating, while standard zero-shot remains nearly flat—exactly the pattern Wei et al. (2022) observed for Few-shot-CoT, but now demonstrated in a purely zero-shot setting.

Small open-source models (MultiArith, Table 26, bottom): All models below 13B parameters show Zero-shot-CoT accuracy indistinguishable from random or standard zero-shot: GPT-2 1.5B (2.2% vs. 3.2% Zero-shot), GPT-Neo 2.7B (1.3% vs. 3.0%), GPT-J 6B (2.5% vs. 2.7%), T0 11B (3.2% vs. 2.8%), OPT 13B (2.2% vs. 3.7%). This establishes a clear lower bound: models below ~60B parameters show essentially zero benefit from the zero-shot reasoning trigger. The capability to leverage chain-of-thought prompting in a zero-shot manner is genuinely emergent—it does not exist at small scale and appears only with sufficiently large models.

Qualitative analysis of scale effects (Appendix B, Tables 14-15, 20). The paper provides example outputs at different model scales that reveal the qualitative change. At 0.3B (ada), the model produces confused, repetitive output that doesn't form coherent reasoning. At 6.7B (curie), reasoning begins to emerge but contains calculation errors. At 175B (davinci), clean step-by-step reasoning appears. At the largest scales (text-davinci-002, PaLM 540B), the reasoning is articulate, multi-step, and mostly logically sound. PaLM 540B examples (Table 20) show four outcome categories: cases where both Zero-shot and Zero-shot-CoT are correct (7.7% of GSM8K test set), cases where Zero-shot-CoT corrects a Zero-shot error (35.3%), cases where Zero-shot-CoT introduces a new error (4.8%), and cases where both fail (52.1%). The 35.3% correction rate vs. 4.8% error introduction rate is a strong asymmetry favoring the reasoning trigger.

Few-shot-CoT Sensitivity to Example-Task Mismatch

Table 5 reports an experiment that tests how Few-shot-CoT performance degrades when the in-context examples come from a different task than the target question. Using CommonsenseQA examples (multiple-choice format) for two target tasks:

• AQUA-RAT (multiple-choice, matched answer format): Zero-shot baseline 22.4%, Few-shot-CoT with mismatched examples 31.9%, Zero-shot-CoT 33.5%, Few-shot-CoT with matched examples 39.0%. The mismatched-example Few-shot-CoT still provides a 9.5-point gain over zero-shot, but it falls 1.6 points below Zero-shot-CoT (which uses no examples at all). The matched-format property allows partial transfer—the model learns the multiple-choice answer format from the examples even though the domain is wrong—but the lack of domain-relevant reasoning examples limits the benefit relative to Zero-shot-CoT.

• MultiArith (numerical answer, mismatched format): Zero-shot baseline 17.7%, Few-shot-CoT with mismatched examples 27.0%, Zero-shot-CoT 78.7%, Few-shot-CoT with matched examples 88.2%. The mismatched examples provide only a 9.3-point gain—a fraction of Zero-shot-CoT's 61.0-point gain. When both domain and answer format are mismatched, the few-shot examples are nearly useless; Zero-shot-CoT is dramatically superior.

This experiment supports Min et al. (2022)'s finding that LLMs primarily leverage few-shot examples for format inference rather than task learning. When the answer format matches, domain-mismatched examples still help; when it doesn't, they barely help at all. Zero-shot-CoT, by providing the reasoning procedure through the trigger phrase and the answer format through the Stage 2 extraction prompt, sidesteps the need for format-matching examples entirely.

Outcome Analysis with PaLM 540B

Table 20 (Appendix D) presents a fine-grained analysis of how Zero-shot-CoT and standard Zero-shot interact on GSM8K with PaLM 540B. Among the 1,319 test questions:

• Both correct: 7.7% — these are questions easy enough that the model gets them right even without step-by-step prompting.
• Zero-shot-CoT correct, Zero-shot wrong: 35.3% — the largest category, representing questions where the reasoning trigger is essential. These are problems the model can solve but doesn't under standard prompting.
• Zero-shot-CoT wrong, Zero-shot correct: 4.8% — cases where the reasoning trigger backfires, causing the model to produce incorrect reasoning that leads to a wrong answer when it would have guessed correctly without reasoning. The paper notes that "some of these errors are due to PaLM model's instability, where simply adding 'Let's think step by step' causes the model to continuously output newline tokens."
• Both wrong: 52.1% — questions that are beyond the model's capability regardless of prompting strategy.

The 35.3% / 4.8% ratio means Zero-shot-CoT is approximately 7.4× more likely to correct an error than to introduce one, quantifying the net benefit. The existence of a non-trivial 4.8% "backfire" rate is important—it shows the trigger is not harmless on questions where the model would otherwise succeed, a limitation the paper acknowledges but doesn't deeply investigate.

Ablation Studies and Robustness Checks

Template phrasing robustness (Table 4): The paper tests 16 alternative trigger templates on MultiArith with text-davinci-002. The accuracy range within the 9 "instructive" templates spans from 45.7% ("The answer is after the proof.") to 78.7% ("Let's think step by step."), a 33-point spread. The semantically closest alternatives to the best template—"First," (77.3%), "Let's think about this logically." (74.5%), and "Let's solve this problem by splitting it into steps." (72.2%)—all achieve strong but measurably lower performance, showing that the exact phrasing matters. The 8 "misleading" and "irrelevant" templates produce accuracy between 9.3% ("By using the fact that the earth is round,") and 18.8% ("Don't think. Just feel."), clustering around or below the zero-shot baseline of 17.7%. Template 11 ("Let's think step by step but reach an incorrect answer.", 18.7%) is the key diagnostic: it contains the best-performing trigger substring but adds a contradictory instruction, and performance collapses to near-baseline. This confirms the model integrates the full semantic instruction, not just a keyword match on "think step by step." The qualitative outputs in Appendix B, Table 13 show that different templates produce genuinely different reasoning styles—Template 9 produces equation-style formatting, Template 1 produces prose, and Template 7 ("Let's think") produces a garbled one-sentence non-answer—indicating the template shapes not just whether reasoning occurs but how it's structured.

Answer extraction format robustness (Tables 1, 9, 10): The paper tests two answer extraction approaches: format-specific extraction prompts (left values in Table 1, e.g., "Therefore, the answer (arabic numerals) is") and a generic extraction prompt (right values, "Therefore, the answer is"). On arithmetic tasks, the generic variant typically performs within 1–3 percentage points of the format-specific variant: MultiArith (78.7% vs. 79.3%), GSM8K (40.7% vs. 40.5%), AQUA-RAT (33.5% vs. 31.9%). Larger differences emerge on some tasks: Date Understanding (67.5% vs. 61.8%), Last Letter (57.6% — the generic variant isn't reported for the right column, but the left uses "Therefore, the answer is" since Last Letter is free-form), and Coin Flip (91.4% vs. 87.8%). The consistent finding is that format-specific extraction provides a modest boost but is not essential—the method works even without per-task answer formatting, supporting the claim of task-agnostic generality.

Self-consistency with PaLM 540B (Table 25 in Appendix D): The paper tests whether generating multiple reasoning paths (40 samples) and taking a majority vote (Wang et al., 2022) improves Zero-shot-CoT. On GSM8K: 43.0% → 70.1% (+27.1), MultiArith: 66.1% → 89.0% (+22.9), AQUA-RAT: 36.1% → 46.5% (+10.4), SVAMP: 63.1% → 80.5% (+17.4). The self-consistency gains for Zero-shot-CoT are comparable in magnitude to those reported for Few-shot-CoT (GSM8K: 56.9% → 74.4%), demonstrating that the reasoning paths generated by the zero-shot trigger are sufficiently diverse and individually accurate to benefit from majority voting. This is important because it shows the zero-shot trigger doesn't just elicit a single "canned" reasoning pattern—it generates genuinely varied reasoning that samples different approaches to the problem. The SVAMP result is particularly interesting: greedy Zero-shot-CoT showed no improvement over standard zero-shot (both 63.1%), but self-consistency reveals that the reasoning is helping—the model is generating correct reasoning paths that get selected by majority voting even though the greedy single-path selection doesn't pick them.

Error categorization on CommonsenseQA (Table 21 in Appendix C): A manual analysis of 50 correct and 50 incorrect Zero-shot-CoT predictions on CommonsenseQA reveals that 78% of correct predictions have logically correct reasoning chains, while 22% have incorrect reasoning that happens to lead to the right answer (often because the model outputs multiple answer choices and the first one happens to be correct). For incorrect predictions, 62% involve commonsense knowledge mistakes (logically valid reasoning from a flawed factual premise), 28% involve logical mistakes in the reasoning itself, and 10% are "other" errors. The high rate of commonsense mistakes among failures suggests the reasoning capability is present but limited by factual knowledge accuracy—the model reasons correctly about the world but makes wrong inferences because its world knowledge is incomplete or incorrect.

Error categorization on MultiArith (Table 23 in Appendix C): A manual comparison of 50 correct and ~50 incorrect predictions each from Zero-shot-CoT and Few-shot-CoT reveals different error profiles. Zero-shot-CoT's most common failure mode is "One unnecessary step" (10% of errors vs. 2.4% for Few-shot-CoT)—the model reaches the correct answer but then continues reasoning and changes its answer to an incorrect one. Few-shot-CoT's most common failure mode is "Calculator error" (26.2% vs. 8% for Zero-shot-CoT)—arithmetic mistakes within otherwise correct reasoning chains. Zero-shot-CoT also exhibits "Others" errors at 20% (vs. 2.4% for Few-shot-CoT), including failures to start reasoning (just rephrasing the question) and hitting maximum text length limits. These systematic differences suggest the methods fail for different reasons: Zero-shot-CoT lacks the implicit stopping criterion that few-shot examples provide, leading to over-reasoning, while Few-shot-CoT's examples teach format and structure but don't improve computational accuracy.

Few-shot-CoT example order sensitivity (Table 2, MultiArith 4-shot results): The paper reports two 4-shot-CoT results using different non-overlapping subsets of the 8 standard examples: 89.2% and 90.5%. The 1.3-point difference is small relative to the overall method variance and consistent with Wei et al. (2022)'s finding of low example-order sensitivity for CoT prompting, but it confirms that example selection is not entirely irrelevant even for CoT methods.

Cross-model consistency (Tables 26, 27 in Appendix E): The scaling pattern—Zero-shot-CoT substantially outperforms Zero-shot only above ~60B parameters—holds across three model families (Original GPT-3, InstructGPT, PaLM) despite their different training procedures, architectures, and datasets. This family-robust pattern supports the claim that zero-shot reasoning is a general emergent property of scale, not an artifact of a particular training recipe.

Dataset difficulty gradient (Table 1 arithmetic tasks): Within the arithmetic tasks, the benefit of Zero-shot-CoT correlates with task difficulty. The two easiest tasks (SingleEq: +3.4, AddSub: −2.6) show minimal benefit or slight harm. The moderate-difficulty tasks show large benefits (MultiArith: +61.0, GSM8K: +30.3, AQUA-RAT: +11.1). SVAMP (+3.3) is an anomaly—it requires multi-step reasoning but shows little greedy-decoding improvement, though self-consistency reveals latent benefit (+17.4 with PaLM 540B). This gradient validates the claim that Zero-shot-CoT specifically helps with multi-step reasoning rather than generally improving model performance.

Critical Assessment

Does Zero-shot-CoT demonstrate that LLMs are "decent zero-shot reasoners"?

The paper's central claim, stated in the title and abstract, is that "LLMs are decent zero-shot reasoners." The experiments provide strong evidence that adding "Let's think step by step" substantially improves accuracy over standard zero-shot prompting on multi-step reasoning tasks. However, the claim requires careful qualification about what "decent" means and under what conditions it holds.

What the experiments actually show. The method produces large absolute gains on tasks where standard zero-shot is near floor: MultiArith (17.7% → 78.7%), GSM8K (10.4% → 40.7%), Last Letter (0.2% → 57.6%), Coin Flip (12.8% → 91.4%). On tasks where standard zero-shot is already moderate-to-high, gains are smaller or negative: SingleEq (74.6% → 78.0%), AddSub (72.2% → 69.6%), CommonsenseQA (68.8% → 64.6%). The 78.7% on MultiArith is genuinely impressive—the model solves nearly 4 out of 5 grade-school multi-step arithmetic problems with no task-specific examples, purely from a generic trigger. The 40.7% on GSM8K is more mixed: it's a 4× improvement over the 10.4% baseline, but it still means the model fails on nearly 60% of problems.

What "decent" depends on. The quality of zero-shot reasoning is heavily conditioned on model scale. Below ~60B parameters, Zero-shot-CoT provides essentially zero benefit across all tested model families (Figure 3, Tables 26-27). The paper's sample outputs (Tables 14-15) show that small models produce incoherent or vacuous reasoning—they don't just reason poorly, they don't reason at all. "Decent zero-shot reasoners" only applies to models at the very largest scales tested (175B+ for GPT-3, 540B for PaLM, or the unknown but presumably large text-davinci-002). For the vast majority of models in the 0.3B–13B range, the method simply doesn't work.

What's not tested. The paper evaluates on 12 datasets, all of which are English-language reasoning benchmarks with well-defined correct answers. The claim that LLMs are "zero-shot reasoners" in a general sense would require testing on reasoning tasks outside these categories—mathematical proof generation, multi-step planning, causal reasoning, counterfactual reasoning, or open-ended analytical writing. The paper's scope is specifically "benchmark reasoning tasks" with clear answer formats, which is reasonable for the contribution but narrower than natural-language "reasoning" writ large.

The "reasoning" vs. "prompting" confound. A critical reader might ask: does the trigger phrase actually elicit reasoning from the model, or does it simply push the model into a different mode of text generation that happens to produce better answers on these tasks? The error analysis provides some evidence for genuine reasoning—the common failure modes (commonsense knowledge gaps, calculator errors, over-reasoning) are the kinds of errors a human reasoner might make, not the failures of a pattern-matching system. But the paper doesn't include control experiments that would more decisively distinguish these interpretations, such as testing whether the trigger works when placed in a position that shouldn't affect reasoning (e.g., after the answer extraction prompt, or in a different language).

Does the two-stage pipeline, rather than the trigger phrase, account for the improvement?

The paper attributes the success to the trigger phrase "Let's think step by step", but the method also introduces a two-stage prompting pipeline that standard zero-shot doesn't use. The question is how much of the gain comes from the trigger vs. the two-stage design.

What the experiments do and don't separate. The paper never ablates the two-stage design. There is no experiment where "Let's think step by step" is used in a single-stage prompt (e.g., "Q: [question] A: Let's think step by step. The answer is"). There is also no experiment where the two-stage design is used with a neutral Stage 1 trigger (e.g., just repeating the question) to measure the contribution of the second stage alone. The gap between Zero-shot (single-stage, no trigger) and Zero-shot-CoT (two-stage, trigger) measures the combined effect, not the marginal contributions.

Why this matters. The Stage 2 answer extraction prompt ("Therefore, the answer (arabic numerals) is") is substantially more directive than the standard zero-shot prompt ("The answer (arabic numerals) is")—it includes the word "Therefore," which implies that reasoning has occurred and a conclusion follows. It's possible that this more directive answer prompt, combined with the additional context from Stage 1 (even if Stage 1 output were vacuous), accounts for some of the observed improvement. The template robustness study (Table 4) partially addresses this: the two-stage design is held constant while the Stage 1 trigger varies, showing that the trigger content matters substantially. But the marginal contribution of the second stage remains unquantified.

Indirect evidence against the two-stage explanation. If the improvement came primarily from the two-stage design rather than the reasoning content, we would expect similar gains across all instructive templates. But Table 4 shows a 33-point accuracy spread among instructive templates, all using the same two-stage pipeline—the trigger content clearly matters. Additionally, the error analysis (Table 24) shows Stage 1 outputs containing genuine multi-step reasoning that the model demonstrably uses in Stage 2 to derive the answer. The reasoning in Stage 1 is not vacuous filler that the second stage ignores.

Do the scaling experiments establish emergence?

The paper claims that zero-shot chain-of-thought reasoning is an emergent property of scale, based on Figure 3 and the associated tables. The evidence is consistent with emergence but has important caveats.

What the data show. The scaling curves in Figure 3 are qualitatively similar to those in Wei et al. (2022): approximately flat at small scales, then a sharp upward inflection at larger scales. This pattern appears across Original GPT-3 (inflection at 175B), InstructGPT (inflection at XL-1/XL-2), and PaLM (continuous improvement from 8B to 540B, with acceleration at 540B). The flat portion is genuinely flat—small models show near-identical performance with and without the trigger (Tables 26-27).

Caveat 1: the inflection point depends on model family. The emergence threshold isn't at a consistent parameter count. InstructGPT's text-davinci-001 (~175B-equivalent) achieves 47.8% while original GPT-3 davinci (175B) achieves only 19.0%. The 62B PaLM achieves 10.5%—below both but above chance, and at a much smaller parameter count than the GPT-3 inflection. This suggests the emergence threshold depends on training details (instruction tuning, data quality, architecture) as well as raw scale, complicating any simple claim about "emergence at N billion parameters."

Caveat 2: the x-axis conflates model size with model family. Figure 3 plots three separate scaling curves for three different model families, each with different training data, objectives, and architectures. There is no single curve showing a smooth transition from small to large within a homogeneous model series. This makes it difficult to distinguish "emergence with scale" from "emergence in specific large models that happen to be trained differently."

Caveat 3: sample sizes for small models are tiny. The small-model results (GPT-2, GPT-Neo, GPT-J, T0, OPT) represent single models at each scale, not families. A single model performing poorly at 6B parameters doesn't prove that no 6B model could benefit from Zero-shot-CoT—it only shows that these specific models don't. A proper emergence study would test multiple models at each scale with different training recipes, which the paper does not do.

Caveat 4: the PaLM curve is monotonic, not emergent. On GSM8K, PaLM shows improvement even at 8B (2.4% vs. 2.1% zero-shot) and 62B (10.5% vs. 7.0% zero-shot) before the large jump at 540B. This is a monotonic improvement with scale, not a phase transition. The emergence narrative is strongest for the GPT-3 families, where the small models show literally zero benefit, but this pattern doesn't fully generalize to PaLM.

Does Zero-shot-CoT genuinely outperform standard few-shot prompting?

Table 2 shows Zero-shot-CoT (78.7% on MultiArith, 40.7% on GSM8K) dramatically exceeding 8-shot standard prompting (33.8% on MultiArith, 15.6% on GSM8K). This is one of the paper's most striking claims.

What's strong about this result. The comparison is fair: both methods use the same base model (text-davinci-002), the same answer extraction procedure, and the same evaluation. The few-shot baseline uses the standard examples from Wei et al. (2022). The gap is enormous and unlikely to be explained by trivial confounds.

What's missing. The paper doesn't explore how many few-shot examples would be needed to match Zero-shot-CoT. Perhaps 16 or 32 standard few-shot examples would close the gap—the paper only tests 2 and 8. The few-shot examples are also identical to those in Wei et al. (2022), which were designed for Few-shot-CoT, not optimized for standard few-shot. Better-chosen standard few-shot examples might perform substantially better. These are not fatal issues—the paper's claim is comparative, not absolute—but they mean the "zero-shot > few-shot" result is a lower bound on few-shot performance rather than a demonstration that zero-shot is fundamentally superior.

Are the symbolic reasoning results (Last Letter, Coin Flip) reliable?

The paper reports dramatic gains on Last Letter (0.2% → 57.6%) and Coin Flip (12.8% → 91.4%). These are the two tasks created by the authors themselves, since the original datasets from Wei et al. (2022) were not publicly available.

What's potentially concerning. The paper provides the templates used to generate these datasets (Appendix A.2.2) and the sample size (500 each), but doesn't report details about the randomization procedure, the name lists, or whether any quality filtering was applied. The Coin Flip task at 91.4% accuracy is close to ceiling—this is an extraordinarily high number for a task that standard zero-shot achieves only 12.8% on. While the task is genuinely simple for humans (tracking 4 coin flips), achieving near-perfect accuracy on 500 procedurally generated samples raises the question of whether the model might have seen very similar patterns in its training data, or whether the task generation introduced unintended regularities that make it easier than intended.

Mitigating factors. The few-shot baseline on Coin Flip (from Wei et al., 2022) presumably also performs well on this task—the paper doesn't report Few-shot-CoT numbers for Last Letter and Coin Flip, but Wei et al. (2022) shows that Few-shot-CoT dramatically improves symbolic reasoning. The 91.4% is consistent with the general pattern that chain-of-thought makes these particular tasks much easier. Additionally, the sample outputs in Table 12 show the model genuinely tracking state step-by-step ("Jeff does not flip the coin, so the coin remains heads up. Jen flips the coin, so the coin is now tails up..."), which is exactly the reasoning the task is designed to test. The high accuracy, while striking, reflects the task's simplicity once state-tracking is externalized.

Missing experiments that would have strengthened the paper

Single-stage CoT baseline. The most significant missing ablation is a one-stage variant where "Let's think step by step." is used in the same prompt as the answer extraction (e.g., "Q: [X] A: Let's think step by step. Therefore, the answer is"). Without this, we can't measure how much the two-stage design contributes over the trigger alone.

Comparative compute cost analysis. The paper uses "number of model calls" as its compute metric, but doesn't report token counts or dollar costs. Zero-shot-CoT uses 2 API calls per question vs. 1 for standard zero-shot, but zero-shot prompts are short while few-shot prompts include thousands of tokens of in-context examples. A proper compute-efficiency comparison would account for total input + output tokens.

Self-consistency with InstructGPT. The self-consistency experiments (Table 25) are only run on PaLM 540B, not on the primary model text-davinci-002. Given that text-davinci-002 achieves 78.7% greedy on MultiArith (vs. 66.1% for PaLM greedy), self-consistency might push it even higher. The paper misses an opportunity to establish a strong zero-shot + self-consistency ceiling on the OpenAI models.

Error analysis on more datasets. The detailed error categorization is only done for CommonsenseQA (Appendix C.1) and MultiArith (Appendix C.2). Understanding why Zero-shot-CoT fails on the remaining 10 datasets—especially the anomalous SVAMP and the high-performing Coin Flip—would strengthen the paper's claims about what the method does and doesn't do.

Controlled experiments on the trigger mechanism. The paper shows that "Let's think step by step." works better than "Let's think about this logically." or "First,", but doesn't investigate why. Is it the word "step"? The phrase "step by step" specifically? The first-person plural "Let's"? The imperative mood? A controlled experiment systematically varying these dimensions would transform the template study from an empirical observation into a mechanistic understanding.

Testing on non-reasoning tasks as a control. If the trigger specifically elicits multi-step reasoning, it should not improve performance on tasks that don't require reasoning—and might even hurt. The paper has natural controls in SingleEq and AddSub (which indeed show minimal benefit), but deliberately testing on a broader set of non-reasoning tasks (sentiment analysis, topic classification, factual recall) would strengthen the claim of specificity.

Summary of experimental strengths and weaknesses

Strengths:

• The breadth of evaluation (12 datasets, 4 reasoning categories, 17 models) is exceptional for a prompting methods paper and provides strong evidence for generality.
• The template robustness study (Table 4) with 16 variants including actively misleading controls is methodologically rigorous and rules out several alternative explanations.
• The error analysis connecting failure modes to underlying capability limitations (commonsense knowledge vs. reasoning, over-reasoning vs. calculation errors) goes beyond standard accuracy reporting.
• The inclusion of small models and open-source models, despite their negative results, provides valuable negative evidence about the scale-dependence of the method.
• The cross-model-family replication (GPT-3, InstructGPT, PaLM) establishes that the phenomenon is not specific to one training recipe.

Weaknesses:

• The two-stage design is never ablated, making it impossible to attribute the improvement specifically to the trigger phrase vs. the two-stage architecture.
• The small-model negative results, while valuable, come from single models at each scale rather than systematic sweeps, weakening the emergence claims.
• Compute costs are not properly compared across methods (API calls vs. token counts vs. dollar costs).
• The self-created symbolic reasoning datasets lack the vetting and community scrutiny of established benchmarks.
• Several important experiments are run only on PaLM (self-consistency) or only on InstructGPT (template robustness), making cross-model comparisons incomplete.
• The test sets range from 254 (AQUA-RAT) to 2,290 (StrategyQA) questions, with some (Last Letter, Coin Flip) at 500—reasonable but not large, and no confidence intervals are reported.

6. Limitations and Trade-offs

6.1 The Two-Stage Design Costs Twice as Many Model Calls as Standard Prompting, with No Ablation Quantifying Its Contribution

The assumption or constraint. Zero-shot-CoT requires two separate calls to the language model per question: one to generate the reasoning chain (Stage 1) and one to extract the answer from that reasoning (Stage 2). Standard zero-shot prompting requires only a single call. The paper never isolates how much of the observed improvement comes from the trigger phrase versus the two-stage architecture itself, and it does not account for the doubled inference cost in its headline comparisons. The authors do not explicitly acknowledge this as a limitation in the main text, though the method description in Section 3.1 makes the two-stage design transparent.

The consequence. A practitioner deciding whether to adopt Zero-shot-CoT over standard zero-shot faces an unquantified tradeoff: they pay 2× the API calls (or 2× the generation latency in serial execution) for an accuracy gain that may partly come from the trigger phrase and partly from the architectural choice of re-prompting the model with its own reasoning. Because the paper never tests a single-stage variant (e.g., "Q: [X] A: Let's think step by step. Therefore, the answer is"), there is no way to know whether the trigger alone would achieve similar gains at the same cost as standard prompting. The closest evidence is indirect: the standard zero-shot baseline already uses an answer extraction prompt ("The answer (arabic numerals) is"), and the gap between that and Zero-shot-CoT on MultiArith is 17.7% → 78.7%—but this gap conflates the trigger, the two-stage design, and the more directive answer extraction phrase ("Therefore, the answer..." vs "The answer..."). Without an ablation, the method's cost-effectiveness relative to a hypothetical single-stage CoT prompt is unknown.

What evidence exists in the paper. The template robustness study (Table 4) holds the two-stage design constant and varies only the Stage 1 trigger, showing that trigger content matters substantially (45.7% to 78.7% accuracy spread). This tells us the trigger is causal, but not whether the second stage is necessary for the trigger to work. The paper's own description of the few-shot methods highlights that they avoid two stages by embedding answer formats in examples—but no analogous single-stage zero-shot design is tested. The error analysis (Appendix C.2, Table 24) shows cases where Stage 1 reasoning is correct but the model over-reasons and changes its answer, suggesting that Stage 2 sometimes fails to correct errors introduced late in the reasoning chain—in other words, the two-stage design does not fully insulate against reasoning errors from Stage 1.

Mitigation status. Not addressed. The paper never discusses the cost of the second stage or attempts to ablate it. This is a methodological gap that makes the headline accuracy numbers incompletely attributable to the claimed mechanism (the trigger phrase) as opposed to the procedural change (prompting twice).

6.2 The Method Provides Essentially Zero Benefit for Models Below ~100B Parameters, Restricting Its Applicability to the Largest Proprietary Systems

The assumption or constraint. Zero-shot-CoT's effectiveness exhibits a strong scale-dependence: it only produces substantial accuracy gains with models at the very largest scales tested. The paper's own data in Figure 3 and Appendix E (Tables 26, 27) demonstrates this clearly. For Original GPT-3, the inflection occurs at 175B parameters (davinci: 19.0% Zero-shot-CoT vs. 3.3% Zero-shot on MultiArith), while smaller models (0.3B–6.7B) show Zero-shot-CoT accuracy indistinguishable from the zero-shot baseline (1.7%–2.3% vs. 1.3%–2.0%). For open-source models (GPT-2 1.5B, GPT-Neo 2.7B, GPT-J 6B, T0 11B, OPT 13B), Zero-shot-CoT accuracy ranges from 1.3% to 3.2%—all within noise of the 2.2%–3.7% zero-shot baselines. The paper states this finding explicitly in Section 4.1: "When the model size is smaller, chain of thought reasoning is not effective. This result aligns with the few-shot experiment results in Wei et al. [2022]."

The consequence. The method is practically unusable for the vast majority of models that practitioners can run locally or fine-tune themselves. At the time of writing (2022), models at the 100B+ scale were exclusively accessible through paid APIs (OpenAI) or internal research infrastructure (Google's PaLM). A practitioner with access to a 13B or even 65B open-source model would see negligible or zero improvement from adding "Let's think step by step" to their prompts. This severely limits the method's practical reach: it works only for users of the largest commercial APIs. The paper's claim that "LLMs are decent zero-shot reasoners" must be qualified to "only the very largest LLMs, at scales inaccessible to most researchers and practitioners, are decent zero-shot reasoners under this method." The paper's own small-model results (Table 26, bottom) are consistently negative—there is not a single model under 100B parameters where Zero-shot-CoT provides a meaningful accuracy gain.

What evidence exists in the paper. Tables 26 and 27 (Appendix E) and Figure 3 provide comprehensive negative results across 10 models under 100B parameters spanning five model families. The sample outputs in Tables 14 and 15 (Appendix B) show the qualitative degradation: GPT-2 1.5B produces repetitive, incoherent output that does not constitute reasoning; GPT-Neo 2.7B generates a series of repeated "Step N:" prefixes without content; T0 11B outputs a single number with no reasoning at all ("A: Let's think step by step. 12"). These are not cases of poor reasoning—they are cases of no reasoning whatsoever. The scaling curve is not gradual; it shows a sharp threshold below which the trigger simply does not elicit the reasoning behavior.

Mitigation status. The paper is transparent about this limitation in reporting the negative small-model results, which is a strength of its experimental design. However, it does not discuss the practical implication that this restricts the method's user base to a tiny fraction of LLM practitioners, nor does it explore whether alternative triggers or prompting formats might lower the scale threshold. The paper frames the scale-dependence as evidence for emergence (which it is), but does not address the accessibility implications.

6.3 The Difficulty Estimation and Strategy Selection Required for Few-shot-CoT Still Apply to Evaluating Zero-shot-CoT, but the Paper Provides No Guidance on When the Method Will or Will Not Help

The assumption or constraint. The paper positions Zero-shot-CoT as a simple, universal zero-shot baseline, but its results reveal substantial task-dependent variance that it does not systematize. On SingleEq (74.6% → 78.0%), AddSub (72.2% → 69.6%), CommonsenseQA (68.8% → 64.6%), and SVAMP (58.8% → 62.1% with greedy decoding), the method provides marginal gains, no gain, or even harms accuracy. The paper acknowledges this pattern in passing—noting that SingleEq and AddSub "do not require multi-step reasoning" (Section 4)—but provides no principled framework for a practitioner to predict whether Zero-shot-CoT will help or hurt on a new task without running an experiment.

The consequence. A practitioner confronting a new reasoning benchmark cannot simply apply Zero-shot-CoT and expect improvement. They must either run a controlled experiment (requiring a held-out set with ground-truth labels) or guess based on whether the task "seems like it requires multi-step reasoning"—a subjective criterion. The negative result on CommonsenseQA (−4.2 percentage points) is particularly concerning because commonsense QA does involve implicit multi-hop inference (as the paper notes for StrategyQA, where the method helps), yet the trigger hurts performance on this specific dataset. The paper's error analysis reveals that the model does generate reasonable reasoning chains on CommonsenseQA (78% of correct predictions have correct reasoning), but the final answer accuracy drops because the model sometimes reasons from incorrect commonsense premises or cannot narrow multiple-choice options to one. This means the trigger is eliciting reasoning that is actively counterproductive for this task's evaluation metric. A practitioner without access to the paper's detailed error analysis would have no way to anticipate this.

What evidence exists in the paper. Table 1 shows the full task-level variance. The error analysis in Appendix C.1 (Table 21) provides partial explanation for CommonsenseQA: "Zero-shot-CoT often output multiple answer choices when the model find it is difficult to narrow it down to one" and "the model often produces a flexible and reasonable chain of thought even when the final prediction is not correct." Table 24 in Appendix C.2 shows MultiArith failures where the model "does not start reasoning, just rephrasing the input question" or produces "unnecessary steps of reasoning after getting the correct prediction." These are task-specific and model-specific failure modes that the paper catalogs but does not systematize into predictive guidance.

Mitigation status. The paper does not attempt to predict when Zero-shot-CoT will help. It provides post-hoc explanations for the negative results (SingleEq and AddSub are single-step; CommonsenseQA requires factual knowledge beyond reasoning) but no ex-ante decision rule. The absence of a difficulty-prediction framework—even a simple heuristic based on dataset characteristics—means the method must be evaluated per-task, which undercuts its claimed universality.

6.4 The Method Inherits and Amplifies Factual Errors from the Base Model's Knowledge, with No Mechanism for Correction

The assumption or constraint. Zero-shot-CoT elicits step-by-step reasoning but provides no mechanism for the model to verify or correct the factual premises underlying that reasoning. The method assumes the model's knowledge is accurate; when it is not, the reasoning chain becomes a vehicle for confidently articulating incorrect conclusions. The paper describes this explicitly in its error analysis of CommonsenseQA (Appendix C.1): "the model often produces a flexible and reasonable chain of thought (logically correct but lacks common sense) even when the final prediction is not correct."

The consequence. On tasks where reasoning quality and factual knowledge are entangled—which includes commonsense QA, medical reasoning, legal analysis, and most real-world applications—Zero-shot-CoT can produce outputs that are more convincing but equally wrong compared to standard prompting. The reasoning chain gives the appearance of careful deliberation, which may lead users to place greater trust in incorrect answers. Table 22 provides concrete examples: the model reasons that "A toy car is likely to be found in a child's room. A child's room is likely to be found in a house. Therefore, a toy car is likely to be found in a house"—logically valid but factually wrong (the correct answer is "child's room," not "house"). A user reading this chain might be more persuaded by the incorrect answer than by a simple unsupported guess, because the reasoning looks sound. This is a safety-relevant failure mode: the method increases the persuasiveness of incorrect outputs without increasing their accuracy on knowledge-dependent tasks.

What evidence exists in the paper. The CommonsenseQA error categorization (Table 21, Appendix C.1) quantifies this: among incorrect Zero-shot-CoT predictions, 62% involve "commonsense mistakes"—the reasoning is logically valid but built on a flawed factual premise. Only 28% involve genuine logical mistakes in the reasoning itself. This means the method fails primarily because it reasons correctly from wrong facts, not because it reasons poorly. Table 22 provides multiple examples of this pattern. The MultiArith error analysis (Table 23, Appendix C.2) shows a related but distinct problem: 10% of incorrect Zero-shot-CoT predictions involve the model reaching the correct answer through correct reasoning, then adding an unnecessary step that changes the answer to an incorrect one—a failure mode where the reasoning process itself introduces error.

Mitigation status. Not addressed. The paper identifies the phenomenon but proposes no solution. The two-stage design does not help here because the factual errors are embedded in the Stage 1 reasoning, and Stage 2 simply extracts an answer from that flawed reasoning. The paper does not explore whether the model's factual errors could be corrected through additional prompting, retrieval augmentation, or verification mechanisms. This is a fundamental limitation of any pure prompting approach: it can change how the model deploys its knowledge but cannot add knowledge the model lacks or correct knowledge the model has wrong.

6.5 The Method Was Evaluated Only on English-Language Reasoning Benchmarks with Clean Answer Formats, Leaving Generality to Other Languages, Open-Ended Tasks, and Messier Real-World Queries Unestablished

The assumption or constraint. All 12 evaluation datasets are English-language benchmarks with well-defined, machine-parseable answer formats: numerical answers, multiple-choice letters, yes/no responses, or concatenated character strings. The answer cleansing procedure (Appendix A.6, Table 11) depends on this structure—it extracts the first number, the first capital letter, or the first "yes"/"no" token from the model's output. The paper does not evaluate on open-ended generation tasks, non-English languages, or tasks where the "correct answer" is not a single token or short string. The authors do not discuss this limitation explicitly, though the dataset descriptions make the answer format constraints clear.

The consequence. It is unknown whether Zero-shot-CoT provides any benefit for tasks where the reasoning and the answer cannot be cleanly separated by a regex parser. Consider a task like "Explain why the Roman Empire fell" or "Write a Python function to merge two sorted lists and explain your approach"—the model might produce excellent step-by-step reasoning in Stage 1, but there is no Stage 2 extraction that can parse a "correct answer" from that reasoning. The method's two-stage design is fundamentally tied to the assumption that the answer is a short, format-predictable string that can be extracted from the reasoning. For open-ended tasks, the reasoning is the answer, and the two-stage design becomes unnecessary (or counterproductive, since the second stage might truncate the reasoning). The paper's claim of task-agnostic generality applies only within the specific category of "benchmark reasoning tasks with extractable short answers"—a category that excludes many real-world use cases.

What evidence exists in the paper. All 12 datasets have answer formats classified in Table 7 (Appendix A.2): Number (6 datasets), Multiple-choice (4 datasets), Yes/No (1 dataset), and Free Format (1 dataset—Last Letters, which is simply a concatenated string). The answer cleansing approaches in Table 11 (Appendix A.6) are all regex-based and depend on the answer being a single identifiable token or substring. There is no experiment or discussion of tasks with longer, unstructured, or multi-sentence answers. The paper also does not test on non-English benchmarks, despite using models (PaLM, GPT-3) that were trained on multilingual data and have demonstrated multilingual capabilities.

Mitigation status. Not addressed. The paper does not acknowledge this scope limitation or discuss how the method would need to be adapted for open-ended generation. The two-stage design is presented as general, but it contains an implicit assumption about answer format that restricts its applicability. A practitioner dealing with summarization, translation, code generation, or creative writing tasks would find no guidance on whether or how to apply Zero-shot-CoT.

6.6 The Symbolic Reasoning Tasks Were Created by the Authors Without Independent Validation, and the Near-Ceiling Performance on Coin Flip Raises Questions About Task Design Artifacts

The assumption or constraint. Two of the 12 evaluation datasets—Last Letter Concatenation and Coin Flip—were created by the authors themselves because the original datasets from Wei et al. (2022) were not publicly available. The paper provides the generation templates in Appendix A.2.2 and the sample counts (500 each), but does not report validation procedures, quality checks, or analyses of potential artifacts in the generation process. The Coin Flip task uses four coin-flip events per sample, randomly choosing between "flips" and "does not flip" for each event, with uniformly sampled human names from the names-dataset library. Zero-shot-CoT achieves 91.4% accuracy on this task compared to 12.8% for standard zero-shot.

The consequence. The 91.4% accuracy on Coin Flip is the single highest number reported in the paper and accounts for one of the two most dramatic claimed improvements (+78.6 percentage points). If the generated dataset contains unintended regularities—for example, if the name distribution, flip-sequence distribution, or template phrasing creates spurious correlations that make the task easier than intended—then this headline number may overstate the method's effectiveness on genuine symbolic reasoning. The concern is not that the authors deliberately designed an easy task, but that without independent vetting (which established benchmarks like GSM8K and CommonsenseQA have received), procedural generation can introduce subtle artifacts that are invisible to the dataset creators. The fact that the model achieves near-ceiling performance on a task that standard prompting fails almost completely (12.8%) is consistent with the method genuinely enabling state-tracking—the sample outputs in Table 12 show the model correctly tracing coin states step by step—but also consistent with the task being easier than the authors intended due to unintended generation regularities.

What evidence exists in the paper. The templates are documented in Appendix A.2.2, along with the randomization procedures. The paper also reports standard zero-shot and Zero-shot-CoT results on the other symbolic task (Last Letter Concatenation: 0.2% → 57.6%), and the gain is similarly dramatic but not near-ceiling, which partially mitigates the concern—if both tasks were artifactually easy, both might show ceiling effects, but only Coin Flip does. However, no Few-shot-CoT baselines are reported for these tasks, making it impossible to benchmark the Zero-shot-CoT results against the established method's performance on the same generated datasets. The paper also does not report results on the original Wei et al. (2022) symbolic reasoning datasets (which used different configurations) to enable cross-validation.

Mitigation status. The paper is transparent about the data generation procedure but does not discuss potential artifacts, perform sensitivity analyses (e.g., varying the number of coin flips or the name list), or compare against Few-shot-CoT on these tasks. The coin flip examples in Table 12 show genuine step-by-step state tracking, which supports the validity of the result, but the absence of external validation for the datasets means the 91.4% figure should be interpreted with more caution than results on established benchmarks like GSM8K.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper executes a conceptual reframing rather than a paradigm shift: it demonstrates that chain-of-thought reasoning is a latent zero-shot capability in LLMs rather than a skill that must be taught through few-shot examples. The magnitude of this reframing is substantial but bounded—it changes how the field understands why CoT works, but the practical gains over Few-shot-CoT are incremental (Zero-shot-CoT recovers ~84% of Few-shot-CoT's benefit on GSM8K, but Few-shot-CoT with carefully crafted examples remains the stronger method at 48.7% vs. 40.7%).

The most immediate impact is methodological: the paper establishes that zero-shot baselines on reasoning tasks are not trivially near-zero and must be reported. Before this work, Wei et al. (2022) did not report zero-shot numbers for reasoning benchmarks, and the field tacitly accepted that few-shot examples were necessary. After this work, any new reasoning method that reports only few-shot results is incomplete—there is now a demonstrated, simple, reproducible zero-shot baseline (the single prompt "Let's think step by step." followed by answer extraction) that any researcher can run. This alone makes the paper influential: it sets a new minimum standard for evaluation rigor in the reasoning-for-LLMs literature.

The paper also reconciles a latent contradiction in the prompting literature. Prior work oscillated between two narratives: (a) LLMs are fundamentally pattern matchers that learn tasks from in-context examples, with few-shot learning as the primary mechanism (Brown et al., 2020; Wei et al., 2022), and (b) LLMs encode broad capabilities that can be activated through instructions without examples (Sanh et al., 2022; Ouyang et al., 2022). The contradiction was whether reasoning specifically fell into category (a) or (b). This paper provides strong evidence for (b): the zero-shot trigger works across 12 diverse tasks, suggesting reasoning is a general capability accessed through the right prompt, not acquired on-the-fly from task-specific demonstrations. The few-shot examples in Wei et al. (2022) are demoted from "teaching reasoning" to "providing format guidance and marginal accuracy improvements," which is a meaningful downgrade in their causal role.

A less obvious but important shift: the paper reframes prompt engineering from a per-task craft to a capability-probing science. By showing that a single prompt works across arithmetic, symbolic manipulation, commonsense inference, and logical deduction, the paper demonstrates that prompts can target broad cognitive abilities rather than narrow task-specific skills. This introduces the Chollet (2019) vocabulary of "narrow generalization" vs. "broad generalization" into the prompting literature, creating a new axis along which prompts can be designed and evaluated: not just "does this prompt maximize accuracy on task X?" but "what general capability does this prompt elicit, and across what range of tasks?" The template robustness study (Table 4) with its instructive/misleading/irrelevant categorization provides a blueprint for systematically investigating this question.

For research directions, this work makes several areas more attractive. Investigating what other broad capabilities can be elicited through minimal zero-shot prompts becomes a natural extension with a clear template to follow. Studying the mechanism by which trigger phrases activate reasoning—why "Let's think step by step" outperforms "Let's think about this logically" by 4.2 points—becomes a tractable empirical question rather than pure speculation. Error analysis as a cognitive probe—comparing failure modes across prompting conditions to understand what models know versus what they can express—is validated as a productive methodology by the paper's Zero-shot-CoT vs. Few-shot-CoT error profile comparison (Table 23).

Conversely, this work makes some directions less urgent. The search for the optimal few-shot reasoning examples per task, while still important for maximizing accuracy, is partially superseded by the finding that a generic zero-shot trigger captures most of the gain. Researchers who were hand-crafting per-task CoT examples can now start from the zero-shot baseline and focus their effort on the marginal 10–15% that examples add, rather than building the entire reasoning scaffolding through examples. The paper also diminishes the case for fine-tuning specifically to enable reasoning—if a 540B model can reason zero-shot at 43.0% on GSM8K (and 70.1% with self-consistency), the bar for arguing that fine-tuning is necessary (rather than helpful) becomes much higher.

Follow-Up Research This Work Enables

Mechanistic investigation of why specific trigger phrasings outperform semantically similar alternatives. The paper's template robustness study (Table 4) reveals a 33-point accuracy spread among instructive templates (45.7% to 78.7%) on MultiArith, but provides no mechanistic explanation for the ranking. "Let's think step by step." (78.7%) outperforms "Let's solve this problem by splitting it into steps." (72.2%) by 6.5 points despite near-identical semantic content. Why? A follow-up study could systematically vary dimensions of the trigger: first-person plural ("Let's") vs. imperative ("Think step by step"), progressive aspect ("step by step") vs. nominal ("in steps"), procedural framing ("solve this problem") vs. cognitive framing ("think about this"). Measuring accuracy across these variants on 3–4 diverse reasoning benchmarks with at least two model families (InstructGPT, PaLM) would reveal whether the effect is driven by syntactic form, semantic priming, frequency in pretraining data, or some interaction. The paper's misleading-template results—particularly Template 11 ("Let's think step by step but reach an incorrect answer.", 18.7%)—suggest the model integrates the full instruction meaningfully, but a controlled decomposition would establish which linguistic features are necessary and sufficient.

Scaling the trigger phrase to smaller models through instruction tuning or distillation. The paper's most significant negative result is that Zero-shot-CoT provides essentially zero benefit for models below ~100B parameters (Tables 26–27, Appendix E). This restricts the method's practical use to the largest proprietary APIs. A direct follow-up would test whether instruction-tuning a smaller model (e.g., 7B–13B parameters) on examples that pair "Let's think step by step" with correct reasoning chains—similar to how Sanh et al. (2022) and Ouyang et al. (2022) taught models to follow instructions—can induce the zero-shot reasoning capability at smaller scales. The experiment would fine-tune a 7B model on a dataset of (question, trigger, reasoning, answer) tuples across diverse reasoning tasks, then measure whether the fine-tuned model generalizes Zero-shot-CoT to held-out reasoning tasks that it wasn't fine-tuned on. The paper's cross-dataset results (Table 5) suggest that reasoning format transfers across domains, making this plausible. Success would dramatically expand the method's accessibility; failure would strengthen the paper's emergence interpretation by showing the capability cannot be implanted through standard fine-tuning at small scales.

Self-consistency with Zero-shot-CoT on InstructGPT models to establish the zero-shot reasoning ceiling. The paper reports self-consistency results (sampling 40 reasoning paths with majority voting) only for PaLM 540B (Table 25, Appendix D): GSM8K jumps from 43.0% greedy to 70.1%, MultiArith from 66.1% to 89.0%. The primary model text-davinci-002 achieves substantially higher greedy Zero-shot-CoT accuracy than PaLM on these tasks (78.7% vs. 66.1% on MultiArith, 40.7% vs. 43.0% on GSM8K), but self-consistency is never tested on it. A direct experiment would run Zero-shot-CoT with non-zero temperature on text-davinci-002, sample 20–40 reasoning paths, apply majority voting, and measure the gap between greedy and self-consistency accuracy across all 12 datasets. Based on PaLM's gains (+17–27 points), text-davinci-002 might approach or exceed Few-shot-CoT + self-consistency numbers (e.g., 74.4% on GSM8K from Wang et al., 2022) using purely zero-shot prompts, which would further erode the case that few-shot examples provide essential capability rather than marginal format guidance.

Zero-shot-CoT as a diagnostic for comparing reasoning capability across model releases. The paper's finding that a single fixed prompt works across 12 diverse tasks with zero modification makes it a candidate for a standardized reasoning audit. A research team could define a protocol: take any new LLM (or model checkpoint during training), prompt it with the exact Zero-shot-CoT template from this paper across the 12 datasets (or an expanded set), and report a single "Zero-shot-CoT reasoning score" as the average across tasks. This would enable apples-to-apples comparisons of reasoning capability across models from different organizations, architectures, and training procedures—something the field currently lacks, since most model releases report different prompting setups and task selections. The protocol would need to address answer extraction consistency (using the paper's Appendix A.5 templates and A.6 cleansing) and model scale thresholds (only meaningful above ~60B parameters based on the paper's scaling results). The paper's cross-model results (GPT-3, InstructGPT, PaLM) demonstrate that the prompt transfers across model families, making this protocol technically straightforward to implement.

Error propagation analysis through the two-stage pipeline to quantify how often correct Stage 1 reasoning is lost in Stage 2 extraction. The paper's error analysis (Appendix C.2, Table 23) identifies "one unnecessary step" errors where the model reaches the correct answer in Stage 1 reasoning but then over-reasons and changes its answer before Stage 2 extraction. But the paper never quantifies how often Stage 2 fails to extract the correct answer from reasoning that is otherwise sound. A targeted error analysis would take a sample of 100–200 Zero-shot-CoT outputs, manually annotate whether the Stage 1 reasoning chain (a) contains the correct answer somewhere, (b) reaches it as the final conclusion, and (c) whether Stage 2 extracts that answer correctly. This would decompose the overall error rate into three sources: reasoning never reaches the right answer (fundamental capability gap), reasoning reaches but then abandons the right answer (stopping-criterion failure), and reasoning contains the right answer but extraction misses it (parsing failure). The paper's data hints that stopping-criterion failures are significant (10% of MultiArith errors), but a rigorous decomposition would tell us whether the priority for improving Zero-shot-CoT should be better reasoning, better stopping, or better extraction.

Adversarial testing of the trigger mechanism through systematic semantic perturbation. The paper's misleading templates (Table 4) show that adding "but reach an incorrect answer" to "Let's think step by step" collapses accuracy from 78.7% to 18.7%. This opens a broader research question: how semantically coherent does an anti-reasoning instruction need to be to suppress the effect? A follow-up could systematically vary the misleading instruction along axes of grammaticality ("Think step but wrong"), semantic contradiction strength ("think step by step but guess"), and position (prepended vs. appended vs. interspersed) to map the boundaries of the model's instruction sensitivity. Finding that grammatically valid but semantically nonsensical instructions ("Think step by step using only prime numbers") suppress reasoning would suggest the model is responding to surface-level instruction-following rather than deep semantic understanding. Finding that only coherent, semantically interpretable counter-instructions suppress reasoning would strengthen the paper's implicit claim that the model genuinely understands the instruction to reason.

Practical Applications and Downstream Use Cases

Rapid prototyping of reasoning-based applications without per-task example engineering. A developer building a question-answering system over a new domain (e.g., internal documentation, legal contracts, medical guidelines) can deploy Zero-shot-CoT immediately by wrapping each user query in the "Q: [query] A: Let's think step by step." template and running the two-stage extraction. They do not need to curate domain-specific reasoning examples, hire annotators to write step-by-step solutions, or iterate on example quality—the generic trigger provides substantial reasoning capability out of the box. Based on the paper's results, they can expect roughly 75–85% of the accuracy of a carefully-engineered few-shot system (Zero-shot-CoT at 78.7% vs. 8-shot-CoT at 93.0% on MultiArith), at zero example-engineering cost and with prompt-engineering effort limited to choosing the answer extraction format. For applications where 80%-of-optimal is sufficient for an MVP or internal tool, this eliminates the primary bottleneck to deploying reasoning-capable LLM systems.

Cost-efficient batch evaluation of reasoning benchmarks for model monitoring. Organizations that regularly evaluate LLMs on reasoning benchmarks (for model selection, regression testing, or capability monitoring) can reduce evaluation costs by using Zero-shot-CoT instead of Few-shot-CoT. Few-shot-CoT requires including 8 reasoning examples in every prompt, which can add thousands of tokens to each API call. For a benchmark like GSM8K with 1,319 test questions, the few-shot examples add roughly 500–1,000 tokens per question, or ~1M extra tokens over the full evaluation. Zero-shot-CoT uses a short fixed trigger (4 tokens for "Let's think step by step.") but requires 2 API calls per question instead of 1. The cost tradeoff depends on the relative cost of input tokens (higher for few-shot) vs. additional API calls (higher for Zero-shot-CoT), but the zero-shot approach also eliminates the labor cost of maintaining and versioning few-shot examples as models and tasks evolve.

Self-consistency-based answer verification in high-stakes settings. The paper shows that combining Zero-shot-CoT with self-consistency (sampling 40 reasoning paths, majority voting) pushes PaLM 540B to 70.1% on GSM8K and 89.0% on MultiArith (Table 25). For applications where accuracy matters more than latency—medical decision support, financial analysis, legal research—a deployment could run Zero-shot-CoT with 20–40 samples in parallel, apply majority voting, and flag answers where the vote is split (indicating uncertainty) for human review. The key advantage over Few-shot-CoT + self-consistency is simplicity: the same sampling-and-voting infrastructure works across all question types without per-domain example management. The paper's results suggest this approach is competitive with or exceeds fine-tuned systems (finetuned GPT-3 175B + verifier at 55% on GSM8K vs. Zero-shot-CoT + self-consistency at 70.1%), making it viable even in domains where fine-tuning data is scarce or expensive to annotate.