Monte Carlo, Puppetry and Laughter: The Unexpected Joys of Prompt Engineering

ArXiv: 2201.11903

🎯 Pitch

Simply showing a language model a few examples that spell out intermediate reasoning steps—instead of just input–output pairs—unlocks powerful multi-step problem solving that was previously absent, yet this ability only emerges abruptly in models with ~100B parameters; smaller models actually perform worse. The approach requires no fine-tuning or new architectures, tripling accuracy on math word problems with just eight exemplars and even beating task-specific trained systems.

1. Executive Summary

This paper introduces chain-of-thought prompting, a method that elicits multi-step reasoning in large language models by providing few-shot exemplars that include intermediate natural language reasoning steps alongside input–output pairs (e.g., decomposing a math word problem into sub-steps before stating the answer). Experiments on PaLM 540B, GPT-3 175B, and LaMDA 137B across arithmetic (GSM8K, SVAMP, MAWPS, ASDiv, AQuA), commonsense (CSQA, StrategyQA, Date Understanding, Sports Understanding, SayCan), and symbolic reasoning (last letter concatenation, coin flip) benchmarks demonstrate that chain-of-thought prompting more than doubles standard prompting performance on challenging tasks like GSM8K—with PaLM 540B achieving 56.9% accuracy to surpass finetuned GPT-3 with a verifier—while also enabling length generalization to inputs longer than those seen at training time. The paper establishes that chain-of-thought reasoning is an emergent ability of model scale, producing improvements only when models reach approximately 100B parameters, but degrading performance in smaller models that generate fluent yet logically incoherent intermediate steps.

2. Context and Motivation

The Core Problem: Language Models Struggle with Multi-Step Reasoning Despite Scale

By early 2022, the NLP field had been transformed by the scaling paradigm—bigger models, more data, and more compute consistently yielded better performance across a wide range of benchmarks (Kaplan et al., 2020; Brown et al., 2020). However, a puzzling gap remained: scaling up model size alone had not proven sufficient for achieving high performance on tasks requiring multi-step reasoning, such as arithmetic word problems, commonsense inference, and symbolic manipulation (Rae et al., 2021). The paper opens by explicitly naming this gap:

"scaling up model size alone has not proved sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning"

This is the central problem the paper addresses. Large language models (LLMs) could generate fluent text, answer factual questions, and perform simple pattern matching via few-shot prompting, but they struggled to decompose complex problems into intermediate steps and reason through them sequentially. The GSM8K benchmark of math word problems (Cobbe et al., 2021) exemplified this: even the largest models at the time achieved only 15–18% accuracy with standard prompting, despite possessing the underlying arithmetic knowledge needed to solve individual steps.

Why This Problem Matters

The inability to perform multi-step reasoning is not a niche limitation—it is a bottleneck that blocks LLMs from a vast class of practically important tasks:

• Arithmetic reasoning: Math word problems require models to parse a natural language scenario, extract relevant quantities, determine the correct sequence of operations, and execute those operations. This is foundational for any application involving numerical reasoning (financial analysis, scientific computation, engineering).
• Commonsense reasoning: Questions about everyday physical and social interactions require chaining together implicit world knowledge. For example, determining whether "Joao Moutinho caught the screen pass in the NFC championship" is plausible requires knowing that Joao Moutinho is a soccer player, that the NFC championship is an American football event, and that these are incompatible.
• Symbolic reasoning: Tasks like concatenating the last letters of words in a name or tracking the state of a coin through multiple flips require systematic manipulation of symbols—a capability that underpins algorithmic reasoning, code execution, and formal logic.
• Robotic instruction following: The SayCan task (Ahn et al., 2022) requires mapping natural language instructions to sequences of discrete robot actions, demanding both semantic understanding and multi-step planning.

Theoretically, the gap is significant because it reveals a fundamental limitation of the few-shot prompting paradigm. Brown et al. (2020) had shown that LLMs can learn new tasks from a handful of input–output examples without gradient updates—a remarkable finding that suggested a path toward general-purpose models. However, as the paper notes:

"for the traditional few-shot prompting method used in Brown et al. (2020), it works poorly on tasks that require reasoning abilities, and often does not improve substantially with increasing language model scale"

This is a crucial observation: the scaling curves for reasoning tasks are flat. Making models larger (e.g., from 8B to 62B to 540B parameters) produces minimal gains on GSM8K under standard prompting, suggesting that raw scale alone is not the answer. Something about the interface between the model and the task—the way the task is presented—needed to change.

Prior Approaches and Their Limitations

The paper identifies two major lines of prior work that each address part of the problem but have fundamental shortcomings:

Rationale-Augmented Training and Fine-Tuning

A substantial body of work had shown that providing models with natural language intermediate steps (rationales, explanations, or solution steps) during training improves reasoning performance. The key papers in this lineage include:

• Ling et al. (2017): Pioneered the idea of generating natural language rationales to solve math word problems, showing that models trained from scratch to produce step-by-step solutions significantly outperformed models that directly predicted answers.
• Cobbe et al. (2021): Extended this by creating GSM8K, a large dataset of math word problems with human-written solution steps, and using it to fine-tune pretrained language models (GPT-3). Their fine-tuned model with a trained verifier achieved 55% accuracy on GSM8K—the prior state of the art.
• Nye et al. (2021): In the program synthesis domain, showed that language models predicting intermediate line-by-line computation results outperform direct final-output prediction.

This line of work demonstrated that intermediate reasoning steps work—models can leverage them to solve problems they otherwise couldn't. However, the approach has a critical practical limitation:

"it is costly to create a large set of high quality rationales, which is much more complicated than simple input–output pairs used in normal machine learning"

Creating training data for rationale-augmented learning requires human annotators to write detailed step-by-step solutions for thousands of problems—a process that is expensive, time-consuming, and hard to scale across diverse tasks. Moreover, the resulting models are task-specific: a model fine-tuned on math word problems cannot also perform commonsense reasoning or symbolic manipulation without additional fine-tuning data for each new task. This contradicts the vision of a single general-purpose model that can perform many tasks.

Few-Shot Prompting Without Intermediate Steps

The second major line of work, popularized by Brown et al. (2020), showed that LLMs can perform new tasks without any fine-tuning simply by conditioning on a few input–output exemplars in the prompt. This few-shot prompting paradigm eliminated the need for large task-specific training datasets and allowed a single model checkpoint to serve many tasks. The key insight was that instruction or demonstration could substitute for gradient updates.

However, this approach hit a wall on reasoning tasks. The paper observes that standard few-shot prompting—where exemplars consist only of ⟨input, output⟩ pairs, asking the model to directly produce the answer—produces flat scaling curves: performance barely improves as model size increases. For example, on GSM8K, standard prompting with PaLM 8B achieved 4.9%, PaLM 62B achieved 9.6%, and PaLM 540B achieved only 17.9% (Table 2). The model was not "learning to reason" from the exemplars; it was attempting to pattern-match from question to answer, which fails when the reasoning chain is complex and variable.

Neuro-Symbolic and Formal Language Methods

A separate line of work tackled reasoning by converting problems into formal languages (e.g., mathematical equations, logical expressions, or executable programs) and using specialized solvers or executors (Roy and Roth, 2015; Chiang and Chen, 2019; Amini et al., 2019; Chen et al., 2019). These methods achieved strong results on specific benchmarks but were domain-specific: a math equation solver doesn't help with commonsense reasoning or date understanding. They required hand-engineered formalisms for each new domain, limiting generality.

How This Paper Positions Itself

The paper's key conceptual move is to synthesize the strengths of these two lines of work while avoiding their weaknesses. Specifically:

1. From rationale-augmented training, it takes the idea of intermediate reasoning steps: The paper recognizes that generating natural language reasoning chains is a powerful mechanism for solving multi-step problems. But instead of requiring expensive fine-tuning datasets with thousands of human-written rationales, it asks: can we elicit this capability through prompting alone, using just a handful of examples?

2. From few-shot prompting, it takes the idea of task specification without fine-tuning: The paper retains the elegant interface of providing a few exemplars in the prompt. But instead of ⟨input, output⟩ pairs, it augments exemplars to be ⟨input, chain of thought, output⟩ triples—showing the model not just what the answer is, but how to think through the problem to arrive at it.

The result is chain-of-thought prompting, defined as:

"a series of intermediate natural language reasoning steps that lead to the final output"

This approach is positioned as having several attractive properties (Section 2, the four enumerated points):

• It decomposes multi-step problems into manageable sub-steps, allowing additional computation (intermediate tokens) to be allocated to harder problems.
• It provides an interpretable window into the model's reasoning process, enabling debugging when the model errs.
• It is broadly applicable to any task humans solve via language-based reasoning—math, commonsense, symbolic manipulation, and potentially beyond.
• It requires no fine-tuning: the capability is elicited in off-the-shelf models simply by including chain-of-thought demonstrations in the prompt.

Critically, the paper positions this approach as revealing something about the nature of large language models: that chain-of-thought reasoning is an emergent ability that only appears at sufficient scale. Small models (under ~10B parameters) actually perform worse with chain-of-thought prompting than with standard prompting, because they produce fluent-sounding but logically incoherent reasoning chains. The capability to generate and leverage coherent intermediate reasoning only "turns on" at around 100B parameters. This positions the paper not just as a method proposal but as an investigation into what capabilities are latent in large language models and how to unlock them.

The Unifying Insight

The paper's framing can be understood as addressing a fundamental tension: fine-tuning works but requires expensive data; prompting is cheap but doesn't work for reasoning. Chain-of-thought prompting resolves this tension by showing that the reasoning capability is already present in sufficiently large models—it just needs the right interface to be expressed. The eight manually-written exemplars used in the paper (Table 20) are a one-time cost, after which the same prompt works across multiple math datasets (GSM8K, SVAMP, ASDiv, MAWPS, AQuA) and the approach generalizes to commonsense and symbolic reasoning with analogous exemplars.

This reframes the reasoning problem from one of training data scarcity to one of prompt engineering: the knowledge and reasoning procedures are in the model; the challenge is eliciting them. This is a conceptual shift with practical significance: it suggests that improving reasoning in LLMs may not require ever-larger fine-tuning datasets, but rather better methods for structuring prompts to guide the model's existing capabilities.

3. Technical Approach

3.1 Reader Orientation

Chain-of-thought prompting is a prompt engineering method—not a new model architecture, training procedure, or fine-tuning recipe. The "system" being built is simply a text prompt (a sequence of exemplars followed by a test question) fed into an off-the-shelf, frozen large language model. The core problem it solves is that standard few-shot prompting, which provides only ⟨input, output⟩ pairs, fails on multi-step reasoning tasks because the model is asked to map directly from a complex question to its answer without intermediate computation. The shape of the solution is elegant in its simplicity: augment each few-shot exemplar with a natural language chain of thought—a series of intermediate reasoning steps that decompose the problem and lead to the final answer—so the model learns, through in-context imitation, to generate its own reasoning chain before producing the answer for a new question.

3.2 Big-Picture Architecture (Diagram in Words)

The system has only three components, and the entire "pipeline" is a single forward pass through a frozen language model:

1. The Prompt Template — a hand-crafted sequence of exemplars, each formatted as a triple: ⟨input question, chain-of-thought reasoning, output answer⟩. This template is identical for all test examples within a task (e.g., the same 8 exemplars are used for all GSM8K problems). The chain of thought is always placed before the final answer, so the model sees reasoning-then-answer as the expected output pattern.

2. The Frozen Large Language Model — an off-the-shelf pretrained model (GPT-3, LaMDA, PaLM, UL2, or Codex) with no fine-tuning, no gradient updates, and no parameter modifications. The model receives the prompt template concatenated with the test question and generates tokens autoregressively.

3. The Generated Output — the model's autoregressive continuation, which (when chain-of-thought prompting works) consists of a chain of reasoning steps followed by the final answer. The answer is extracted via simple string parsing (e.g., looking for "The answer is X" pattern). For arithmetic tasks, an optional external calculator can be applied post-hoc to evaluate any mathematical expressions in the generated chain.

Information flow is sequential and single-pass: the prompt (exemplars + test question) is fed into the model → the model generates tokens one by one, conditioned on all preceding tokens → the generation ideally follows the demonstrated pattern of step-by-step reasoning → the final answer is extracted. There is no iterative refinement, no search, no verifier, and no feedback loop during generation—the "reasoning" happens entirely within the autoregressive generation of a single output sequence.

3.3 Roadmap for the Deep Dive

This section breaks down chain-of-thought prompting into its constituent design decisions and mechanisms, proceeding from the most concrete (what exactly goes into the prompt) to the most abstract (why the method works and when it fails):

• First, the prompt construction methodology: exactly how exemplars are selected, how chains of thought are written, how prompts are formatted, and what variations were tested in ablation studies. This is the "recipe" that practitioners need to replicate the method.
• Second, the inference procedure: how the model generates from the prompt, how answers are extracted, and the role of greedy decoding versus sampling.
• Third, the ablation experiments that isolate why chain-of-thought prompting works: equation-only prompting, variable-computation-only prompting, and reasoning-after-answer prompting—each designed to test a specific hypothesis about the mechanism.
• Fourth, the emergent ability phenomenon: why model scale matters, what happens when small models attempt chain-of-thought reasoning, and the qualitative error analysis that reveals the nature of the scale-dependent capability.
• Fifth, the robustness analysis: how sensitive the method is to different annotators, different exemplars, different numbers of exemplars, and different prompt orderings.
• Sixth, the task-specific adaptations: how the general method is instantiated differently for arithmetic, commonsense, and symbolic reasoning tasks, including the external calculator augmentation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core idea is that multi-step reasoning can be elicited from sufficiently large language models simply by including examples of step-by-step reasoning in the few-shot prompt. No models are trained, fine-tuned, or architecturally modified. The entire contribution is a prompting strategy and the empirical demonstration of its effectiveness, scaling properties, and robustness.

Prompt Construction: The Exemplar Design Recipe

The central mechanism of chain-of-thought prompting is the construction of few-shot exemplars as triples rather than pairs. In standard few-shot prompting (Brown et al., 2020), each exemplar is formatted as:

Q: [question text]
A: [final answer]

The model, conditioned on several such exemplars and then the test question, learns to directly output the answer. For chain-of-thought prompting, each exemplar is instead formatted as:

Q: [question text]
A: [step-by-step reasoning in natural language, culminating in the final answer]

The critical design choice is that the reasoning chain precedes and subsumes the answer—there is no separate "reasoning" and "answer" field; the answer is the natural conclusion of the reasoning chain, typically marked with a phrase like "The answer is X." This means the model sees a single coherent output that decomposes the problem into sub-steps and then states the conclusion, reinforcing the pattern that intermediate computation should precede the final answer.

Exemplar Selection for Arithmetic Reasoning

For all arithmetic reasoning benchmarks except AQuA, the authors used a fixed set of 8 exemplars that they manually composed. The full prompt is provided in Table 20 of the paper. The exemplars were not drawn from any specific training set; they were written by the authors (referred to as "Annotator A") to demonstrate the general pattern of step-by-step arithmetic reasoning. An example from Table 20:

Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. 
After they are done, there will be 21 trees. How many trees did the grove workers plant today?
A: There are 15 trees originally. Then there were 21 trees after some more were planted. 
So there must have been 21 - 15 = 6. The answer is 6.

Several design decisions are evident in this exemplar. First, the chain of thought restates the given information in the model's own words ("There are 15 trees originally"), demonstrating that the model should parse and verbalize the problem state. Second, it identifies the relationship between quantities ("Then there were 21 trees after some more were planted")—this is the semantic reasoning step that translates natural language into mathematical structure. Third, it produces the equation ("So there must have been 21 - 15 = 6")—the formal mathematical operation. Fourth, it states the answer ("The answer is 6") in a consistent format that enables reliable parsing.

The 8 exemplars vary in the arithmetic operations required (addition, subtraction, multiplication, division) and in the number of reasoning steps (from simple one-step problems like "3 + 2 = 5" to multi-step problems like computing computers added over 4 days). This variety is intentional: it demonstrates to the model that the pattern generalizes across problem types and can be composed for problems requiring multiple operations.

For the AQuA dataset, which is multiple choice rather than free response, the authors used 4 exemplars from the AQuA training set (Table 21). These exemplars include the answer choices in the question and demonstrate reasoning that selects among the options. For example:

Q: John found that the average of 15 numbers is 40. If 10 is added to each number 
then the mean of the numbers is?
Answer Choices: (a) 50 (b) 45 (c) 65 (d) 78 (e) 64
A: If 10 is added to each number, then the mean of the numbers also increases by 10. 
So the new mean would be 50. The answer is (a).

This exemplar shows a different reasoning pattern: rather than computing from scratch, it uses the property that adding a constant to all numbers shifts the mean by that constant—demonstrating that the model can reason about mathematical properties rather than just performing arithmetic.

Exemplar Design for Other Reasoning Domains

The paper demonstrates that the same principle—augmenting exemplars with chains of thought—generalizes beyond arithmetic. For each task domain, the authors manually composed chains of thought following the same pattern: decompose the problem into intermediate steps, verbalize the reasoning, and conclude with the answer.

Commonsense reasoning (CSQA and StrategyQA): The chains of thought follow a deductive elimination pattern. For CSQA (Table 24), each chain states a constraint that the answer must satisfy, then eliminates options that don't satisfy it:

Q: What do people use to absorb extra ink from a fountain pen? 
Answer Choices: (a) shirt pocket (b) calligrapher's hand (c) inkwell (d) desk drawer (e) blotter
A: The answer must be an item that can absorb ink. Of the above choices, 
only blotters are used to absorb ink. So the answer is (e).

For StrategyQA (Table 25), which requires yes/no answers to questions requiring implicit multi-hop reasoning, the chains explicitly connect facts through logical steps:

Q: Do hamsters provide food for any animals?
A: Hamsters are prey animals. Prey are food for predators. Thus, hamsters provide food 
for some animals. So the answer is yes.

This chain demonstrates three reasoning steps: (1) classify hamster as prey, (2) invoke the general rule that prey are food for predators, (3) conclude that hamsters provide food. The model must learn from such exemplars to retrieve relevant facts and chain them together.

Symbolic reasoning (last letter concatenation and coin flip): For these toy tasks, the chains of thought are almost algorithmic—they demonstrate a deterministic step-by-step procedure that the model can replay with new symbols. For last letter concatenation (Table 22):

Q: Take the last letters of the words in "Elon Musk" and concatenate them.
A: The last letter of "Elon" is "n". The last letter of "Musk" is "k". 
Concatenating them is "nk". The answer is nk.

This exemplar demonstrates the procedure: (1) identify each word, (2) extract the last letter of each, (3) concatenate in order, (4) state the answer. The model need not understand why this procedure works—it only needs to replicate the symbol manipulation pattern with new names.

For coin flip (Table 23):

Q: A coin is heads up. Ka flips the coin. Sherrie flips the coin. Is the coin still heads up?
A: The coin was flipped by Ka and Sherrie. So the coin was flipped 2 times, 
which is an even number. The coin started heads up, so after an even number of flips, 
it will still be heads up. So the answer is yes.

This demonstrates: (1) count how many people flipped the coin, (2) determine whether the count is even or odd, (3) apply the rule that even flips preserve the original state while odd flips invert it. The exemplar set (Table 23) includes examples with 0, 1, and 2 flips to cover all cases the model might encounter.

Robot instruction following (SayCan): The chains of thought here serve a dual purpose: they explain why certain actions are appropriate (providing semantic grounding) and then enumerate the action sequence. From Table 28:

Human: How would you bring me something to eat that isn't a fruit?
Explanation: The user has asked for a food that isn't a fruit, I will bring an energy bar to them.
Plan: 1. find(energy bar), 2. pick(energy bar), 3. find(user), 4. put(energy bar), 5. done().

Prompt Formatting Conventions

The paper uses consistent formatting across all prompts:

• Questions are prefixed with Q: and answers/reasoning with A: for math problems
• For classification tasks, answer choices are presented as a list
• For yes/no questions, the expected answer is "yes" or "no" as the final token after reasoning
• For date tasks, the date format MM/DD/YYYY is consistently used
• Chain-of-thought reasoning always appears before the final answer, never after
• The final answer is demarcated with phrases like "The answer is X" or "So the answer is X" to enable reliable extraction

The total prompt length is constrained by the model's context window (1024 tokens for the models used). This limits the number of exemplars to approximately 8 for most tasks, though robustness experiments varied this number (Section 3.4, Figure 11).

Why Hand-Written Exemplars Over Training-Set Sampling?

A natural alternative would be to randomly sample exemplars from a training set that already contains reasoning chains (e.g., the GSM8K training set includes human-written solutions). The paper tests this (Section 3.4, Figure 6, and Table 6) and finds that randomly sampled GSM8K exemplars also work—they outperform standard prompting by a large margin. However, the manually composed exemplars were used for the main experiments for several practical reasons:

• Cross-dataset generalization: The same 8 manually written exemplars were used for GSM8K, SVAMP, ASDiv, and MAWPS—the authors wanted to test whether exemplars from one distribution could elicit reasoning on different math word problem distributions. The results (Table 6, "exemplars from GSM8K" rows) confirm that this works: GSM8K-sourced exemplars improve performance on SVAMP, ASDiv, and MAWPS, even though those datasets have different problem styles.
• Control over exemplar properties: Manual composition allowed the authors to ensure exemplars demonstrated a range of arithmetic operations and reasoning depths, avoiding degenerate cases (e.g., exemplars that are trivially simple or too long to fit in context).
• Most benchmarks lack training sets: SVAMP, ASDiv, and MAWPS only have evaluation splits; there are no training examples to sample from. Using a single manually composed set enables consistent evaluation across all benchmarks.

Inference Procedure: Greedy Decoding and Answer Extraction

Once the prompt is constructed, inference proceeds as a standard autoregressive generation:

1. The full prompt (exemplars + test question with Q: prefix and A: prompt) is fed to the model.
2. The model generates tokens one at a time, conditioned on all preceding tokens, until a stop condition is reached (typically a maximum length or end-of-sequence token).
3. For most experiments, the authors use greedy decoding—at each step, the token with the highest probability is selected. This means the generated chain of thought is deterministic for a given prompt and model.
4. The final answer is extracted from the generated text by searching for the pattern "The answer is X" or equivalent variations.

For LaMDA experiments specifically, the authors report results averaged over five random seeds, where each seed uses a different randomly shuffled order of the few-shot exemplars. This accounts for the sensitivity of in-context learning to exemplar order (Zhao et al., 2021). For GPT-3, PaLM, Codex, and UL2, results are reported for a single exemplar order to save compute, with the justification that LaMDA experiments showed relatively low variance across seeds.

External Calculator Augmentation (Post-Hoc)

A key observation from the error analysis (Appendix D.2) is that many model-generated chains of thought contain correct reasoning but incorrect arithmetic—the model correctly identifies which operations to perform and in what order, but makes arithmetic errors when computing the result (e.g., computing 3 × 25 × 8 = 300 instead of 600). This is a calculator error, not a reasoning error.

To address this, the authors apply an external calculator as a post-processing step on the generated chain of thought (Section 3.2, Table 1, "ext. calc" rows). The procedure works as follows:

1. Parse the generated chain of thought to extract all mathematical equations.
2. Evaluate each equation using Python's eval() function.
3. If an equation evaluates to a different value than what the model stated, replace the model's computed value with the correct value.
4. Propagate the corrected values through subsequent equations in the chain (since later steps often depend on earlier computed values). This propagation is done via string matching: if an earlier equation computes "X = 300" but the correct value is "X = 600", all subsequent references to "X = 300" or "300" in the context of X are replaced with "600".

This external calculator consistently improves performance across all arithmetic benchmarks (Table 1). For example, LaMDA 137B on GSM8K improves from 14.3% (chain-of-thought only) to 17.8% (chain-of-thought + calculator), and PaLM 540B improves from 56.9% to 58.6%. The gains are largest for models that produce correct reasoning but poor arithmetic—the calculator decouples reasoning quality from arithmetic precision.

Importantly, the external calculator is not part of the prompting method itself—it is a post-hoc correction that could be applied to any method that generates equations. The paper treats it as a complementary technique rather than a core contribution.

Majority Voting (Not in This Paper, but Noted)

The paper briefly notes (Section 3.1) that follow-up work by Wang et al. (2022a) shows chain-of-thought prompting can be improved by sampling multiple generations (with non-zero temperature) and taking the majority vote over final answers. This is mentioned as a natural extension but is not used in the paper's experiments, which all use greedy decoding.

Ablation Experiments: Isolating the Mechanism

Section 3.3 (Figure 5) presents three ablation experiments designed to test specific hypotheses about why chain-of-thought prompting improves performance. Each ablation isolates one potential mechanism and shows it is insufficient to explain the gains.

Equation-Only Prompting

Hypothesis tested: Chain-of-thought prompting helps because it produces the mathematical equation to be evaluated—the natural language reasoning in between is unnecessary; the model just needs to output the equation before the answer.

Method: The prompt exemplars are modified so that, instead of full natural language reasoning, the model outputs only the mathematical equation (or a minimal symbolic representation) before the answer. For example, instead of:

A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 
5 + 6 = 11. The answer is 11.

the exemplar becomes:

A: 5 + 2 * 3 = 11. The answer is 11.

Result (Figure 5, "Equation only" bar): On GSM8K, equation-only prompting performs substantially worse than full chain-of-thought prompting and only marginally better than standard prompting (for LaMDA 137B: 5.4% vs. 14.3% for chain-of-thought; for PaLM 540B: the gap is even larger). This shows that the natural language reasoning steps are essential—the model cannot reliably translate the semantics of a complex word problem directly into a mathematical equation without the intermediate verbal reasoning.

Caveat (Appendix Table 6): On simpler datasets where problems require only one or two steps (SVAMP, ASDiv, subsets of MAWPS), equation-only prompting does help—because the semantic gap between the problem text and the equation is small enough for the model to bridge without verbal mediation. For instance, on SVAMP, LaMDA 137B achieves 35.1% with equation-only vs. 29.5% for standard prompting. This suggests the natural language reasoning is most valuable when the problem requires multi-step semantic decomposition.

Variable-Compute-Only Prompting

Hypothesis tested: Chain-of-thought prompting helps because it allows the model to spend more computation (in the form of intermediate tokens) on harder problems. The content of those tokens doesn't matter—only the additional computational budget they represent. This is analogous to the "scratchpad" idea: the model benefits from having space to "think," regardless of what it writes.

Method: The prompt exemplars are modified so that, instead of natural language reasoning, the model outputs a sequence of dots (...) equal in length (number of characters) to the equation that would solve the problem. For example, if the correct equation is "21 - 15 = 6" (12 characters), the model would output 12 dots before giving the answer. This controls for the quantity of intermediate computation while removing the semantic content.

Result (Figure 5, "Variable compute only" bar): This variant performs about the same as the standard prompting baseline (no intermediate computation at all). For GSM8K with LaMDA 137B: 6.4% vs. 6.5% baseline; with PaLM 540B: the gap is similarly negligible. This demonstrates that the semantic content of the intermediate tokens matters—the model is not just benefiting from extra computation time. The reasoning expressed in natural language is what enables better final answers.

Why this result matters: This ablation directly addresses a subtle alternative explanation. Large language models perform a fixed amount of computation per token generated (each token requires one forward pass through all layers). By generating a chain of thought, the model is essentially allocating more FLOPs to the problem. One might hypothesize that this additional computation, regardless of what tokens are generated, allows the model to "think longer" and arrive at better answers. The dots experiment rules this out: if extra tokens alone helped, dots would work. They don't, which means the model is genuinely using the semantic content of the reasoning chain to structure its computation.

Reasoning-After-Answer Prompting

Hypothesis tested: Chain-of-thought prompting helps because the exemplars expose the model to relevant knowledge or patterns that it can then apply—the chain of thought serves as additional context that activates useful information, but the model doesn't actually depend on the sequential reasoning process to produce the answer. In other words, the model might generate the answer first and only then produce an explanation, with the explanation being epiphenomenal.

Method: The prompt exemplars are modified so that the answer is given first, followed by the chain of thought (i.e., the reasoning appears after the answer, as a post-hoc explanation). For example:

A: The answer is 6. (There are 15 trees originally. Then there were 21 trees after 
some more were planted. So there must have been 21 - 15 = 6.)

At test time, the model sees exemplars where the answer precedes the reasoning, so it is prompted to produce the answer first.

Result (Figure 5, "Reasoning after answer" bar): This variant also performs about the same as the standard prompting baseline. For GSM8K with LaMDA 137B: 6.1% vs. 6.5%; with PaLM 540B: the pattern holds. This shows that the sequential, step-by-step reasoning before the answer is crucial—the model must think through the problem before committing to an answer. The reasoning is not merely activating knowledge; it is performing a causal role in computing the answer.

Why this result matters: This is the most theoretically important ablation because it distinguishes chain-of-thought prompting from explanation-based training methods where rationales are used for interpretability or knowledge distillation. If reasoning-after-answer worked, one could argue that chain-of-thought prompting simply provides more relevant context (analogous to adding more training data to the prompt). The fact that the order matters—reasoning must precede the answer—suggests that the model is using the intermediate tokens to iteratively build toward the solution, with each step's output conditioning the next.

Summary of Ablation Results

The three ablations collectively establish that chain-of-thought prompting works through a specific mechanism: the model uses natural language to decompose a problem into sequential intermediate reasoning steps, where each step's output causally contributes to the final answer. The method succeeds not because of extra computation per se (dots don't work), not because of simple equation extraction (equations alone don't suffice for hard problems), and not because of knowledge activation (reasoning after answer doesn't help). The sequential, semantic, step-by-step verbal reasoning is the active ingredient.

The Emergent Ability Phenomenon: Why Scale Matters

One of the paper's central findings is that chain-of-thought prompting is an emergent ability—it does not positively impact performance for small models and only yields gains when used with models of sufficient scale (approximately 100B parameters). This is visible across all three model families tested (Section 3.2, Figure 4 and Table 2).

Quantitative Evidence for Emergence

Table 2 shows the performance of standard prompting vs. chain-of-thought prompting across model sizes for each model family. The pattern is consistent:

LaMDA (GSM8K):

• 422M parameters: standard = 2.6%, chain-of-thought = 0.4% (performance decreases)
• 2B parameters: standard = 3.6%, chain-of-thought = 1.9% (decrease)
• 8B parameters: standard = 3.2%, chain-of-thought = 1.6% (decrease)
• 68B parameters: standard = 5.7%, chain-of-thought = 8.2% (first positive gain)
• 137B parameters: standard = 6.5%, chain-of-thought = 14.3% (substantial gain)

GPT-3 (GSM8K):

• 350M parameters: standard = 2.2%, chain-of-thought = 0.5% (decrease)
• 1.3B parameters: standard = 2.4%, chain-of-thought = 0.5% (decrease)
• 6.7B parameters: standard = 4.0%, chain-of-thought = 2.4% (slight decrease)
• 175B parameters: standard = 15.6%, chain-of-thought = 46.9% (massive gain—performance triples)

PaLM (GSM8K):

• 8B parameters: standard = 4.9%, chain-of-thought = 4.1% (slight decrease)
• 62B parameters: standard = 9.6%, chain-of-thought = 29.9% (gain appears)
• 540B parameters: standard = 17.9%, chain-of-thought = 56.9% (massive gain—performance more than triples)

The pattern is non-monotonic and exhibits a phase transition: below approximately 50–70B parameters, chain-of-thought prompting either hurts or provides negligible benefit; above approximately 100B parameters, it yields large and growing gains. This is the hallmark of an emergent ability (Wei et al., 2022b): a capability that is not present at smaller scales and cannot be predicted by extrapolating from small-model performance.

The same pattern holds across arithmetic benchmarks (GSM8K, SVAMP, ASDiv, MAWPS—Table 2), commonsense reasoning benchmarks (CSQA, StrategyQA, Date, Sports, SayCan—Table 4), and symbolic reasoning tasks (Table 5). On commonsense tasks, the threshold for emergence appears somewhat lower: LaMDA 68B shows gains on CSQA (37.0% → 44.1%) and StrategyQA (59.6% → 62.2%), suggesting that the semantic knowledge required for commonsense reasoning is accessible at smaller scales than the procedural knowledge required for multi-step arithmetic.

Qualitative Error Analysis: What Goes Wrong in Small Models

The paper provides qualitative insight into why small models fail at chain-of-thought reasoning (Appendix A.1). The authors note that small models "produced fluent but illogical chains of thought, leading to lower performance than standard prompting." The error analysis of LaMDA 137B (Appendix D.2) and PaLM 62B (Appendix A.1) reveals several failure modes:

For small models in general (Section 3.2, qualitative observation):

1. Incoherent reasoning: Chains of thought contain statements that don't follow from prior statements or violate basic world knowledge. The reasoning sounds fluent but is logically disconnected (see Table 11 for examples).
2. Failure to produce parseable answers: Small models sometimes generate repetitive text, get stuck in loops, or never arrive at a final answer statement.
3. Inherently weaker arithmetic: Even when the reasoning logic is correct, small models make more arithmetic errors (consistent with Brown et al., 2020's finding that arithmetic ability scales with model size).

Specific error types in PaLM 62B (Appendix A.1, Figure 9):

The authors manually analyzed 45 errors made by PaLM 62B and categorized them into three types:

• Semantic understanding errors (20 of 45, 44%): The model misunderstands the problem's meaning—for example, confusing "4 feet long wire cut into 6-inch pieces" as requiring multiplication (4 × 6 = 24) rather than unit conversion and division (4 feet = 48 inches, 48 ÷ 6 = 8 pieces). These errors reflect a failure to correctly translate the natural language scenario into the appropriate mathematical structure.
• One-step-missing errors (18 of 45, 40%): The model's reasoning is largely correct but omits one crucial step—for example, calculating a 25% fee as 125% of the original rather than computing 25% of the original and then adding it. These are errors of omission where the chain is incomplete rather than wrong.
• Other errors (7 of 45, 16%): Hallucinations (making up facts not in the problem), repetitive outputs, and symbol mapping errors where the model uses wrong numbers in equations.

Errors fixed by scaling to PaLM 540B (Appendix A.1, Figure 9):

Of the 45 errors made by PaLM 62B, scaling to 540B fixed:

• 6 of 20 semantic understanding errors (30%)
• 12 of 18 one-step-missing errors (67%)
• 4 of 7 other errors (57%)

The largest improvement is in fixing one-step-missing errors—suggesting that larger models are better at maintaining a complete, coherent reasoning chain without skipping steps. This is consistent with the hypothesis that larger models have stronger abilities to track multiple constraints and intermediate states simultaneously.

Figure 10 provides concrete before-and-after examples. For instance, on the wire-cutting problem described above, PaLM 62B outputs "4 * 6 = 24 pieces" (multiplying feet by inches without conversion), while PaLM 540B correctly outputs "4 * 12 = 48 inches... 48 / 6 = 8 pieces" (performing the unit conversion and division).

Why This Matters for Understanding Chain-of-Thought Prompting

The emergent ability finding has a crucial practical implication: chain-of-thought prompting is not a universal method that improves any language model on any reasoning task. It requires the base model to have already acquired, through pretraining, the underlying capabilities needed to:

1. Parse complex natural language scenarios into semantic representations
2. Map semantic relationships to mathematical or logical operations
3. Maintain coherence across multiple reasoning steps
4. Perform arithmetic or symbolic manipulation accurately enough that errors don't compound

For models below the emergence threshold, the method is actually counterproductive—forcing the model to generate reasoning steps that are likely to be wrong introduces errors that compound and lead to worse performance than simply guessing the answer directly. This explains why standard prompting (which asks for the answer directly) sometimes outperforms chain-of-thought prompting for small models: an incorrect reasoning chain can actively mislead the model into a wrong answer that it might not have produced if it had simply pattern-matched from question to answer.

Chain-of-Thought Prompt Construction: Robustness Analysis

A critical practical question for any prompting method is sensitivity to the exact prompt. Prior work (Zhao et al., 2021) had shown that few-shot prompting performance can vary dramatically—from near-chance to near-state-of-the-art—based on seemingly minor factors like exemplar ordering. Section 3.4 (Figure 6, Tables 6–7) systematically evaluates the robustness of chain-of-thought prompting across several dimensions.

Robustness to Different Annotators

The main experiments use chains of thought written by "Annotator A" (a co-author). To test whether the method depends on a particular writing style, two additional co-authors ("Annotator B" and "Annotator C") independently wrote chains of thought for the same 8 exemplars. The annotators were given no specific instructions other than to "write the step-by-step reasoning process that led to the final answer." Their chains of thought are provided in Tables 29 and 30.

The resulting chains differ in linguistic style:

• Annotator A (original, Table 20): Uses multiple short sentences, restating given information before each operation. Example: "There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The answer is 6."
• Annotator B (Table 29): More concise, combining information and operation in fewer sentences. Example: "There are 21 trees now and there are 15 trees in the beginning, so the workers plant 21 - 15 = 6 trees. The answer is 6."
• Annotator C (Table 30): More formal and explicit about the logical structure. Example: "We start with 15 trees. Later we have 21 trees. The difference must be the number of trees they planted. So, they must have planted 21 - 15 = 6 trees. The answer is 6."

Results (Figure 6, "different annotator" bars; Table 6, "different annotator" rows):

On GSM8K with LaMDA 137B:

• Annotator A: 14.3% (±0.4%)
• Annotator B: 15.5% (±0.6%)
• Annotator C: 17.6% (±1.0%)
• Standard prompting baseline: 6.5% (±0.4%)

All three annotators substantially outperform the baseline. There is variance—Annotator C's prompts perform notably better—but the key takeaway is that all annotators produce large gains over standard prompting. The method does not depend on a specific writing style; any coherent step-by-step reasoning demonstration appears to work.

Robustness to Intentionally Concise Style

Annotator A also wrote a version that was intentionally more concise, following the style of solutions in Cobbe et al. (2021)'s GSM8K training set. For instance, whereas the original uses several short sentences, the concise version reads "5 * 4 = 20 new computers were added. So there are 9 + 20 = 29 new computers in the server room now" (single sentence combining the addition).

On GSM8K with LaMDA 137B (Table 6, "intentionally concise style"): 11.1% (±0.3%). This is lower than the full chain-of-thought (14.3%) but still substantially above the baseline (6.5%). The drop suggests that more verbose reasoning may be helpful for the hardest problems (GSM8K is the most challenging arithmetic benchmark), but the concise style still provides meaningful gains.

Robustness to Different Exemplars from a Different Source

To test whether the method requires carefully curated exemplars, the authors randomly sampled three sets of 8 exemplars from the GSM8K training set (which contains problems with human-written solutions by crowdworkers). These are an independent source of reasoning chains—written by non-experts without knowledge of the paper's goals.

Results (Figure 6, "exemplars from GSM8K" bars; Table 6):

• Set α: 12.6% (±0.6%)
• Set β: 12.7% (±0.5%)
• Set γ: 12.6% (±0.7%)
• Annotator A (original): 14.3% (±0.4%)
• Baseline: 6.5% (±0.4%)

All three GSM8K-sourced sets substantially outperform the baseline and perform comparably to the manually composed exemplars (within 1–2 percentage points). Importantly, these GSM8K exemplars were also tested on SVAMP, ASDiv, and MAWPS (Table 6) and yielded gains on all four datasets, even though the exemplars were drawn from a different distribution (GSM8K) than the test sets. This demonstrates that chain-of-thought prompting does not require exemplars from the same distribution as the test examples—the general pattern of step-by-step reasoning transfers across math word problem datasets.

Robustness to Exemplar Order

The LaMDA experiments report standard deviations across five random seeds with different exemplar orders (Table 6). The standard deviations are relatively small:

• GSM8K chain-of-thought: 14.3% ± 0.4% (coefficient of variation = 2.8%)
• SVAMP: 36.7% ± 0.4%
• ASDiv: 46.6% ± 0.7%
• MAWPS: 57.9% ± 1.5%

The exception is the coin flip task (Table 7, "Concat" and "Coin" columns), where standard deviations are larger—particularly for the Coin Flip with Annotator C (71.4% ± 11.1%). This is attributed to the classification-like nature of the task, consistent with Zhao et al. (2021)'s finding that classification tasks are more sensitive to exemplar order (many exemplars of the same category in a row biases the model).

Robustness to Number of Exemplars

The main experiments use 8 exemplars (constrained by the 1024-token context window). To test whether the gains depend on having exactly 8 exemplars, the authors varied the number from 1 to 8 (Figure 11, Appendix D.3). On all five tasks tested (GSM8K, MultiArith, Sports Understanding, Coin Flip, Last Letter Concatenation), chain-of-thought prompting outperforms standard prompting across all numbers of exemplars. The gap is smaller with only 1–2 exemplars but is already substantial at 4 exemplars, suggesting the method is not brittle to this parameter.

Why Robustness Matters

The robustness results address a central criticism of prompting-based methods: that they are unreliable and require extensive prompt engineering to work. The paper demonstrates that chain-of-thought prompting, while it does exhibit variance (as all prompting methods do), is robust enough to work across:

• Different human annotators with different writing styles
• Different sets of exemplars from different sources
• Different numbers of exemplars
• Different datasets (exemplars from one dataset work on others)

This robustness is important for practical adoption: practitioners do not need to find the "perfect" chain-of-thought prompt; any reasonable step-by-step reasoning demonstration for the task domain is likely to help, provided the model is large enough.

Task-Specific Adaptations

While the core method (augment few-shot exemplars with chains of thought) is domain-general, the paper makes several task-specific adaptations in experimental setup and evaluation.

Arithmetic Reasoning: Multiple Benchmarks with a Single Prompt

For the five arithmetic reasoning benchmarks (GSM8K, SVAMP, ASDiv, AQuA, MAWPS), the key adaptation is minimal: the same 8-exemplar prompt (Table 20) is used for four of the five benchmarks. The only exception is AQuA, which is multiple choice and requires selecting among options (a)–(e), so a separate 4-exemplar prompt is used (Table 21).

This is noteworthy because the benchmarks have different characteristics:

• GSM8K (1,319 test problems): Linguistically diverse, multi-step (2–8 steps), requiring composition of multiple arithmetic operations.
• SVAMP (1,000 problems): Designed to test sensitivity to irrelevant information and varying question structures; problems are generally shorter than GSM8K.
• ASDiv (2,096 problems): Diverse problem types covering different arithmetic operations and linguistic patterns.
• AQuA (254 problems): Algebraic word problems with multiple choice answers; requires selecting the correct option.
• MAWPS (2,065 problems): Includes subsets of varying difficulty: SingleOp (one operation), SingleEq (one equation), AddSub (addition/subtraction), MultiArith (multi-step). This stratification (Table 3) enables analysis of where chain of thought helps most.

The finding that a single prompt works across GSM8K, SVAMP, ASDiv, and MAWPS (all free-response) suggests that the model learns a general "solve math word problems by decomposing into steps" pattern rather than memorizing specific problem templates from the exemplars. This is further supported by the cross-dataset exemplar experiment (using GSM8K exemplars for other datasets—Table 6).

MAWPS Stratification: When Does Chain of Thought Help and When Doesn't It?

The MAWPS benchmark is stratified into four subsets by problem complexity (Table 3), enabling a more nuanced analysis:

• SingleOp (562 problems, single operation): PaLM 540B already achieves 94.1% with standard prompting; chain-of-thought also achieves 94.1%—no gain because there's no headroom.
• SingleEq (508 problems, single equation): Standard = 86.5%, chain-of-thought = 92.3%—a moderate gain.
• AddSub (395 problems, addition/subtraction): Standard = 93.9%, chain-of-thought = 91.9%—a small decrease (within noise).
• MultiArith (600 problems, multi-step): Standard = 42.2%, chain-of-thought = 94.7%—a massive gain of 52.5 percentage points.

This stratification reveals a key boundary condition: chain-of-thought prompting helps most when the problem requires multi-step reasoning and the baseline performance is low. On simple problems that the model already solves well, the method provides minimal additional benefit (and can sometimes hurt slightly by introducing unnecessary complexity). This is consistent with the paper's positioning: chain of thought is a method for enabling complex reasoning, not a universal performance booster.

Commonsense Reasoning: Domain-Specific Knowledge Activation

For commonsense reasoning tasks, the chains of thought serve a different function than in arithmetic. In arithmetic, the chain primarily decomposes a procedural computation into steps. In commonsense reasoning, the chain makes implicit world knowledge explicit and chains multiple facts together.

For CSQA (Table 24), each exemplar demonstrates a pattern of (1) identifying the constraint the answer must satisfy, (2) evaluating each option against that constraint, (3) selecting the option that matches. For example, "The answer must be an item that can absorb ink. Of the above choices, only blotters are used to absorb ink." This teaches the model to not just pattern-match but to reason about answer choices using world knowledge.

For StrategyQA (Table 25), the chains demonstrate multi-hop fact chaining: retrieving one fact, then using it to infer another, then drawing a conclusion. For example, "Hamsters are prey animals. Prey are food for predators." → "Thus, hamsters provide food for some animals." This requires the model to (1) know that hamsters are prey, (2) know that predators eat prey, (3) compose these facts logically. The chain-of-thought exemplars teach the model to make these connections explicit rather than jumping directly to the answer.

For Date Understanding (Table 26), the chains demonstrate temporal arithmetic: computing dates by adding/subtracting days, handling month boundaries, and converting between formats. For example, "One day after 06/01/1943 is 06/02/1943" demonstrates date increment; "10 days before today is 05/23/1943" demonstrates subtraction across month boundaries.

For Sports Understanding (Table 27), the chains demonstrate cross-domain knowledge integration: recognizing whether a named entity (athlete) belongs to a sport, and whether the described action is part of that sport. For example, "Joao Moutinho is a soccer player. The NFC championship is part of American football, not soccer. So the answer is no."

For SayCan (Table 28), the chains demonstrate task decomposition for robotics: interpreting a natural language instruction, selecting appropriate objects from an available set, and sequencing actions. The exemplars show the model reasoning about what objects satisfy a user's request (e.g., "something to eat that isn't a fruit" → "energy bar") and then sequencing find(), pick(), and put() actions.

The key adaptation across all commonsense tasks is that the chains of thought externalize knowledge that the model has acquired during pretraining. The exemplars teach the model to use its stored knowledge in a structured reasoning process rather than relying on surface-level pattern matching. This is why the method shows gains on commonsense tasks even though they don't involve procedural computation in the same way arithmetic does.

Symbolic Reasoning: Algorithmic Pattern Replication

For the symbolic reasoning tasks (last letter concatenation and coin flip), the chains of thought are essentially algorithms expressed in natural language. The exemplars (Tables 22 and 23) demonstrate a fixed procedure:

Last letter concatenation (2-word names):

1. Identify the last letter of the first word
2. Identify the last letter of the second word
3. Concatenate them in order
4. State the concatenated result as the answer

Coin flip (varying numbers of flips):

1. Count the number of people who flipped the coin
2. Determine if the count is even or odd
3. Apply the rule: even → state unchanged, odd → state inverted
4. State the result

These are "toy tasks" in the sense that the exemplars provide a complete algorithmic solution template. The model's job is to instantiate this template with new symbols at test time—replacing the specific names in last-letter concatenation, or counting a different set of flippers in coin flip. The fact that small models still fail at these tasks (Table 5: LaMDA 422M achieves only 1.6% on 2-word last letter concatenation with chain-of-thought) is striking because it shows that even pure symbol manipulation following a demonstrated template requires a certain model scale.

The out-of-domain (OOD) evaluation on symbolic tasks tests a stronger capability: length generalization. The exemplars for last letter concatenation only show 2-word names, but the model is tested on names with 3 and 4 words. The exemplars for coin flip show 0, 1, or 2 flips, but the model is tested on scenarios with 3 and 4 flips. Success on these OOD tests requires the model to extend the demonstrated algorithm to longer sequences—applying the same step iteratively more times than seen in training.

The results (Table 5, Figure 8) show that PaLM 540B achieves strong OOD performance:

• Last letter concatenation, 3 words: 94.8% (vs. 0.2% for standard prompting)
• Last letter concatenation, 4 words: 63.0% (vs. 0.0% for standard prompting)
• Coin flip, 3 flips: 98.6%
• Coin flip, 4 flips: 90.2%

Performance degrades as the test sequence gets longer (63.0% for 4 words vs. 99.4% for 2 words), but remains far above standard prompting (which is near zero for OOD lengths). This length generalization is a meaningful form of "reasoning": the model has learned, from the exemplars, a procedural pattern that it can apply iteratively, and it maintains coherence even as the sequence length grows beyond what was demonstrated.

Design Choice Summary: What Makes This Method Work?

Stepping back from the details, the paper's technical approach can be understood as a set of interlocking design choices, each justified by experimental evidence:

1. Triple format (⟨input, chain of thought, output⟩) over pair format (⟨input, output⟩): The chain of thought provides an intermediate computation budget that the model uses for step-by-step decomposition. The ablation experiments prove this is not merely about extra tokens (dots don't work) or knowledge activation (reasoning after answer doesn't work).

2. Natural language over formal language: The chains of thought are expressed in natural language rather than mathematical notation, logical formalisms, or code. This is what makes the method domain-general—the same approach works for math, commonsense, and symbolic reasoning. Equation-only prompting (the "equation only" ablation) shows that natural language is essential for bridging the semantic gap on hard problems.

3. Off-the-shelf models over fine-tuned models: The method uses frozen pretrained models with no gradient updates. This preserves the generality of the base model (one checkpoint serves all tasks) and eliminates the need for task-specific training data. The comparison to fine-tuned baselines (Figure 2: PaLM 540B with chain-of-thought prompting outperforms fine-tuned GPT-3 175B with a verifier on GSM8K) shows that prompting can match or exceed fine-tuning when the base model is large enough.

4. Consistent answer format over varied formats: All exemplars end with "The answer is X" or "So the answer is X," enabling reliable answer extraction via simple string matching. This is a practical consideration that makes the method reproducible and automatable.

5. Fixed exemplar set over per-task optimization: The same 8 exemplars are used across multiple arithmetic benchmarks, and the method transfers across annotators and exemplar sources. This demonstrates that chain-of-thought prompting is not a brittle prompt-engineering trick but a robust method that works as long as the exemplars demonstrate the general pattern of step-by-step reasoning.

4. Key Insights and Innovations

Innovation 1: Reframing Reasoning from a Training Problem to an Elicitation Problem

The paper's most fundamental conceptual move is not the method itself—providing intermediate steps—but the reframing of the reasoning challenge in language models. Prior to this work, the dominant paradigm for enabling reasoning in neural models was training: you either trained a model from scratch on rationale-augmented data (Ling et al., 2017) or fine-tuned a pretrained model on a large dataset of human-written reasoning chains (Cobbe et al., 2021). The implicit assumption was that reasoning capability must be instilled through supervised learning on reasoning examples—the model doesn't know how to reason until you teach it, and teaching requires many examples with gradient updates.

Chain-of-thought prompting overturns this assumption. The paper demonstrates that the reasoning capability is already present in sufficiently large language models, latent in their pretrained weights, and can be elicited through prompting alone—eight hand-written exemplars with no gradient updates. This is a fundamental shift from "we must train models to reason" to "we must discover how to prompt models to express the reasoning they already possess." The distinction is not merely practical (saving annotation cost) but conceptual: it recasts the reasoning problem as one of interface design rather than capability acquisition.

The evidence for this reframing is in what the paper does not do. No model is fine-tuned. No architectural modification is made. The same model checkpoint that achieves 17.9% on GSM8K with standard prompting achieves 56.9% with chain-of-thought prompting (Table 2)—a 3.2× improvement from changing only the text fed into the model. The capability was always there; the standard prompting interface simply failed to access it. This is reinforced by the emergent ability finding (Figure 4): the fact that the capability "turns on" at around 100B parameters—rather than improving gradually—suggests that larger models are not just better at reasoning but have crossed a qualitative threshold where the reasoning procedure becomes accessible through in-context imitation.

This reframing connects to a broader intellectual shift in the field circa 2022: the recognition that large language models acquire capabilities during pretraining that are not visible under standard evaluation protocols, and that prompt engineering can reveal them. The paper is part of a constellation of work (including instruction tuning by Wei et al., 2022a, and the emergent abilities framework by Wei et al., 2022b) that collectively challenged the assumption that what you see with standard prompting is what the model can do. The paper's specific contribution to this shift is demonstrating that multi-step reasoning—long considered a hard capability requiring specialized architectures or training—falls into this category of latent, elicitable abilities.

Significance beyond performance: This reframing changed the research agenda. If reasoning can be elicited through prompting, then progress on reasoning shifts from collecting ever-larger reasoning datasets to understanding how to structure prompts that guide models through reasoning procedures. This opened the door to an entire subsequent literature on prompt engineering for reasoning (including tree-of-thoughts, self-consistency, and program-aided language models) that builds directly on the elicitation paradigm established here.

Innovation 2: Establishing the Causal Role of Intermediate Computation Through Ablations

A naïve interpretation of chain-of-thought prompting's success would be that the model benefits from seeing more text—more context, more examples, or a helpful explanation alongside the answer. The paper preempts and refutes this interpretation through a set of ablation experiments (Section 3.3, Figure 5) that are among the most conceptually important results in the paper, even though they are not the headline performance numbers.

The three ablations—equation-only, variable-compute-only (dots), and reasoning-after-answer—collectively establish a specific causal mechanism: the model uses the natural language reasoning chain to sequentially decompose the problem, with each step's output causally contributing to the final answer. The evidence for causality (rather than correlation or knowledge activation) comes from the reasoning-after-answer ablation: when the reasoning chain appears after the answer in the exemplars, performance drops to baseline, showing that the model does not simply learn to produce explanations alongside answers—it must produce the reasoning before the answer, suggesting the reasoning steps causally determine the answer rather than merely accompanying it.

This ablation-driven argument is methodologically significant because the paper defines what chain-of-thought prompting is not. It is not merely spending more compute (dots don't work), not merely producing equations (equations alone don't help on hard problems), and not merely activating knowledge (reasoning after answer doesn't help). By systematically ruling out plausible alternative mechanisms, the paper strengthens the claim that the specific format—sequential, natural language, step-by-step reasoning that precedes the answer—is the active ingredient.

The "variable compute only" ablation deserves particular attention because it rules out a subtle hypothesis that would otherwise be difficult to dismiss. Large language models perform a fixed amount of computation per token generated. By generating a chain of thought, the model allocates more FLOPs to the problem. One might hypothesize that this additional computation, independent of what tokens are generated, allows the model to "think longer" and arrive at better answers—essentially, chain of thought works because it's a scratchpad for extra computation. The dots experiment rules this out: generating 12 dots (matching the character count of an equation) provides no benefit, meaning the semantic content of the intermediate tokens is necessary. This is a clean experimental design that isolates a specific mechanism.

Comparison to prior work: Prior rationale-augmented methods (Ling et al., 2017; Cobbe et al., 2021) demonstrated that providing rationales improves performance, but they did not isolate why—they trained models on rationale data and observed improvements, leaving open whether the rationales served as additional training signal, as knowledge activation, or as a template for sequential computation. The paper's ablation design provides causal evidence that distinguishes these hypotheses. This is an incremental but important methodological contribution: it establishes a standard for what counts as evidence that a model is actually "reasoning" through intermediate steps versus merely benefiting from additional context.

Innovation 3: The Emergent Ability Framing—Chain of Thought as a Scale-Dependent Phenomenon

The finding that chain-of-thought prompting only yields gains above approximately 100B parameters (Figure 4, Table 2, Table 4, Table 5) is not just an empirical observation—it is a diagnostic concept with implications for how we understand language model capabilities. The paper shows that for models below the emergence threshold, chain-of-thought prompting actually degrades performance: small models produce fluent but logically incoherent reasoning chains that lead to worse answers than simply guessing. This non-monotonic scaling behavior—where a method that helps large models hurts small ones—is characteristic of emergent abilities and challenges the assumption that scaling curves are smooth and predictable.

The significance of this finding lies in what it reveals about the nature of the capability. If chain-of-thought reasoning improved gradually with scale, one could attribute it to models becoming incrementally better at arithmetic, semantic parsing, or coherence—quantitative improvements in existing abilities. The sharp transition—from harmful to helpful around 100B parameters—suggests something qualitatively different: the capability to generate and leverage coherent multi-step reasoning is not a continuous function of scale but a phase change. Small models can generate text that looks like reasoning (it's fluent and follows the pattern of the exemplars), but they cannot maintain the logical consistency across steps needed for the reasoning to actually improve answer accuracy. Large models cross a threshold where the generated reasoning becomes reliable enough to improve rather than degrade performance.

This connects to the paper's broader framing through Wei et al. (2022b): chain-of-thought reasoning is an emergent ability that cannot be predicted by extrapolating from small-model behavior. The practical implication is that evaluating prompting methods on small models (which is cheaper and more common in academic research) can be actively misleading—a method that fails at 6B parameters might succeed dramatically at 175B. The paper provides a concrete, well-documented example of this phenomenon across three model families (LaMDA, GPT-3, PaLM) and multiple task domains, establishing emergent reasoning as a reproducible empirical phenomenon rather than a quirk of a specific model.

The qualitative error analysis (Appendix A.1, Figure 9) adds depth to this finding by categorizing what kinds of errors scaling fixes. The fact that scaling PaLM from 62B to 540B fixes 67% of "one-step-missing" errors but only 30% of "semantic understanding" errors suggests that maintaining coherence across multiple reasoning steps is more scale-dependent than basic semantic parsing. This provides a more granular picture of what "emergence" means: it's not that small models can't do any of the sub-tasks required for reasoning, but that they fail to compose them reliably across multiple steps.

Comparison to prior work: Prior to this paper, the scaling literature (Kaplan et al., 2020; Brown et al., 2020) characterized scaling as smooth and predictable—loss improves as a power law, and downstream task performance generally improves with scale. The observation that some prompting methods exhibit non-monotonic, emergent scaling behavior was novel and challenged the smooth-scaling assumption. This paper, together with contemporaneous work on emergent abilities, established that the interface between model and task (prompting method) can determine whether a capability appears gradually or abruptly, and that standard prompting provides only a lower bound on model capabilities.

Innovation 4: Establishing In-Context Learning as a Mechanism for Procedure Acquisition (Not Just Pattern Matching)

A subtle but important conceptual contribution of the paper is expanding our understanding of what in-context learning can accomplish. Prior to this work, few-shot prompting was primarily understood as a form of pattern matching: the model sees ⟨input, output⟩ pairs and learns to map new inputs to outputs by analogy, recognizing surface-level patterns in the mapping. Brown et al. (2020) had demonstrated this for tasks like translation, question answering, and classification, where the mapping from input to output is relatively direct.

Chain-of-thought prompting demonstrates that in-context learning can also serve as a mechanism for procedure acquisition: the model learns, from a few demonstrations, not just what answer to produce but how to produce it—a step-by-step algorithmic procedure that it can then apply to novel instances, including instances that require more steps than any demonstration (the OOD length generalization results: Table 5, Figure 8).

The symbolic reasoning experiments are the cleanest demonstration of this. In the last letter concatenation task, the model sees exemplars only for 2-word names—it never sees a 3-word or 4-word name during prompting. Yet PaLM 540B achieves 94.8% on 3-word names and 63.0% on 4-word names (Table 5). This is not pattern matching—the model has never seen an example of concatenating three last letters. It has learned, from the 2-word exemplars, a procedure: (1) identify each word, (2) extract the last letter of each, (3) concatenate in order. It then applies this procedure iteratively to longer sequences. This is a form of algorithmic reasoning acquired entirely through in-context learning, with no architectural support for iteration or recursion.

Similarly, in the coin flip task, the model learns a procedure from exemplars with 0, 1, or 2 flips and applies it to test cases with 3 or 4 flips. The procedure—count the flips, check parity, apply the state-transition rule—generalizes beyond the demonstrated sequence length.

Why this matters conceptually: It suggests that large language models, through pretraining on code and natural language, have acquired implicit procedural knowledge—they can recognize and replicate algorithmic patterns when they are demonstrated in natural language. The few-shot exemplars serve not just as examples of input–output mappings but as program demonstrations that the model can compile into an executable procedure. This blurs the line between "learning from examples" and "programming by demonstration," and it anticipates subsequent work on program-aided language models and code generation for reasoning.

The length generalization result is particularly significant because it addresses a classic criticism of neural networks: that they can only interpolate within their training distribution and fail to generalize systematically to longer sequences (the "length generalization" challenge in sequence-to-sequence models). Chain-of-thought prompting enables a form of systematic generalization—applying a learned procedure to longer inputs—without any architectural modifications or training. This is not perfect (performance degrades from 99.4% at length 2 to 63.0% at length 4 for last letter concatenation), but it is dramatically better than standard prompting (near zero at OOD lengths) and demonstrates that the model has acquired something more abstract than surface-level pattern matching.

Comparison to prior work: Prior work on neural program execution (Zaremba and Sutskever, 2014; Nye et al., 2021) required training models specifically on program execution traces—large datasets of step-by-step computation. Chain-of-thought prompting achieves a form of procedural generalization with only a handful of demonstrations and no specialized training, suggesting that the procedural knowledge was already latent in the pretrained model. This is a qualitative advance in what we understand in-context learning to be capable of.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on five arithmetic reasoning benchmarks (GSM8K: 1,319 test problems, math word problems from Cobbe et al., 2021; SVAMP: 1,000 problems with varying structures from Patel et al., 2021; ASDiv: 2,096 diverse math word problems from Miao et al., 2020; AQuA: 254 algebraic word problems from Ling et al., 2017; MAWPS: 2,065 problems, stratified into SingleOp, SingleEq, AddSub, and MultiArith subsets from Koncel-Kedziorski et al., 2016), five commonsense reasoning benchmarks (CSQA from Talmor et al., 2019; StrategyQA from Geva et al., 2021; Date Understanding and Sports Understanding from BIG-bench collaboration, 2021; SayCan from Ahn et al., 2022), and two synthetic symbolic reasoning tasks (last letter concatenation and coin flip, generated by the authors). Most benchmarks have only evaluation splits; for CSQA and StrategyQA, exemplars were randomly sampled from training sets, while for BIG-bench tasks, the first 10 evaluation examples served as few-shot exemplars with results reported on the remainder. Example problems for each benchmark are provided in Appendix Table 12 (arithmetic) and Figure 3 (all domains).

• Base model(s). The paper evaluates five large language model families spanning multiple scales: LaMDA (422M, 2B, 8B, 68B, 137B parameters; Thoppilan et al., 2022), GPT-3 (text-ada-001 ~350M, text-babbage-001 ~1.3B, text-curie-001 ~6.7B, text-davinci-002 ~175B; Brown et al., 2020, with the davinci variant corresponding to InstructGPT from Ouyang et al., 2022), PaLM (8B, 62B, 540B; Anil et al., 2023, cited as a contemporaneous work), UL2 20B (Tay et al., 2022), and Codex (code-davinci-002; Chen et al., 2021). The multi-family, multi-scale design is intentional: it enables testing whether chain-of-thought prompting's effects generalize across model architectures and training procedures, and whether the emergent ability pattern (gains only at large scale) replicates across model families. The models were chosen to be "representative of the capabilities of many contemporary LLMs" (Section 3.1).

• Metrics. The primary metric is accuracy (solve rate)—the fraction of test examples for which the model's final answer matches the ground truth. For arithmetic benchmarks, the answer is extracted by parsing the generated text for the pattern "The answer is X" and comparing to the ground truth using the grading functions released by the respective dataset creators. For multiple-choice tasks (AQuA, CSQA), the answer is the selected option letter. For yes/no tasks (StrategyQA, Sports Understanding, coin flip), the answer is "yes" or "no." For date tasks, the answer is a date string in MM/DD/YYYY format. For SayCan, accuracy is measured as successful execution of the planned action sequence. For LaMDA experiments, standard deviations are reported across five random seeds with different exemplar orders (e.g., GSM8K chain-of-thought: 14.3% ± 0.4%); for other models, results are reported for a single exemplar order to save compute, with the justification that "LaMDA experiments did not show large variance among different seeds" (Section 3.1).

• Baselines. The primary baseline is standard few-shot prompting (Brown et al., 2020), where the model is given in-context exemplars formatted as ⟨input question, output answer⟩ pairs and asked to directly produce the answer for a test question. For arithmetic benchmarks, an external calculator baseline is applied post-hoc to both standard and chain-of-thought prompting (Table 1, "ext. calc" rows). For comparison to prior work, the paper includes prior supervised best results: finetuned GPT-3 175B with a verifier achieving 55% on GSM8K (Cobbe et al., 2021), prior best on SVAMP of 57.4% (Pi et al., 2022), prior best on MAWPS of 88.4% (Lan et al., 2021), prior best on ASDiv of 75.3% (Lan et al., 2021), and prior best on AQuA of 37.9% (Piękos et al., 2021). For StrategyQA, the prior best is 69.4% (single-model, from the StrategyQA leaderboard as of May 5, 2022). For Sports Understanding, human performance of 84% serves as a baseline (from BIG-bench collaboration, 2021). For SayCan, the baseline is the original SayCan system (Ahn et al., 2022).

• Generation budget / compute accounting. The paper does not use a formal compute budget (e.g., FLOPs or number of generations) as its primary unit of comparison, since all methods involve a single forward pass per test example. The "budget" in this work is implicitly the number of few-shot exemplars, which is fixed at 8 for most experiments (constrained by the 1024-token context window of the models used). Robustness experiments vary the number of exemplars from 1 to 8 (Figure 11, Appendix D.3). The external calculator (Section 3.2) is a post-hoc correction with negligible computational cost. For LaMDA, the cost of five random seeds is reported but not treated as a compute budget tradeoff—the seeds are for variance estimation, not for ensembling.

• Cross-validation / statistical protocol. There is no cross-validation or train/validation/test split optimization, since the method involves no training or hyperparameter tuning. The same 8 manually composed exemplars are used across all four free-response arithmetic benchmarks and across all model families, with no per-dataset or per-model optimization. For the out-of-domain (OOD) symbolic reasoning experiments, the test sets are constructed with longer sequence lengths than the exemplars (3–4 word names vs. 2-word exemplars; 3–4 flips vs. 0–2 flip exemplars), and the exemplars are not modified. Statistical significance is assessed via standard deviation across five random exemplar orders for LaMDA 137B (Tables 6–7), with the finding that variance is generally small (coefficient of variation <5% for most arithmetic tasks, though larger for classification-style tasks like coin flip at 71.4% ± 11.1% for Annotator C). For GPT-3 and PaLM, single-order results are reported without error bars.

Main Quantitative Results

Arithmetic Reasoning: Chain-of-Thought Prompting More Than Doubles Performance on the Hardest Task

The headline arithmetic result appears in Figure 2 and Table 2: on the GSM8K benchmark—the most challenging arithmetic dataset with the lowest baseline performance—chain-of-thought prompting with PaLM 540B achieves 56.9% accuracy, compared to 17.9% for standard prompting, a gain of 39.0 percentage points (more than tripling performance). This single result establishes the practical significance of the method: PaLM 540B with 8 hand-written chain-of-thought exemplars and no fine-tuning surpasses the prior state of the art of 55% achieved by a finetuned GPT-3 175B with a trained verifier (Cobbe et al., 2021). The full set of arithmetic results across all five benchmarks and all model families appears in Table 1 (headline results with largest models and external calculator) and Table 2 (full scaling results across all model sizes).

Scaling behavior (Figure 4, Table 2): The relationship between model scale and chain-of-thought effectiveness is non-monotonic and exhibits a phase transition. On GSM8K:

• LaMDA: 422M (2.6% → 0.4%, decrease), 2B (3.6% → 1.9%, decrease), 8B (3.2% → 1.6%, decrease), 68B (5.7% → 8.2%, first gain), 137B (6.5% → 14.3%, substantial gain of +7.8 points).
• GPT-3: 350M (2.2% → 0.5%, decrease), 1.3B (2.4% → 0.5%, decrease), 6.7B (4.0% → 2.4%, decrease), 175B (15.6% → 46.9%, massive gain of +31.3 points).
• PaLM: 8B (4.9% → 4.1%, slight decrease), 62B (9.6% → 29.9%, gain of +20.3 points), 540B (17.9% → 56.9%, gain of +39.0 points).

The same pattern holds across SVAMP (e.g., PaLM 540B: 69.4% → 79.0%, +9.6 points), ASDiv (PaLM 62B: 58.7% → 61.9%, +3.2; PaLM 540B: 72.1% → 73.9%, +1.8—a smaller gain because standard prompting already achieves strong performance), and MAWPS (PaLM 540B: 79.2% → 93.3%, +14.2 points). On AQuA, PaLM 540B improves from 25.2% to 35.8% (+10.6 points), though this remains below the prior best of 37.9%.

Codex results (Table 1): Codex (code-davinci-002) exhibits the strongest performance overall, with chain-of-thought prompting achieving 63.1% on GSM8K (vs. 19.7% standard), 76.4% on SVAMP (vs. 69.9%), 80.4% on ASDiv (vs. 74.0%), 45.3% on AQuA (vs. 29.5%), and 92.6% on MAWPS (vs. 78.7%). The fact that a code-focused model benefits from natural language reasoning chains is noteworthy—it suggests that the procedural reasoning patterns acquired during code pretraining transfer to natural language math problems.

External calculator impact (Table 1, "ext. calc" rows): Adding a post-hoc external calculator consistently improves chain-of-thought prompting across all arithmetic benchmarks and model families, though the magnitude varies. On GSM8K: LaMDA 137B improves from 14.3% to 17.8% (+3.5 points), GPT-3 175B from 46.9% to 49.6% (+2.7 points), PaLM 540B from 56.9% to 58.6% (+1.7 points). On SVAMP, the calculator provides larger gains for smaller models (UL2 20B: 12.5% → 28.3%, +15.8 points; LaMDA 137B: 37.5% → 42.1%, +4.6 points), suggesting that smaller models benefit more from arithmetic correction because they make more arithmetic errors in otherwise correct reasoning chains.

Difficulty stratification via MAWPS (Table 3): The MAWPS benchmark is stratified into subsets by problem complexity, revealing that chain-of-thought prompting's benefit is concentrated on multi-step problems:

• SingleOp (1 operation): PaLM 540B achieves 94.1% with both standard and chain-of-thought prompting (no gain, ceiling effect).
• SingleEq (1 equation): 86.5% → 92.3% (+5.8 points, small gain).
• AddSub (addition/subtraction only): 93.9% → 91.9% (−2.0 points, small decrease within noise).
• MultiArith (multi-step): 42.2% → 94.7% (+52.5 points, the largest gain in the entire paper).

This stratification demonstrates that chain-of-thought prompting is not a universal performance booster—it specifically helps when problems require composing multiple reasoning steps, and it provides minimal benefit (or slight harm) on simple problems that the model already solves well.

Comparison to prior supervised best (Figure 4, Table 2 caption): PaLM 540B with chain-of-thought prompting achieves new state-of-the-art on GSM8K (56.9% vs. 55% prior best), SVAMP (79.0% vs. 57.4% prior best—though standard prompting already passed the prior best at 69.4%), and MAWPS (93.3% vs. 88.4% prior best). On ASDiv, PaLM 540B with chain-of-thought achieves 73.9% vs. 75.3% prior best (within 1.4 points). On AQuA, PaLM 540B achieves 35.8% vs. 37.9% prior best (within 2.1 points). Codex with chain-of-thought surpasses prior best on all five arithmetic benchmarks.

Commonsense Reasoning: Chain-of-Thought Prompting Enables Multi-Hop Inference Across Diverse Domains

The commonsense reasoning results appear in Figure 7 and Table 4. The headline finding is that chain-of-thought prompting improves performance across all five commonsense benchmarks for PaLM 540B, with the largest gains on tasks requiring multi-hop reasoning or cross-domain knowledge integration.

StrategyQA (Figure 7, Table 4): This is the standout commonsense result. PaLM 540B with standard prompting achieves 68.6%; chain-of-thought prompting raises this to 75.6% (+7.0 points), surpassing the prior single-model state of the art of 69.4% (from the StrategyQA leaderboard). StrategyQA questions require inferring a multi-hop strategy—for example, "Did Aristotle use a laptop?" requires knowing that Aristotle lived in ancient Greece and laptops are modern inventions. The chain-of-thought exemplars (Table 25) demonstrate connecting multiple facts: "Hamsters are prey animals. Prey are food for predators. Thus, hamsters provide food for some animals. So the answer is yes." The model learns to make its implicit factual knowledge explicit and chain it logically.

Sports Understanding (Figure 7, Table 4): PaLM 540B achieves 95.4% with chain-of-thought prompting, compared to 80.5% with standard prompting (+14.9 points). This exceeds the human baseline of 84% from BIG-bench, meaning the prompted model outperforms "an unaided sports enthusiast" (Section 4). Sports Understanding requires recognizing whether a named athlete belongs to a particular sport and whether a described action is part of that sport—for example, "Joao Moutinho caught the screen pass in the NFC championship" is implausible because Moutinho is a soccer player and the NFC championship is American football. The chain-of-thought exemplars (Table 27) explicitly demonstrate this cross-domain reasoning: identify the athlete's sport, identify the sport associated with the action, compare them.

Date Understanding (Figure 7, Table 4): PaLM 540B improves from 49.0% to 65.3% (+16.3 points). This task requires temporal arithmetic—computing dates by adding or subtracting days, handling month boundaries and leap years. The chain-of-thought exemplars (Table 26) demonstrate step-by-step date manipulation: "One day after 06/01/1943 is 06/02/1943, so today is 06/02/1943. 10 days before today is 05/23/1943." The model learns to perform these temporal calculations explicitly rather than attempting to intuit the date directly.

CSQA (Figure 7, Table 4): The gains are minimal—PaLM 540B improves from 78.1% to 79.9% (+1.8 points). The paper notes this explicitly: "gain was minimal on CSQA" (Section 4). CSQA questions test general commonsense knowledge about the world (e.g., "What do people use to absorb extra ink from a fountain pen?"). The small gain suggests that for questions requiring primarily factual retrieval rather than multi-step inference, chain-of-thought prompting adds limited value—the model can already retrieve the answer directly.

SayCan (Figure 7, Table 4): PaLM 540B improves from 80.8% to 91.7% (+10.9 points) in mapping natural language instructions to robot action sequences. The chain-of-thought exemplars (Table 28) demonstrate a two-part reasoning process: (1) interpret the user's request and select appropriate objects, (2) sequence the necessary actions. For example, for "bring me something to eat that isn't a fruit," the model explains "I will bring an energy bar to them" and then generates the plan: find(energy bar), pick(energy bar), find(user), put(energy bar). The improvement suggests that the model learns to ground natural language requests in the available objects and actions through the demonstrated reasoning pattern.

Scaling behavior for commonsense (Figure 7, Table 4): The emergent ability pattern is less pronounced for commonsense than for arithmetic—gains appear at somewhat smaller scales and are more gradual. For CSQA, LaMDA 68B already shows improvement (37.0% → 44.1%), and gains increase with scale. For StrategyQA, LaMDA 68B shows 59.6% → 62.2% (+2.6 points), LaMDA 137B shows 62.4% → 65.4% (+3.0 points), and PaLM 540B shows 68.6% → 77.8% (+9.2 points—note Figure 7 reports 75.6% for PaLM 540B, while Table 4 reports 77.8% for PaLM 540B on StrategyQA; this discrepancy likely reflects different evaluation configurations). This suggests that the factual knowledge and basic inference capabilities needed for commonsense reasoning are accessible at smaller model scales than the procedural composition capabilities needed for multi-step arithmetic.

GPT-3 on commonsense (Table 4): GPT-3 175B (text-davinci-002) shows a notable pattern: on CSQA, chain-of-thought prompting decreases performance from 79.5% to 73.5% (−6.0 points), and on StrategyQA, performance is essentially flat (65.9% → 65.4%, −0.5 points). This is the only model family where chain-of-thought prompting does not help on these tasks. The paper does not offer an explanation for this discrepancy, but it suggests that the effectiveness of chain-of-thought prompting may depend on model-specific factors such as pretraining data composition, instruction tuning procedure, or architecture.

Symbolic Reasoning: Chain-of-Thought Prompting Enables Length Generalization

The symbolic reasoning results appear in Figure 8 and Table 5. These experiments test a fundamentally different capability: can the model learn an algorithmic procedure from few-shot exemplars and apply it to inputs longer than any seen during prompting?

Last letter concatenation (Figure 8, Table 5): In the in-domain setting (2-word names, matching the exemplars), PaLM 540B achieves 99.4% with chain-of-thought prompting, compared to 7.6% with standard prompting (+91.8 points). However, this is a "toy task"—the model only needs to replicate the demonstrated procedure with new 2-word names. The more informative results are the OOD settings:

• 3-word names (OOD): PaLM 540B standard = 0.2%, chain-of-thought = 94.8% (+94.6 points)
• 4-word names (OOD): PaLM 540B standard = 0.0%, chain-of-thought = 63.0% (+63.0 points)

The model generalizes the learned procedure (extract last letter of each word, concatenate) to longer sequences never seen in the exemplars. Performance degrades as length increases (99.4% → 94.8% → 63.0%), but remains dramatically above the near-zero baseline, demonstrating that the model has acquired a procedure rather than just memorizing a pattern.

Coin flip (Figure 8, Table 5): In the in-domain setting (0–2 flips), PaLM 540B achieves 100.0% with chain-of-thought prompting (standard prompting already achieves 98.1% for 2 flips, so the headroom is small). For OOD settings:

• 3 flips (OOD): PaLM 540B standard = 49.3% (near chance), chain-of-thought = 98.6%
• 4 flips (OOD): PaLM 540B standard = 54.8%, chain-of-thought = 90.2%

Again, the model generalizes beyond the demonstrated sequence lengths, maintaining high accuracy even when the state must be tracked across 4 sequential flips.

Scaling behavior for symbolic reasoning (Table 5): The emergent ability pattern is stark. For last letter concatenation (2 words):

• LaMDA 422M: standard = 0.3%, chain-of-thought = 1.6% (minimal gain)
• LaMDA 2B: 2.3% → 6.0% (small gain)
• LaMDA 8B: 1.5% → 11.5% (gain appears)
• LaMDA 68B: 4.4% → 52.0% (large gain)
• LaMDA 137B: 5.8% → 77.5% (very large gain)
• PaLM 8B: 2.6% → 18.8%
• PaLM 62B: 6.8% → 85.0%
• PaLM 540B: 7.6% → 99.4%

For the OOD settings (3 and 4 words), small models achieve near-zero performance with both methods; only at 68B+ parameters do substantial OOD gains appear. For coin flip, the pattern is similar: LaMDA 68B achieves 83.2% (in-domain, 2 flips) and 69.1% (OOD, 3 flips), while LaMDA 137B achieves 99.6% (in-domain) and 91.0% (OOD, 3 flips). The ability to learn and generalize even simple algorithmic procedures in-context is scale-dependent, appearing only above approximately 50–100B parameters.

UL2 20B on symbolic reasoning (Table 5): UL2 20B performs notably poorly on last letter concatenation—even with chain-of-thought prompting, it achieves only 18.8% on 2 words, 0.2% on 3 words, and 0.0% on 4 words. For coin flip, it achieves 67.1% on 2 flips but only 52.2% on 3 flips and 50.4% on 4 flips (essentially chance). This is an interesting negative result: despite its 20B parameters, UL2 underperforms similarly-sized or even smaller models from other families (LaMDA 68B achieves 52.0% on 2-word concatenation). This suggests that model architecture and pretraining objective may matter as much as raw parameter count for emergent symbolic reasoning.

Ablation Studies and Robustness Checks

All ablation results for arithmetic reasoning are in Table 6 and Figure 5 (LaMDA 137B and PaLM 540B on GSM8K); ablation results for commonsense and symbolic reasoning are in Table 7. The three core ablations are discussed in detail in Section 3 but are formally ablation experiments, so their quantitative results are summarized here:

Equation-only prompting (Figure 5, Table 6): On GSM8K with LaMDA 137B, equation-only prompting achieves 5.4% (±0.2%) compared to 14.3% (±0.4%) for full chain-of-thought and 6.5% (±0.4%) for standard prompting—statistically indistinguishable from standard prompting. On PaLM 540B (Figure 5), the gap is even larger (visual estimate: approximately 20% for equation-only vs. 57% for chain-of-thought). However, on simpler datasets (Table 6): SVAMP equation-only = 35.1% vs. chain-of-thought = 36.7% (close, suggesting equations suffice for simpler problems), ASDiv = 45.9% vs. 46.6%, MAWPS = 50.1% vs. 57.9% (natural language helps more on the multi-step MAWPS subset).

Variable compute only (dots) (Figure 5, Table 6): On GSM8K with LaMDA 137B: 6.4% (±0.3%)—essentially identical to standard prompting (6.5%). On SVAMP: 28.0% (vs. 29.5% standard—actually slightly worse). On all other arithmetic datasets (Table 6) and commonsense/symbolic datasets (Table 7), dots perform at or slightly below the standard prompting baseline. This ablation is uniformly negative across all tasks and model scales.

Reasoning after answer (Figure 5, Table 6): On GSM8K with LaMDA 137B: 6.1% (±0.4%)—again identical to standard prompting (6.5%). On SVAMP: 30.7% (vs. 29.5% standard—marginally better but far below chain-of-thought at 36.7%). The pattern holds for commonsense and symbolic tasks (Table 7): for Sports Understanding with LaMDA 137B, reasoning after answer achieves 63.0% (vs. 59.5% standard, 85.8% chain-of-thought—a small gain but far below the main method); for coin flip, 50.2% (vs. 49.0% standard, 99.6% chain-of-thought—no gain). For last letter concatenation, it achieves 0.0% (catastrophic failure—the model never produces a parseable answer when reasoning is placed after the answer).

Robustness to annotator (Figure 6, Table 6): On GSM8K with LaMDA 137B, three annotators produce chain-of-thought accuracy of 14.3% (A, ±0.4%), 15.5% (B, ±0.6%), and 17.6% (C, ±1.0%)—all substantially above the 6.5% baseline. On MAWPS, the spread is 57.9% (A), 58.2% (B), 60.1% (C). On SVAMP: 36.7% (A), 35.2% (B), 37.5% (C). On ASDiv: 46.6% (A), 46.5% (B), 48.7% (C). For commonsense and symbolic tasks (Table 7), annotator variance is larger: on Sports Understanding, 85.8% (A), 75.4% (B), 81.1% (C); on coin flip, 99.6% (A), 77.5% (B), 71.4% (C). The coin flip task shows the largest sensitivity, with Annotator C achieving only 71.4% ± 11.1%—still above the 49.0% baseline but with high variance and notably lower than Annotator A's 99.6%.

Robustness to intentionally concise style (Table 6): On GSM8K with LaMDA 137B, the concise chain-of-thought achieves 11.1% (±0.3%)—lower than the full chain-of-thought (14.3%) but still substantially above standard (6.5%). On SVAMP, the concise style achieves 38.7%—slightly outperforming the full chain-of-thought (36.7%). On ASDiv, 48.0% vs. 46.6%. On MAWPS, 59.6% vs. 57.9%. The concise style either slightly underperforms or slightly outperforms the verbose style depending on the dataset, but consistently outperforms standard prompting by a wide margin.

Robustness to different exemplars from GSM8K training set (Figure 6, Table 6): Three randomly sampled sets of 8 exemplars from the GSM8K training set (α, β, γ) achieve 12.6% (±0.6%), 12.7% (±0.5%), and 12.6% (±0.7%) on GSM8K with LaMDA 137B—all clustering tightly and outperforming standard prompting (6.5%). Crucially, these GSM8K-sourced exemplars also improve performance on SVAMP (32.8%–35.6% across the three sets, vs. 29.5% standard), ASDiv (44.1%–46.9% vs. 40.1%), and MAWPS (53.9%–60.9% vs. 43.2%)—demonstrating cross-dataset generalization from a single exemplar source.

Robustness to number of exemplars (Figure 11, Appendix D.3): Across five tasks (GSM8K, MultiArith, Sports Understanding, Coin Flip, Last Letter Concatenation), chain-of-thought prompting with LaMDA 137B consistently outperforms standard prompting when using 1, 2, 4, 6, or 8 exemplars. The gap is smallest with only 1 exemplar and grows with more exemplars, but even a single chain-of-thought exemplar provides a notable gain on most tasks. The paper notes that further increasing exemplars beyond 8 did not improve standard prompting enough to catch up with chain-of-thought prompting (Section A.2).

Robustness to exemplar order (Table 6, Table 7): Standard deviations for LaMDA 137B across five random exemplar orders are reported in Tables 6–7. For arithmetic, standard deviations are small (GSM8K chain-of-thought: ±0.4%; SVAMP: ±0.6%; ASDiv: ±0.7%; MAWPS: ±1.5%). For commonsense, variance is larger: Date Understanding chain-of-thought: ±2.1%; Sports Understanding standard prompting: ±3.0%. For symbolic reasoning, coin flip chain-of-thought shows very small variance (±0.3%) with Annotator A but much larger with Annotator B (±7.9%) and Annotator C (±11.1%). The paper attributes this to the classification-like nature of the coin flip task, consistent with Zhao et al. (2021)'s finding that classification tasks are more sensitive to exemplar order.

External calculator as post-hoc correction (Table 1): The external calculator is applied to both standard and chain-of-thought prompting. For chain-of-thought prompting, it consistently improves accuracy across all model families and arithmetic benchmarks, though the magnitude varies: +3.5 points for LaMDA 137B on GSM8K, +2.7 for GPT-3 175B, +1.7 for PaLM 540B, +2.3 for Codex. The calculator's benefit is largest for smaller models (e.g., UL2 20B on MAWPS: 19.1% → 42.7%, +23.6 points) and on benchmarks requiring more arithmetic operations. This confirms the paper's qualitative observation (Appendix D.2) that many chain-of-thought errors are "calculator errors"—correct reasoning with incorrect arithmetic.

Error analysis of chain-of-thought correctness (Appendices D.1, D.2): This is not a standard ablation but provides critical evidence about whether the model is actually reasoning. For LaMDA 137B on GSM8K:

• Of 50 randomly sampled correct final answers (Appendix D.1), 49 had logically and mathematically correct chains of thought. One arrived at the correct answer through incorrect reasoning ("correct by chance"). Five had minor imperfections (underspecified statements, unrelated statements, imperfect equations, inverted semantics) while maintaining coherent logic. Examples of correct chains are in Table 8; examples of imperfect-but-functional chains are in Table 9.
• Of 50 randomly sampled incorrect final answers (Appendix D.2): 8% had calculator errors only (the chain would be correct with external arithmetic); 16% had symbol mapping errors (the chain was logically correct but used wrong numbers; Table 10 provides examples); 22% had one step missing (the chain would be correct with one additional reasoning step); and 54% had major errors in semantic understanding or coherence that would require substantial rewriting (Table 11 provides examples).

This analysis reveals that when the model gets the right answer, it almost always (49/50) does so through correct reasoning—suggesting chain-of-thought prompting genuinely elicits reasoning rather than lucky guessing. When the model gets the wrong answer, the errors span a spectrum from easily fixable (calculator errors, symbol mapping) to fundamental (semantic misunderstanding, incoherence).

Scaling fixes specific error types (Appendix A.1, Figure 9): Analysis of 45 errors made by PaLM 62B on GSM8K shows that scaling to 540B fixes 12 of 18 one-step-missing errors (67%), 6 of 20 semantic understanding errors (30%), and 4 of 7 other errors (57%). Figure 10 provides before-and-after examples for each error type. This is the paper's closest attempt to explain why scaling enables chain-of-thought reasoning: larger models are better at maintaining complete, coherent reasoning chains without skipping steps, and they show some improvement (though more limited) in correctly understanding problem semantics.

Critical Assessment

The paper makes three central claims, and the experimental evidence supports each to varying degrees. Rather than a blanket assessment, I examine each claim against the specific experiments conducted.

Claim: Chain-of-thought prompting significantly improves performance on arithmetic, commonsense, and symbolic reasoning tasks. This claim is empirically well-supported within the scope tested. The GSM8K result is the strongest single piece of evidence: PaLM 540B more than triples its accuracy (17.9% → 56.9%, Table 2), overtaking a fine-tuned baseline that required thousands of training examples. The result replicates across three model families (LaMDA, GPT-3, PaLM), two additional model families (UL2, Codex), and five arithmetic benchmarks (Table 1). The commonsense results show consistent albeit smaller gains (Table 4), with StrategyQA standing out as a new state of the art (75.6% vs. 69.4% prior best).

However, the scope of testing is narrower than the claim suggests. All experiments use a single prompt per task domain with 8 hand-written exemplars. The paper does not test whether chain-of-thought prompting works for tasks beyond the specific ones evaluated—for instance, it does not test logical reasoning (beyond the toy coin flip), reading comprehension, or multi-step text generation. The claim that the method is "broadly applicable" rests on the diversity of the tested tasks (math, commonsense, symbolic, robotics) rather than on breadth of testing within each domain. More critically, the paper does not test on tasks where chain-of-thought prompting might be harmful beyond the scale-dependent degradation in small models—for instance, tasks requiring rapid, intuitive judgments where step-by-step reasoning could introduce errors or over-analysis.

Moreover, the absolute performance levels on the hardest benchmarks remain modest. On GSM8K, even PaLM 540B with chain-of-thought prompting solves only 56.9% of problems—nearly half the test set remains unsolved. On AQuA, performance is 35.8%—well below what would be considered reliable for practical deployment. The improvements are large relative to baseline, but the absolute capability remains limited. The paper is transparent about this (the numbers are reported without embellishment), but readers should not overinterpret the gains as "solving" reasoning.

Claim: Chain-of-thought reasoning is an emergent ability of model scale. This claim is strongly supported by the scaling experiments across three model families and multiple tasks. The evidence is consistent and striking: for models below approximately 50–100B parameters, chain-of-thought prompting either degrades performance or provides minimal benefit; above that threshold, large and growing gains appear (Table 2, Table 4, Table 5). The pattern is non-monotonic—it is not simply that larger models benefit more, but that there is a qualitative threshold below which the method is actively harmful. This is the hallmark of emergence and is difficult to explain through smooth scaling alone.

However, there are important caveats. First, the paper does not isolate which properties of scale cause the emergence. Is it model depth, width, total parameters, training tokens, or some combination? The three model families (LaMDA, GPT-3, PaLM) differ in architecture and training, and yet the emergence threshold is roughly similar (50–100B parameters), but this is an observation, not a controlled experiment. The paper speculates in Appendix A.1 that "a variety of emergent abilities (semantic understanding, symbol mapping, staying on topic, arithmetic ability, faithfulness, etc.)" are involved, but does not isolate them.

Second, the paper does not test models between 8B and 62B for PaLM, or between 6.7B and 175B for GPT-3. The "phase transition" is observed across gaps of an order of magnitude in scale. There could be intermediate-sized models (e.g., 20B, 40B) that show partial emergence, and the paper's choice of model sizes (dictated by availability of pretrained checkpoints) may make the transition appear sharper than it is.

Third, the UL2 20B results complicate the narrative. UL2 20B on GSM8K achieves only 4.4% with chain-of-thought prompting (slightly above standard at 4.1%)—it does not show the large gains that similarly-sized models from other families might be expected to show (LaMDA 68B achieves 8.2%). On last letter concatenation, UL2 20B achieves 18.8% vs. PaLM 8B's 18.8%—both low, but PaLM 62B achieves 85.0%. This suggests that raw parameter count is not the sole determinant of emergence; model architecture and pretraining objective matter. The paper acknowledges architecture as a factor (Appendix A.1: "model scale is often conflated with other factors, such as amount of training compute") but does not systematically investigate it.

Claim: The method works because models learn to decompose problems—not because of extra computation, knowledge activation, or equation extraction. This claim is supported by the ablation experiments, but the ablations have limitations that the paper does not fully address.

The equation-only ablation (Figure 5) shows that natural language reasoning steps are essential for the hardest problems (GSM8K) but less so for simpler problems (SVAMP, ASDiv). This is a nuanced finding, not a universal one, and the paper's headline messaging ("chain of thought is necessary") is slightly stronger than what the ablation supports—on simpler arithmetic, equations alone nearly match chain-of-thought.

The dots ablation (variable compute only) is the cleanest: generating meaningless tokens provides no benefit, so the semantic content of the intermediate reasoning matters. This is a strong and well-designed control. However, one could imagine a more nuanced hypothesis: perhaps the model benefits not just from any extra tokens, but from tokens that are linguistically well-formed (sentences, even if nonsensical) because they keep the model in a "language generation mode" that enables better computation. The dots control doesn't test this—dots aren't language. A control generating plausible but irrelevant sentences (e.g., describing a completely different problem) would test whether linguistic coherence matters beyond semantic relevance. This control is not run.

The reasoning-after-answer ablation (Figure 5) makes the causal argument: reasoning must precede the answer. This is the strongest evidence that the model uses the reasoning steps to compute the answer rather than producing post-hoc rationalization. But it's worth noting that this ablation changes the exemplar format, and few-shot prompting is known to be sensitive to format changes for reasons that may not reflect causal dependence. The model may simply imitate the format of the exemplars: if exemplars show answer-then-reasoning, the model does the same, and since the answer is generated first (without the benefit of reasoning), accuracy suffers. This doesn't necessarily prove that the model depends causally on its own reasoning steps when they precede the answer—only that the model's behavior is format-consistent. A stronger test would be a "counterfactual" intervention: have the model generate chain-of-thought reasoning, but then force it to output a different answer than the one implied by its reasoning. This is impossible with standard autoregressive generation, so the causal claim remains necessarily inferential.

Missing experiments that would have strengthened the paper:

• Zero-shot chain-of-thought: The paper focuses exclusively on few-shot prompting. A natural ablation would be to test whether simply prepending "Let's think step by step" to the prompt (without any exemplars) elicits reasoning. This zero-shot variant was explored in subsequent work (Kojima et al., 2022) and found to be surprisingly effective, which suggests that the few-shot exemplars may not be as essential as the paper implies—the model may already have a latent "reasoning mode" that can be triggered with a simple instruction.

• Comparison to fine-tuning on the same number of exemplars: The paper compares chain-of-thought prompting to prior fine-tuning results that use thousands of training examples (Cobbe et al., 2021). A fairer comparison at the same data scale would be to fine-tune a model on the 8 hand-written exemplars and test whether the resulting model performs better or worse than prompting with those same exemplars. This would disentangle the value of the examples from the value of the delivery mechanism (prompting vs. gradient updates).

• Testing on truly novel reasoning formats: The symbolic reasoning tasks (last letter concatenation, coin flip) are the closest the paper comes to testing systematic generalization, but they are deliberately simple and the "algorithm" is directly demonstrated in the exemplars. The paper does not test whether chain-of-thought prompting enables solving problems that require novel combinations of reasoning steps not seen in the exemplars—for instance, a math problem requiring both arithmetic and commonsense reasoning in sequence, or a problem whose solution strategy differs from any exemplar.

• Failure mode analysis for commonsense and symbolic tasks: The error analysis (Appendices D.1, D.2) is conducted only on GSM8K. There is no equivalent analysis for why chain-of-thought prompting fails on, say, CSQA for GPT-3 175B (where it hurts performance) or why it achieves only 63.0% on 4-word last letter concatenation. Understanding whether these failures are systematic (e.g., the model loses track of position in long sequences) would clarify the method's limitations.

Sample size and statistical rigor: The test sets vary in size: GSM8K has 1,319 examples (the largest), AQuA has only 254, and the OOD symbolic reasoning sets are constructed by the authors with unspecified sizes. For the coin flip task, the standard deviation across exemplar orders is as high as ±11.1% (Table 7, Annotator C), meaning that different random orderings of the same exemplars can produce accuracy ranging from ~60% to ~82%—a range that nearly spans the gap between standard prompting (49.0%) and the best chain-of-thought result (99.6% for Annotator A). This sensitivity to exemplar order complicates the interpretation of the "emergent ability" claim: is the 99.6% for Annotator A a reliable estimate of the model's capability, or an artifact of a particularly favorable exemplar ordering? The paper acknowledges this sensitivity (Section A.2) but does not resolve it—results for GPT-3 and PaLM are reported for a single exemplar order, so their sensitivity is unknown.

The external calculator confound: The largest gains on arithmetic benchmarks are achieved with the external calculator (Table 1), which is a post-hoc correction, not part of the prompting method. For instance, LaMDA 137B on GSM8K: standard prompting = 6.5%, chain-of-thought = 14.3%, chain-of-thought + calculator = 17.8%. The calculator contributes +3.5 points on top of chain-of-thought's +7.8 points over standard—a substantial fraction of the total improvement. The paper is transparent about this (it reports all three numbers) and the calculator is described as a complementary technique, but the headline results in Figure 2 and the abstract ("surpassing even finetuned GPT-3 with a verifier") are for chain-of-thought without the external calculator, while many of the gains that close the gap to prior best are with the calculator. A reader focused on the abstract might miss this nuance.

Single model checkpoint per model family: The paper evaluates one checkpoint per model size per family. Given known variability across training runs and checkpoints, it is unclear whether the emergence threshold (e.g., LaMDA 68B vs. 137B) reflects a genuine capability transition or sampling a single well-performing large checkpoint. The paper does not report results across multiple training seeds or checkpoints for any model family other than LaMDA's exemplar-order seeds (which vary the prompt, not the model weights).

The Codex results raise questions about the source of reasoning ability: Codex achieves the strongest chain-of-thought results across all arithmetic benchmarks (Table 1), outperforming PaLM 540B on GSM8K (63.1% vs. 56.9%) despite being a smaller model (Codex is based on GPT-3, so likely ~175B parameters vs. PaLM's 540B). The paper does not analyze why Codex benefits so much—is it the code pretraining that teaches procedural reasoning, or is it some other factor? This is a missed opportunity to understand what pretraining data or objectives facilitate chain-of-thought reasoning.

In summary, the paper's experimental evidence robustly demonstrates that chain-of-thought prompting can substantially improve performance on specific reasoning benchmarks when applied to sufficiently large models. The emergence at scale is well-documented, and the ablation experiments provide reasonable (if not airtight) evidence for the claimed mechanism. The limitations are primarily in the narrowness of the task scope, the absence of truly novel generalization tests, the modest absolute performance on the hardest benchmarks, and the unresolved sensitivity to prompt design in some task settings. These limitations do not undermine the paper's core contributions—the method is genuinely effective and the emergent ability finding is robust—but they bound the generality of the claims.

6. Limitations and Trade-offs

Limitation 1: The Method Does Not Help on Problems Outside the Model's Capability Range

The assumption or constraint. Chain-of-thought prompting relies entirely on the base model's pretrained capabilities—it elicits reasoning that already exists but does not instill new capabilities. The paper is candid about this boundary: on the hardest math word problems (GSM8K), PaLM 540B with chain-of-thought prompting achieves only 56.9% accuracy, leaving nearly half the benchmark unsolved. More sharply, on the MultiArith subset of MAWPS (Table 3), standard prompting achieves only 42.2% while chain-of-thought achieves 94.7%—but on AQuA algebraic word problems, even the largest models plateau at 35.8% (PaLM 540B) or 45.3% (Codex), well below reliable deployment thresholds.

The consequence. For any problem where the base model's standard prompting performance is near zero—either because it lacks the required factual knowledge, the reasoning patterns are too far from its training distribution, or the required symbolic manipulation exceeds its procedural competence—chain-of-thought prompting provides no rescue. The method amplifies existing capability but cannot create it de novo. This is clearest in the symbolic reasoning OOD results (Table 5): on 4-word last letter concatenation, PaLM 540B drops from 99.4% (2 words) to 63.0%, and on coin flip with 4 flips, it drops from 100.0% (2 flips) to 90.2%. Performance degrades gracefully with sequence length but does not remain reliable. For a practitioner, this means chain-of-thought prompting is not a substitute for improving the base model—if your model systematically fails at a class of problems (e.g., proofs requiring more than ~4 reasoning steps, or arithmetic with numbers larger than those in the exemplars), chain-of-thought prompting will not bridge that gap.

What evidence exists in the paper. The per-difficulty breakdown for arithmetic is implicit in the MAWPS stratification (Table 3): SingleOp and AddSub problems, which the model already solves well (93–94%), show no gain, while MultiArith, where performance is initially poor (42.2%), shows massive gain (94.7%). But the hardest problems (GSM8K as a whole, where even the best model achieves only 56.9%, and AQuA, where it achieves 35.8%) remain largely unsolved. The paper's error analysis on LaMDA 137B (Appendix D.2) confirms that 54% of incorrect chains of thought have "major errors in semantic understanding or coherence"—these are not minor fixable mistakes but fundamental comprehension failures that additional exemplars are unlikely to resolve. The paper acknowledges this boundary explicitly only in passing: "though note that gain was minimal on CSQA" (Section 4), and in the broader framing that chain-of-thought reasoning "is an emergent ability of model scale" (Section 3.2) rather than a method that works uniformly.

Mitigation status. Not addressed. The paper does not propose any mechanism for extending chain-of-thought prompting to problem classes where the base model fundamentally lacks capability. Follow-up work on combining chain-of-thought with external tools (calculators, code interpreters, retrieval) partially addresses this, but the core limitation—that prompting alone cannot confer new capabilities—remains intrinsic to the method.

Limitation 2: The Prompt Requires Manual, Task-Specific Engineering with Non-Trivial Sensitivity to Annotator Choices

The assumption or constraint. Every task domain evaluated in the paper uses a custom, hand-written set of chain-of-thought exemplars composed by the authors. The paper acknowledges that "prompt engineering still matters, though" (Appendix A.2), and provides evidence of sensitivity to exact wording and annotator identity. On the coin flip task (Table 7), the same model (LaMDA 137B) achieves 99.6% with Annotator A's exemplars but only 71.4% with Annotator C's—a gap of 28.2 percentage points despite both annotators being co-authors familiar with the method's goals. The standard deviation across exemplar orders for Annotator C is ±11.1%, meaning different random orderings of the same prompt produce accuracy ranging from roughly 60% to 82%—nearly the entire gap between standard prompting (49.0%) and the best chain-of-thought result (99.6%).

The consequence. The method is not "write any chain of thought and it will work"—effective exemplar design requires skill, iteration, and task-specific understanding that the paper does not formalize. For a practitioner deploying chain-of-thought prompting on a new task, the paper provides no recipe beyond "write step-by-step reasoning that leads to the final answer." There is no guidance on exemplar selection (how many? which problems? what reasoning style?), no automated method for generating or optimizing chains of thought, and no principled way to anticipate whether a given prompt will succeed or fail without expensive trial-and-error evaluation on a held-out set. The paper's robustness experiments (Section 3.4) test only three annotators (all co-authors) and one alternative source (GSM8K crowdworkers)—this establishes a lower bound on sensitivity but does not characterize the full variance. A practitioner writing chain-of-thought exemplars for their own domain (legal reasoning, medical diagnosis, financial analysis) has no guarantee that their first attempt will produce gains, or that gains on one task will transfer to related tasks.

What evidence exists in the paper. The annotator robustness results (Figure 6, Tables 6–7) show that all three annotators outperform standard prompting on arithmetic (GSM8K: 14.3%, 15.5%, 17.6%; MAWPS: 57.9%, 58.2%, 60.1%), but the variance is larger on commonsense (Sports Understanding: 85.8%, 75.4%, 81.1%) and symbolic tasks (coin flip: 99.6%, 77.5%, 71.4%). The intentionally concise style (Table 6) achieves 11.1% on GSM8K vs. 14.3% for the verbose style—a 3.2-point gap from changing sentence structure alone. The GSM8K-sourced exemplars (three random sets) cluster tightly (12.6–12.7%) but underperform the manually composed ones (14.3%). The paper's broader claim that "successful use of chain of thought does not depend on a particular linguistic style" (Section 3.4) is only true relative to the standard prompting baseline: all styles beat the baseline, but the magnitude of improvement varies considerably.

Mitigation status. Partially addressed. The paper acknowledges the issue explicitly (Section A.2: "prompt engineering still does matter, and can improve performance significantly in many cases") and suggests future work on "using a large language model to automatically generate chains of thought via prompting (and potentially optimize this over a validation set)." However, no automated method is developed or evaluated. The fixed 8-exemplar prompts used in the main experiments are provided in full (Appendix G) for reproducibility, but a practitioner applying the method to a new task receives no actionable guidance for prompt construction beyond the provided examples.

Limitation 3: Chain-of-Thought Reasoning Is Not Guaranteed to Be Factual, Coherent, or Faithful

The assumption or constraint. The paper explicitly acknowledges that "there is no guarantee of correct reasoning paths, which can lead to both correct and incorrect answers" (Section 6, Limitations paragraph). This is not a hypothetical concern—it is empirically demonstrated throughout the paper. Appendix D.1 shows that even when the model produces a correct final answer, the reasoning chain can be imperfect (Table 9: 5 of 50 correct chains had underspecified statements, irrelevant statements, or incorrect equations that coincidentally produced the right answer). Appendix D.2 shows that 54% of incorrect chains of thought have major semantic or coherence errors (Table 11), and that 8% of incorrect chains would be correct with an external calculator—meaning the model stated the correct reasoning but computed incorrectly.

The consequence. For any application where the reasoning process itself matters—explainable AI, education, debugging model behavior, or high-stakes decision-making where incorrect reasoning cannot be tolerated—chain-of-thought prompting provides no reliability guarantee. The model can produce a fluent, persuasive-sounding chain of thought that is entirely wrong (as in the coin flip and CSQA examples in Tables 15 and 14). A user who reads the chain of thought may be misled into trusting an incorrect answer because the reasoning sounds plausible, a failure mode arguably worse than simple wrong answers where the lack of explanation signals uncertainty. This limitation is especially acute for multiple-choice and binary classification tasks (as noted in Appendix D.1: "it was much more likely that models could arrive at the correct answer via an incorrect reasoning path"), which constitute all commonsense reasoning benchmarks, StrategyQA, CSQA, Sports Understanding, and the SayCan task—the majority of the paper's evaluation suite beyond arithmetic.

What evidence exists in the paper. The error analysis in Appendices D.1 and D.2 provides direct evidence. Table 9 shows examples where the model's reasoning is flawed but the answer is correct: on a penguin sunbathing problem, the model computes one-third of 36 twice as 12 each but never adds them together, yet coincidentally arrives at 12 as the answer. Table 10 shows chains that are logically correct but contain symbolic errors (wrong multiplication terms) that happen to produce a wrong answer. Table 11 shows incoherent chains where statements do not follow from prior ones (e.g., "the percentage of the entire students enrolled in hip-hop dance is the percentage of the students enrolled in hip-hop dance, minus the percentage of the students enrolled in contemporary dance"—a tautological and meaningless statement). The paper notes the calculator error category (8% of errors: "the chain of thought could be made correct just by running the equation through an external calculator") as evidence that reasoning and arithmetic are separable failure modes. The external calculator (Table 1) partially mitigates arithmetic errors but does not address logical, semantic, or factual errors in the reasoning itself.

Mitigation status. Acknowledged but not solved. The paper lists this as a limitation (Section 6) and references prior work on improving factual generations (Rashkin et al., 2021; Ye and Durrett, 2022; Wiegreffe et al., 2022) as potential directions. The external calculator (Section 3.2) mitigates arithmetic errors specifically, and the paper notes that follow-up work on self-consistency (Wang et al., 2022a) can improve reliability by sampling multiple chains and taking the majority answer, but neither technique addresses the fundamental issue of logically incorrect reasoning that happens to produce a right answer, nor reasoning that is internally coherent but factually wrong.

Limitation 4: The Method Is Evaluated on a Narrow Set of Reasoning Tasks, All in English, with No Evidence of Cross-Lingual or Cross-Modal Transfer

The assumption or constraint. All experiments in the paper use English-language benchmarks from a specific set of reasoning categories: arithmetic word problems (5 benchmarks), commonsense QA (5 benchmarks), and synthetic symbolic manipulation (2 tasks). The paper claims the method is "broadly applicable" (Section 6) and "potentially applicable (at least in principle) to any task that humans can solve via language" (Section 2), but this generality claim is an extrapolation from a narrow empirical base. Moreover, all benchmarks are in English and assume Anglo-centric commonsense knowledge (CSQA) and US-centric sports knowledge (Sports Understanding). The paper does not evaluate on multi-lingual reasoning, non-English tasks, multi-modal reasoning, or domains where the reasoning chain is not naturally expressible in natural language (visual reasoning, mathematical proofs requiring diagrams, procedural tasks with physical constraints).

The consequence. A practitioner cannot assume that chain-of-thought prompting will work for their task simply because it involves reasoning. The paper identifies three conditions under which chain of thought helps most (Appendix A.3): "(1) the task is challenging and requires multi-step reasoning, (2) a large language model is used, and (3) the scaling curve is relatively flat." But these conditions are post-hoc generalizations from the evaluated tasks—they are not validated on held-out task types. Tasks that require different reasoning modalities (spatial reasoning, causal reasoning with temporal dynamics, reasoning about uncertainty or probabilities) may not benefit, or may require substantially different exemplar designs. Tasks in languages other than English may fail if the base model's pretraining data in that language is insufficient to support the semantic parsing and arithmetic capabilities that chain-of-thought reasoning depends on. The SayCan results (robotics instruction following, 91.7%) suggest some transfer to embodied task planning, but this is a single domain with a small number of available actions—it is unclear whether the method scales to open-ended planning or reasoning about physical dynamics.

What evidence exists in the paper. The paper provides no experiments in non-English languages, no multi-modal tasks, and no systematic evaluation across reasoning types beyond the three categories evaluated. The SayCan task is the closest to a "real-world" application beyond QA-style benchmarks. The paper's Appendix A.3 offers informal guidance on when chain of thought helps, but this guidance is derived from the very tasks being evaluated (in-sample generalization) and is not tested on held-out task families. The robustness to different exemplars from GSM8K (Table 6) shows cross-distribution generalization within math word problems (GSM8K exemplars work on SVAMP, ASDiv, MAWPS), but all these datasets share the same underlying reasoning structure—the generalization is across problem distributions, not reasoning types.

Mitigation status. Not addressed. The paper acknowledges the scope of evaluation only implicitly, by describing what was tested. It does not claim to have tested cross-lingual or cross-modal transfer, and the "in principle" qualification in Section 2 is appropriately cautious. However, the paper's strong claims about broad applicability ("chain-of-thought reasoning can be used for tasks such as math word problems, commonsense reasoning, and symbolic manipulation, and is potentially applicable... to any task that humans can solve via language") invite overgeneralization that the experimental design does not warrant. No future work is proposed on evaluating the method beyond the tested task categories.

Limitation 5: The Method Is Fundamentally Bottlenecked by the Model's Context Window, Limiting the Complexity of Problems and Number of Exemplars

The assumption or constraint. All experiments use models with a 1024-token context window, which constrains the total prompt length (exemplars + test question + generated output) to at most 1024 tokens. The paper uses 8 exemplars for arithmetic reasoning (except AQuA, which uses 4), with the note that "further increasing the number of exemplars in standard prompting did not lead to significant gains (e.g., increasing from 8 to 16 exemplars did not improve the performance of standard prompting enough to catch up with chain-of-thought prompting)" (Appendix A.2). The exemplars themselves are kept short—the paper samples GSM8K training examples "≤60 tokens to fit into our input context window, and also limit the examples to ≤2 steps to solve" (Section 3.4, footnote 2).

The consequence. This context-length constraint imposes a hard ceiling on the method's applicability. Problems requiring very long reasoning chains (e.g., multi-page mathematical proofs, complex legal reasoning spanning many precedents, or scientific reasoning requiring extensive background knowledge) cannot fit both the necessary exemplars and the full reasoning trace within 1024 tokens. The generated chain of thought consumes context tokens that could otherwise be used for additional exemplars or for the problem statement itself—there is an inherent tradeoff between providing more demonstrations and having space for the model to "show its work." This is visible in the OOD symbolic reasoning results: on 4-word last letter concatenation, PaLM 540B achieves only 63.0% (Table 5), and on 4-flip coin flip, 90.2%. These tasks are still well within the context window, yet performance degrades with length—suggesting that as the reasoning chain grows, the model's ability to maintain coherence across many steps diminishes. For real-world problems requiring dozens or hundreds of reasoning steps, this degradation would likely be severe.

Furthermore, the context limitation constrains exemplar design. The 8 arithmetic exemplars cover only basic operations (addition, subtraction, multiplication, division) and relatively simple problem structures. A task requiring more diverse reasoning patterns (e.g., a math competition problem requiring modular arithmetic, combinatorics, and geometric reasoning) would need either more exemplars (impossible within 1024 tokens) or exemplars that somehow cover these diverse patterns densely—a difficult prompt engineering challenge that the paper does not address.

What evidence exists in the paper. The paper does not explicitly vary context length, as all evaluated models have the same 1024-token limit. The robustness experiment varying number of exemplars from 1 to 8 (Figure 11) shows that chain-of-thought performance improves monotonically with more exemplars on most tasks, suggesting that even more exemplars might yield further gains if the context window permitted it. The length generalization results (Table 5, Figure 8) provide indirect evidence: performance on 4-word concatenation (63.0%) is substantially lower than on 2-word (99.4%) and 3-word (94.8%), suggesting that longer reasoning chains—which consume more context tokens for the same number of exemplars—are harder for the model to handle. The error analysis for long sequences is not performed, so it is unclear whether the degradation is due to loss of coherence, increased arithmetic errors, or some other mechanism.

Mitigation status. Not addressed. The paper does not discuss the context-window limitation or propose strategies for extending chain-of-thought prompting to longer problems (e.g., chunking, iterative prompting, or leveraging models with larger context windows). At the time of writing (2022), the 1024-token limit was standard for the evaluated models (GPT-3, LaMDA, PaLM), and the paper treats it as a fixed constraint rather than a limitation to be overcome. Subsequent models (GPT-4, Claude) with much larger context windows partially alleviate this constraint, but the paper's experiments provide no guidance on how chain-of-thought prompting scales with context length.

7. Implications and Future Directions

How This Work Changes the Landscape

Chain-of-thought prompting caused a conceptual reframing of the reasoning problem in language models rather than a paradigm shift in model architecture or training. Prior to this work, the dominant approaches for enabling multi-step reasoning in neural models fell into two camps: (1) training from scratch or fine-tuning on rationale-augmented datasets, which required thousands of human-written reasoning chains per task (Ling et al., 2017; Cobbe et al., 2021), or (2) neuro-symbolic methods that converted problems into formal languages and used specialized solvers (Roy and Roth, 2015; Chiang and Chen, 2019). Both approaches treated reasoning as a capability deficit—the model did not know how to reason, so you had to teach it through supervised examples or hand-engineered formalisms.

The paper's central demonstration—that PaLM 540B with 8 hand-written exemplars and no gradient updates achieves 56.9% on GSM8K, surpassing a fine-tuned GPT-3 175B with a trained verifier (55%)—fundamentally challenged this assumption. The reasoning capability was already latent in the model; the standard prompting interface simply failed to access it. This reframed reasoning from a training problem (collect more rationale data, fine-tune on more tasks) to an elicitation problem (discover prompt structures that cause the model to express its existing reasoning capabilities). The practical consequence was immediate: a single model checkpoint, prompted differently for each task, could perform arithmetic, commonsense, and symbolic reasoning without any task-specific training data—a qualitative shift from the prior paradigm where each reasoning domain required its own fine-tuned model.

The paper's emergent ability finding (Figure 4, Tables 2, 4, 5) provided empirical shape to this reframing. The observation that chain-of-thought prompting degrades performance in models below approximately 50–70B parameters but yields large and growing gains above approximately 100B parameters established that the capability is not a smooth function of scale but exhibits a phase transition. This was a concrete counterexample to the prevailing smooth-scaling narrative (Kaplan et al., 2020) and provided an early, well-documented instance of what Wei et al. (2022b) would formalize as "emergent abilities." The practical implication was cautionary: evaluating prompting methods only on small models (a common academic practice due to cost) could produce actively misleading conclusions—a method that fails at 6B parameters might succeed dramatically at 175B.

The paper also resolved a tension in the prior literature that had not been explicitly articulated as a contradiction but was visible in retrospect. On one hand, rationale-augmented training worked (Cobbe et al., 2021 achieved 55% on GSM8K with fine-tuning), suggesting that intermediate reasoning steps were valuable. On the other hand, standard few-shot prompting—which provided no intermediate steps—produced flat scaling curves on reasoning tasks (Rae et al., 2021), suggesting that larger models did not naturally acquire reasoning ability through scale alone. Chain-of-thought prompting reconciled these findings by showing that the knowledge of how to reason was present in large models, but the interface for expressing it was missing. The flat scaling curves of standard prompting were not evidence that models couldn't reason; they were evidence that models couldn't express their reasoning through direct answer prediction. This reconciliation opened up a new axis of research: rather than asking "can models reason?" (a binary question), the field shifted to asking "under what prompting conditions do models express reasoning?" (a question about interface design).

Research directions that became more attractive after this paper:

• Prompt engineering for reasoning. The paper established that prompt structure—specifically, the inclusion of intermediate reasoning steps before the answer—could unlock capabilities invisible under standard prompting. This validated and accelerated research into more sophisticated prompting strategies, including self-consistency (Wang et al., 2022a, cited in the paper), tree-of-thoughts, and program-aided language models. The paper made prompt engineering for reasoning a legitimate research pursuit rather than an ad-hoc trick.

• Understanding and improving emergent abilities. The documentation of a specific emergent ability (chain-of-thought reasoning) across three model families provided a concrete phenomenon for the field to study. Questions about which pretraining data, architectural features, or training objectives cause emergence became empirically tractable—researchers could use chain-of-thought prompting as a probe for whether a model had crossed the reasoning threshold.

• Inference-time compute allocation. The paper's demonstration that generating a chain of thought allocates more computation (intermediate tokens) to harder problems, and that this computation is usefully deployed (not wasted, as the dots ablation showed), connected prompting to the broader question of how to optimally spend inference compute. This connection was made more explicit in subsequent work on test-time compute scaling.

Research directions that became less critical:

• Collecting large rationale-augmented fine-tuning datasets. The paper showed that for sufficiently large models, 8 hand-written exemplars could outperform fine-tuning on thousands of examples. While fine-tuning remained valuable (and the paper did not claim prompting universally dominated it), the marginal value of ever-larger rationale datasets was called into question—at least for the class of reasoning tasks tested.

• Neuro-symbolic methods for math word problems. The paper demonstrated that a purely neural, language-based approach (no formal symbolic solver, no equation parser, no separate arithmetic module) could achieve state-of-the-art on several math word problem benchmarks. This did not eliminate the value of neuro-symbolic methods (the external calculator in Table 1 shows that combining neural reasoning with symbolic arithmetic still helps), but it reduced the urgency of developing domain-specific formalisms for each reasoning task.

Follow-Up Research This Work Enables

Zero-shot chain-of-thought: is the few-shot demonstration necessary, or is the capability triggerable with a simple instruction? The paper exclusively uses few-shot prompting with manually written exemplars, but the ablation experiments (Section 3.3) raise a natural question: if the model already possesses latent reasoning capabilities, can they be elicited without any exemplars at all—simply by prepending "Let's think step by step" to the question? This would test whether the few-shot exemplars teach the model how to reason or merely signal that reasoning is expected. A strong follow-up would compare zero-shot chain-of-thought (instruction only, no exemplars) to few-shot chain-of-thought on the same benchmarks (GSM8K, StrategyQA, last letter concatenation), across the same model scales evaluated in this paper. If zero-shot performs comparably to few-shot at large scales, it would suggest that the exemplars are acting as a format signal rather than teaching a procedure—a finding that would substantially simplify deployment (no manual exemplar writing needed). If zero-shot substantially underperforms few-shot, it would confirm that the exemplars provide necessary procedural guidance. The paper's robustness experiments (varying number of exemplars, Figure 11) show that even a single exemplar provides a gain, which weakly suggests that exemplars matter—but the zero-shot condition was never tested. This experiment was subsequently conducted by Kojima et al. (2022), confirming that zero-shot chain-of-thought is effective, which retroactively suggests the few-shot exemplars were partly acting as a format specifier.

What pretraining data or objectives cause the emergence of chain-of-thought reasoning? The paper documents that chain-of-thought reasoning emerges at approximately 100B parameters across three model families, but offers no causal explanation for why. The Codex results (Table 1: 63.1% on GSM8K vs. PaLM 540B's 56.9%, despite Codex being likely ~175B parameters) strongly hint that code pretraining may be a key factor—code corpora contain abundant examples of sequential, step-by-step procedural reasoning that natural language text may lack. A controlled experiment would ablate the pretraining data mixture: train multiple models of the same architecture and scale on different data mixtures (natural language only, natural language + code, natural language + structured math, natural language + formal proofs) and measure whether chain-of-thought prompting's effectiveness at a given scale depends on the presence and proportion of procedurally-structured data. The paper's UL2 20B results (Table 2: only 4.4% on GSM8K with chain-of-thought, vs. LaMDA 68B's 8.2% at a larger scale but different architecture) suggest that architecture and pretraining objective also matter—UL2 uses a different pretraining paradigm (mixture of denoisers) that may not foster the same reasoning capabilities. A systematic ablation across pretraining objectives (causal LM, prefix LM, span corruption, mixture-of-denoisers) at matched scale would begin to isolate the causal factors.

Does chain-of-thought prompting enable compositional generalization to novel reasoning patterns, or only length generalization and cross-distribution transfer? The paper demonstrates length generalization on symbolic tasks (applying a learned procedure to longer sequences) and cross-distribution transfer on arithmetic (GSM8K exemplars working on SVAMP, ASDiv, MAWPS). But both forms of generalization stay within the same type of reasoning: concatenating more letters, tracking more coin flips, solving more math word problems with the same underlying structure. A stronger test would evaluate whether the model can compose reasoning patterns from different exemplars to solve a problem requiring a novel combination. For instance: provide exemplars demonstrating (a) unit conversion reasoning and (b) multi-step arithmetic with percentages, then test on a problem that requires both unit conversion and percentage calculation in sequence—a combination never seen in any single exemplar. If chain-of-thought prompting enables such compositional generalization, it would suggest that the model is not just replaying demonstrated procedures but genuinely recombining reasoning primitives. If it fails, it would reveal that the method's generalization is limited to instantiating known templates with new values—an important boundary condition. The paper's current evidence cannot distinguish these hypotheses.

What is the failure mode at OOD lengths, and can it be mitigated? On last letter concatenation, PaLM 540B drops from 99.4% (2 words) to 94.8% (3 words) to 63.0% (4 words). On coin flip, the drop is from 100.0% (2 flips) to 98.6% (3 flips) to 90.2% (4 flips). The paper does not analyze why performance degrades—does the model lose track of position (generating the same letter twice or skipping a word), does it fail to parse longer names into individual words, or does it successfully reason but make errors in the final concatenation? An error analysis on the OOD failures, analogous to the LaMDA 137B GSM8K analysis in Appendix D.2, would characterize the degradation mode. If the errors are primarily positional (e.g., on 4-word names, the model correctly extracts all four last letters but concatenates them in the wrong order 30% of the time), that suggests a specific deficit in positional tracking that might be addressable through prompt design (e.g., numbering the steps). If the errors are parsing failures (the model treats a 4-word name as 2 or 3 words), that suggests a more fundamental limitation. The experiment is straightforward: collect the model's outputs on OOD examples, manually categorize the error types, and compute how each error category grows with sequence length.

Can chain-of-thought prompting be automated? The paper's largest practical limitation is the need for manual, task-specific exemplar writing. The robustness experiments (Section 3.4) show that automatically-sourced exemplars (randomly sampled from GSM8K training set) work nearly as well as manually composed ones on GSM8K (12.6–12.7% vs. 14.3% for LaMDA 137B), but this still assumes a training set with human-written reasoning chains exists. For a new task with no such dataset, a natural approach would be to use a large language model itself to generate chain-of-thought exemplars: prompt the model (perhaps with a few seed exemplars or a meta-instruction) to produce step-by-step reasoning for training problems, then use those auto-generated chains as exemplars for the actual task. The paper's Appendix A.2 explicitly mentions this idea: "use a large language model to automatically generate chains of thought via prompting (and potentially optimize this over a validation set)." A follow-up could evaluate this on a task where human-written chains are expensive (e.g., medical diagnosis reasoning) by comparing auto-generated exemplars to human-written exemplars and to a no-chain-of-thought baseline. If auto-generation approaches human-level prompt quality, it removes the main barrier to deploying chain-of-thought prompting at scale across diverse domains.

Does chain-of-thought prompting improve robustness to input perturbations, or does it create new brittleness through reliance on fragile reasoning chains? Prior work on rationale-augmented training (Chen et al., 2022, cited in the paper's related work) found that generating rationales can improve model robustness to small input changes. For chain-of-thought prompting, the question is whether the model's reliance on an explicit reasoning chain makes it more or less vulnerable to perturbations that break the reasoning—e.g., slight rewordings of a math problem that preserve the underlying structure but change surface features, or the insertion of irrelevant distracting information (of the kind tested in the SVAMP benchmark). The SVAMP results (Table 2) show that chain-of-thought prompting improves accuracy on this structure-varying benchmark (PaLM 540B: 69.4% → 79.0%, +9.6 points), but this is aggregate accuracy, not an analysis of which types of perturbations the method is robust to. A systematic follow-up would take a set of problems that the model solves correctly with chain-of-thought prompting, apply controlled perturbations (synonym substitution, clause reordering, insertion of irrelevant sentences, changing numerical values while preserving the problem structure), and measure whether chain-of-thought accuracy degrades more or less than standard prompting accuracy under each perturbation type. This would reveal whether explicit reasoning chains act as a stabilizing force (the model follows the same logical steps regardless of surface form) or a fragility (the chain breaks when the surface form changes, even if the underlying structure is identical).

Practical Applications and Downstream Use Cases

Cost-efficient deployment of a single model across diverse reasoning tasks without per-task fine-tuning. The paper's most direct practical application is replacing multiple task-specific fine-tuned models with a single large language model prompted differently for each reasoning task. A technology company maintaining separate fine-tuned models for math tutoring (GSM8K-style word problems), commonsense QA for customer support (CSQA-style questions), date calculations for scheduling applications, and sports knowledge for a trivia bot could consolidate these into a single PaLM 540B (or equivalent) instance, with the appropriate chain-of-thought prompt prepended to each query. The cost savings come from eliminating fine-tuning infrastructure (training data collection, GPU-hours for fine-tuning, model storage and serving costs for multiple checkpoints). The paper provides evidence that this consolidation does not sacrifice quality: PaLM 540B with chain-of-thought prompting matches or exceeds prior fine-tuned state-of-the-art on GSM8K (56.9% vs. 55%), SVAMP (79.0% vs. 57.4%), MAWPS (93.3% vs. 88.4%), and StrategyQA (75.6% vs. 69.4%). The primary deployment cost is prompt design (hand-writing ~8 exemplars per task domain), which is a one-time cost amortized across all queries—the paper's robustness results (Figure 6) show that even non-expert annotators (GSM8K crowdworkers) produce effective prompts, suggesting prompt design is not a specialized skill requiring ML expertise.

Interpretable output for high-stakes reasoning applications where users need to verify the reasoning process. In educational settings (automated math tutoring, homework feedback), medical decision support, or legal reasoning, the final answer alone is often insufficient—users need to understand how the answer was derived to trust it, learn from it, or verify its correctness. Chain-of-thought prompting provides this interpretability for free: the model generates the reasoning steps as part of its output, and the paper's error analysis (Appendix D.1) shows that for GSM8K, 49 of 50 correct answers were accompanied by logically and mathematically correct reasoning chains. This means that in approximately 98% of cases where the model gets the right answer, the accompanying reasoning is trustworthy enough for a human to follow. The caveat, documented in Appendix D.1, is that for classification tasks (multiple choice, yes/no), "it was much more likely that models could arrive at the correct answer via an incorrect reasoning path"—so this use case is most viable for free-response tasks where coincidentally correct reasoning is unlikely. For educational math tutoring specifically (free-response word problems), the generated chain of thought doubles as a worked solution that a student can study, with the paper's error categories (Appendix D.2: calculator errors, one-step-missing errors) providing a diagnostic vocabulary for identifying where the model's reasoning broke down when it does err.

Data generation for training or fine-tuning smaller, specialized models. Chain-of-thought prompting with a large model can generate high-quality reasoning traces that serve as training data for smaller, more efficient models. The paper shows that PaLM 540B generates correct reasoning chains for 56.9% of GSM8K problems; with the external calculator, this rises to 58.6%. These chains—including the reasoning steps, not just the final answers—can be filtered for correctness (using known ground-truth answers on a training set) and used to fine-tune a smaller model (e.g., a 7B-parameter model) on the task of generating step-by-step reasoning. The resulting fine-tuned model could potentially match or exceed the prompting-based performance of the large model while being much cheaper to serve. The paper does not perform this distillation experiment, but the quality of the generated chains (49/50 correct reasoning traces when the answer is correct, per Appendix D.1) suggests the training signal is clean. The practical benefit is enabling reasoning-capable models at a fraction of the inference cost—critical for latency-sensitive or high-volume applications where querying a 540B-parameter model on every request is infeasible. The paper's external calculator results (Table 1) further suggest that the generated data could be post-processed to correct arithmetic errors before being used for fine-tuning, improving training data quality.

Prompt-based debugging and error analysis for model improvement. The paper's error analysis framework (Appendices D.1, D.2 and A.1) provides a practical methodology for understanding why a model fails on reasoning tasks. By categorizing errors into calculator errors, symbol mapping errors, one-step-missing errors, semantic understanding errors, and incoherence (Appendix D.2, Table 10, Table 11), a practitioner can diagnose the primary failure modes of their model on a given task and target interventions accordingly. For instance, if 8% of errors are calculator errors, adding an external calculator (as in Table 1) directly addresses that fraction. If 22% are one-step-missing errors, improving the model's ability to maintain complete reasoning chains (perhaps through better prompt design or by scaling to a larger model, per Figure 9) is the priority. If 54% are major semantic errors, the model fundamentally lacks the needed comprehension and no amount of prompt engineering will help—the intervention should be at the pretraining or fine-tuning level. This diagnostic workflow, made possible by the interpretability of chain-of-thought outputs, enables more targeted model improvement than aggregate accuracy metrics alone. The paper's scaling error analysis (Appendix A.1, Figure 9) demonstrates this approach in action: by categorizing PaLM 62B's errors and then checking which categories PaLM 540B fixes, the authors identified that scaling is most effective at fixing one-step-missing errors (67% fixed) and least effective at fixing semantic understanding errors (30% fixed)—a granular finding that would be invisible from aggregate accuracy improvements alone.

When to Prefer This Method

The paper positions chain-of-thought prompting not against a specific named alternative method but as a general approach that is "more helpful for some tasks than others" (Appendix A.3). It does not articulate an explicit decision rule of the form "prefer chain-of-thought prompting over alternative X when conditions A, B, C hold." Instead, it offers a set of conditions under which the method is likely to be effective, derived from the empirical results:

• The task requires multi-step reasoning. The paper states: "Our intuition is that chain of thought helps the most when three conditions are met: (1) the task is challenging and requires multi-step reasoning, (2) a large language model is used, and (3) the scaling curve is relatively flat" (Appendix A.3). The MAWPS stratification (Table 3) provides direct evidence: on SingleOp problems requiring a single operation, chain-of-thought provides zero gain (94.1% for both standard and chain-of-thought with PaLM 540B); on MultiArith problems requiring multiple steps, the gain is massive (42.2% → 94.7%, +52.5 points). The paper also notes that "the performance gain is small for the subsets of MAWPS that only require one or two steps (SingleOP, SingleEq, and AddSub), for which PaLM 540B already achieves performance of 90% or higher" (Appendix A.3). The implication is practical: before applying chain-of-thought prompting, assess whether the task actually requires composing multiple reasoning steps. If the task can be solved by direct retrieval or single-step inference, standard prompting may be sufficient.

• A large language model (approximately 100B+ parameters) is available. The emergent ability finding (Figure 4, Tables 2, 4, 5) establishes a hard constraint: chain-of-thought prompting degrades performance for models below ~50B parameters and only provides substantial gains above ~100B parameters. The paper states that "chain-of-thought reasoning is emergent in the sense that its success cannot be predicted only by extrapolating the performance of small scale models, as chain of thought actually hurts performance for most models smaller than 10B parameters" (Appendix A.1). For a practitioner with access only to smaller models (e.g., GPT-3 6.7B, which achieves 4.0% standard vs. 2.4% chain-of-thought on GSM8K), chain-of-thought prompting is actively harmful and should not be used. This boundary is unusually crisp for a machine learning method and should be taken seriously in deployment decisions.

• The scaling curve for standard prompting is relatively flat. The paper observes that tasks where standard prompting already achieves high performance with large models (e.g., SingleOp on MAWPS: PaLM 540B at 94.1%) show minimal room for improvement from chain-of-thought prompting. The flat-scaling-curve condition identifies tasks where the model has the capability but standard prompting fails to access it—precisely the regime where chain-of-thought prompting is most valuable. The paper notes that "conversely, the benefits are smaller when one or more of these conditions are not met" (Appendix A.3). For a new task, a practitioner can cheaply evaluate this condition by measuring standard prompting accuracy at the largest available model scale; if accuracy is low and has not improved substantially with scale (a flat scaling curve), chain-of-thought prompting is a promising intervention.

• The reasoning can be expressed naturally in language. The paper implicitly assumes this condition (all evaluated tasks involve language-based reasoning), but the generality claim in Section 2—"potentially applicable (at least in principle) to any task that humans can solve via language"—suggests a boundary: tasks whose reasoning process is not naturally verbalizable (visual pattern recognition, motor planning, intuitive judgments) are unlikely to benefit. The SayCan results (91.7%, Table 4) show that robotic action planning—which is naturally expressed as a sequence of language-described actions—falls within the method's scope, but the paper does not evaluate tasks where reasoning is non-linguistic.

These conditions are offered as informal guidance rather than a rigorous decision procedure. The paper does not provide a comparative evaluation against named alternatives (e.g., "in regime X, use prompting; in regime Y, use fine-tuning"), so a structured decision matrix would be speculative. The closest the paper comes to such a comparison is the implicit contrast with fine-tuning in the GSM8K result (Figure 2: prompting surpasses fine-tuning), but this is a single benchmark and the comparison is not systematically explored across tasks, model scales, or data availability regimes.