Self-Consistency Improves Chain of Thought Reasoning in Language Models

🎯 Pitch

Simply sampling multiple reasoning paths from a frozen language model and taking a majority vote can boost arithmetic accuracy by over 17 absolute points on GSM8K—without any additional training or annotation, the model’s own diversity of thought effectively self-corrects errors that would derail a single greedy decode.

1. Executive Summary

This paper introduces a new decoding strategy called self-consistency, which replaces the naive greedy decoding used in chain-of-thought prompting with a "sample-and-marginalize" procedure: the language model first samples a diverse set of reasoning paths (via temperature sampling and top-k truncation) instead of selecting only the single most likely path, then aggregates the final answers across those paths by taking a majority vote to select the most consistent answer. Evaluated on arithmetic (GSM8K, SVAMP, AQuA) and commonsense (StrategyQA, ARC-challenge) reasoning benchmarks across four models—UL2-20B, LaMDA-137B, GPT-3-175B, and PaLM-540B—self-consistency boosts chain-of-thought prompting by striking margins, including +17.9% absolute accuracy on GSM8K with PaLM-540B and achieving new state-of-the-art performance across nearly all tasks. The method is entirely unsupervised, requires no additional training, auxiliary models, or human annotation, and works off-the-shelf with pretrained language models—establishing that diversity in the reasoning process, even when the final answer is fixed, can substantially improve reasoning accuracy without any task-specific specialization.

2. Context and Motivation

The Core Problem: Greedy Decoding Leaves Reasoning Performance on the Table

The fundamental question this paper tackles is straightforward: when a language model performs chain-of-thought reasoning, should we trust only its single most likely reasoning path, or should we consider multiple possible paths? Prior to this work, the standard approach for decoding in reasoning tasks was greedy decoding — at each token position, select the single highest-probability token and proceed sequentially. This is the approach used in the original chain-of-thought prompting paper (Wei et al., 2022) and was the default in large language models at the time (Chowdhery et al., 2022).

The problem is that greedy decoding, by construction, sees only one way through the reasoning process. If that single path happens to contain an error — a miscalculation, a misinterpreted premise, a flawed logical step — the model's final answer will be wrong, even if the model "knows" the correct answer and could have arrived at it through a different reasoning trajectory. This is not a hypothetical concern: the paper provides concrete examples where greedy decoding produces incorrect answers that alternative sampled reasoning paths from the very same model correctly resolve (Table 4, Tables 12–13 in the appendix). For instance, on a GSM8K problem about a 60-mile bike trip with two stops, greedy decoding on PaLM-540B produces the answer "40 miles" by skipping a subtraction step, while two sampled paths independently arrive at "25 miles" through different but correct arithmetic sequences (Table 4).

This gap matters for several practical reasons the paper makes clear:

Reasoning benchmarks are high-stakes evaluation targets. Tasks like GSM8K, AQuA, and ARC-challenge represent the frontier of what language models can do — they test multi-step logical deduction, not simple pattern matching. Improvements on these benchmarks signal genuine progress in model capability. When greedy decoding underperforms relative to what the model could achieve, it undersells the model's true reasoning ability.
The cost of a wrong answer in reasoning is binary and unforgiving. Unlike open-ended text generation where multiple outputs might be acceptable, math word problems and logical reasoning tasks have exactly one correct answer. A single arithmetic mistake in a 5-step reasoning chain invalidates the entire solution. Greedy decoding provides no mechanism to recover from such local errors.
Human reasoning is naturally multi-path. As the authors note (Section 2, citing Stanovich & West, 2000), humans tackling complex problems think in multiple different ways about the same question. If someone arrives at the same answer through two different reasoning strategies, their confidence increases. Greedy decoding, by contrast, simulates a single, rigid thinking process — it is an unnatural constraint on the reasoning capacity that the model may possess.

Conflicting Design Philosophy in Language Model Decoding

The paper is motivated by a tension between two established practices in language model decoding:

For open-ended text generation, sampling-based decoding strategies (temperature sampling, top-k sampling, nucleus sampling) are the norm (Radford et al., 2019; Brown et al., 2020; Holtzman et al., 2020). These methods introduce diversity precisely because there is no single "correct" continuation — for a chat response or a story, multiple outputs can be equally valid, and sampling prevents the repetitive, degenerate text that greedy decoding often produces.

For reasoning tasks with fixed answers, however, greedy decoding has been the default (Wei et al., 2022; Chowdhery et al., 2022). The intuition is straightforward: if the correct answer is fixed and determined, diversity in the decoding process might seem like noise — why would you want multiple different reasoning paths when only the correct one matters?

This paper challenges that intuition head-on. The key insight is that diversity in the reasoning process is valuable even when the final answer is fixed, because a complex reasoning problem "typically admits multiple different ways of thinking leading to its unique correct answer" (Section 1). The model might know how to solve the problem through several distinct approaches, but greedy decoding only reveals one of them. By sampling multiple reasoning paths and aggregating their answers via majority vote, the correct answer can emerge from the consensus even if no single sampled path is guaranteed correct.

This is fundamentally different from simply increasing the model's temperature and hoping for a lucky sample. Self-consistency relies on a statistical property: correct reasoning paths, even when diverse, tend to converge on the same answer, while incorrect paths tend to scatter across different wrong answers. The method works because of this asymmetry — wrong answers are idiosyncratic, correct answers are convergent. The paper explicitly states this hypothesis in Section 2:

"We hypothesize that correct reasoning processes, even if they are diverse, tend to have greater agreement in their final answer than incorrect processes."

Where Prior Approaches Fall Short

The paper identifies several existing approaches to improving generation quality and positions self-consistency against each:

Sample-and-rank (Adiwardana et al., 2020). This method samples multiple sequences from the decoder and then ranks them according to each sequence's log probability, selecting the top-ranked output. The limitation is that language model probabilities are poorly calibrated for correctness — the paper explicitly notes in Section 2 (footnote 2) that "the language model is not well calibrated and thus cannot distinguish well between correct solutions and wrong solutions." This means the highest-probability reasoning path is often not the correct one. Table 1 demonstrates this concretely: using "weighted avg (unnormalized)" — which scores answers by their raw sequence probability — performs essentially no better than greedy decoding on GSM8K (56.3% vs. 56.5%). The model cannot reliably identify its own correct reasoning through probability alone.

Verifier training (Cobbe et al., 2021). The GSM8K paper demonstrated that training a separate "verifier" model to re-rank generated solutions substantially improves solve rates on math problems. However, this requires thousands of training examples and a separate model trained specifically for verification. Self-consistency achieves similar or better gains (e.g., 74.4% on GSM8K vs. the verifier-augmented baseline of 55% reported in Cobbe et al., 2021, Table 2) with no training, no auxiliary model, and no additional human annotation.

Human-annotation-based re-ranking (Thoppilan et al., 2022). LaMDA used a re-ranker trained on human-annotated data to filter responses. Again, this requires substantial human effort and is task-specific. Self-consistency is entirely unsupervised.

Beam search. The paper explicitly compares against beam search in Table 6 and finds that beam search decoding significantly underperforms self-consistency, and that performance actually degrades as beam size increases (e.g., AQuA accuracy drops from 23.6% at beam size 1 to 10.2% at beam size 40). The authors attribute this to beam search yielding lower diversity in outputs (citing Li & Jurafsky, 2016) — beam search tends to produce sequences that differ only in the final few tokens, sharing the same high-probability prefix. Self-consistency's strength comes precisely from the diversity of the full reasoning paths, which beam search suppresses.

Ensemble-based approaches. The paper compares against two few-shot ensembling strategies in Table 7: (1) randomly permuting the exemplar order in the prompt 40 times and taking a majority vote over greedy-decoded answers (Zhao et al., 2021; Lu et al., 2021), and (2) manually writing 3 different sets of chain-of-thought prompts and ensembling their greedy-decoded outputs (Gao et al., 2021). On GSM8K with LaMDA-137B, prompt-order ensemble achieves only 19.2% vs. 27.7% for self-consistency; multi-prompt ensemble achieves 18.6%. The gains from these ensemble approaches are small (roughly +1–2% over the baseline) compared to the +10.6% gain from self-consistency. The paper also tests model ensembles in Appendix A.1.3 (Table 10), where combining predictions from multiple different models (e.g., LaMDA-137B + PaLM-540B) actually hurts performance because weaker models drag down the ensemble, achieving only 36.9% on GSM8K vs. 74.4% for self-consistency on PaLM-540B alone.

Chain-of-thought prompting itself (Wei et al., 2022). While chain-of-thought prompting was a breakthrough in eliciting reasoning from language models, it has a documented failure mode: Ye & Durrett (2022) showed that adding chain-of-thought can sometimes hurt performance compared to standard prompting without rationales on certain NLP tasks. The paper explicitly tests this scenario in Section 3.3 (Table 5) on tasks like ANLI, e-SNLI, and RTE, where chain-of-thought with greedy decoding underperforms standard prompting. Self-consistency not only recovers the lost performance but surpasses standard prompting, making it a "reliable way to add rationales in few-shot in-context learning" even when rationales alone are detrimental.

How This Paper Positions Itself

The paper frames self-consistency as occupying a unique and previously unexplored design point in the space of decoding strategies. It is neither an open-ended text generation method (where diversity is standard but answer aggregation is meaningless) nor a fixed-answer optimization method (where greedy decoding was standard but diversity was ignored). The paper's position is that reasoning tasks with fixed answers benefit from open-ended-style diversity in the intermediate reasoning process, followed by answer-level aggregation.

This positioning is captured in the paper's own framing (Section 2):

"Self-consistency explores an interesting space between open-ended text generation and optimal text generation with a fixed answer."

The method is also distinguished by what it does not require. The paper repeatedly emphasizes four negatives that set it apart from prior work:

No additional training or fine-tuning. The method works off-the-shelf with any pretrained language model capable of few-shot chain-of-thought prompting.
No auxiliary models. Unlike verifier-based approaches that require training a separate scoring model, self-consistency uses only the base language model.
No additional human annotation. The prompts are the same manually written chain-of-thought exemplars already used in Wei et al. (2022).
No task-specific specialization. The identical self-consistency procedure — sample reasoning paths, parse final answers, take majority vote — is applied uniformly across arithmetic, commonsense, symbolic reasoning, and standard NLP tasks without modification.

This is what the paper means by calling self-consistency a "self-ensemble" (Section 1): it ensembles the outputs of a single model sampled multiple times, rather than ensembling across different models, different prompts, or different training runs. The model essentially serves as its own committee of reasoners.

Finally, it is important to note what the paper explicitly rules out as its scope: self-consistency "can be applied only to problems where the final answer is from a fixed answer set" (Section 2). The paper acknowledges this limitation and suggests that in principle it could be extended to open-text generation if a good metric of consistency between multiple generations can be defined (e.g., whether two answers agree or contradict), but no such extension is developed. The method as presented is strictly for tasks where answers can be extracted, parsed, and compared for exact match — arithmetic answers, multiple-choice selections, yes/no decisions, and short factoid responses. This constraint is fundamental to how majority voting operates.

3. Technical Approach

3.1 Reader Orientation

The "system" here is not a trained model or a complex pipeline — it is a decoding strategy: a procedure for how to extract answers from an already-trained, frozen language model at inference time. It solves the problem that greedy decoding (taking the single most likely token at each step) sees only one reasoning path, and if that path contains a mistake, the final answer is wrong even though the model might be capable of producing the correct answer through a different chain of thought. The shape of the solution is: generate multiple diverse reasoning paths from the same model via stochastic sampling, parse the final answer from each path, and take a majority vote — letting the correct answer emerge from the consensus of different reasoning attempts rather than betting everything on a single most-likely trajectory.

3.2 Big-Picture Architecture (Diagram in Words)

The self-consistency pipeline has three stages, all operating on a frozen pretrained language model with no fine-tuning or auxiliary components:

Prompt construction: The same few-shot chain-of-thought prompt from Wei et al. (2022) is given to the model — a set of manually written exemplars showing step-by-step reasoning followed by a final answer, followed by the target question. This establishes the expected output format (reasoning path, then "The answer is X.").
Diverse sampling: Instead of greedy decoding a single output, the model's decoder is sampled multiple times (typically 40 paths) using temperature sampling with top-k truncation to introduce stochasticity. Each independent forward pass produces a different reasoning path r_i and final answer a_i.
Answer aggregation via marginalization: The final answers a_i are extracted from each sampled path using a task-dependent parser (e.g., the text following "The answer is" in arithmetic tasks, or the full answer string in multiple-choice tasks). The most frequent answer across all samples is selected as the final output via unweighted majority vote: arg max_a Σᵢ 𝟙(a_i = a).

Information flows linearly: question → prompt + language model → N sampled (reasoning path, answer) pairs → answer extraction → majority vote → final answer. There is no feedback loop and no model state carried across samples — each of the N paths is generated independently from the same prompt.

3.3 Roadmap for the Deep Dive

First, the probabilistic formulation of self-consistency — the introduction of a latent reasoning-path variable r_i and the marginalization operation that defines the method mathematically. This explains why majority voting is the correct aggregation procedure, not just an ad-hoc heuristic.
Second, the answer aggregation strategies — the paper's empirical comparison of different ways to combine sampled answers (weighted vs. unweighted, normalized vs. unnormalized, sum vs. average). This establishes why simple unweighted majority vote is both sufficient and optimal among the compared options, and what the alternatives reveal about the model's calibration.
Third, the sampling mechanism — the specific decoding parameters (temperature, top-k, and the interaction between them) and why diversity in the reasoning paths, not just the final answers, is the essential ingredient. This includes the paper's ablation studies showing robustness to different sampling strategies and the failure mode of beam search.
Fourth, the task-dependent answer extraction — how final answers are parsed from the generated text, since the mechanism relies on exact answer matching for majority voting. The parser is simple but task-specific: arithmetic tasks look for numerical output after a trigger phrase, while commonsense tasks match the full answer string.
Fifth, the self-consistency as "self-ensemble" concept — how the approach differs from model ensembles, prompt ensembles, and other aggregation methods, and why the within-model diversity of reasoning paths is the key distinguishing feature.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a decoding strategy paper whose core idea is that for reasoning tasks with fixed answers, sampling diverse reasoning paths from a single language model and aggregating their final answers via majority vote substantially outperforms selecting a single most-likely reasoning path via greedy decoding. The method introduces no new parameters, requires no training, and works with any off-the-shelf language model capable of chain-of-thought prompting.

Probabilistic Formulation: The Latent Reasoning Path Variable

The paper formalizes self-consistency by introducing a latent variable into the generation process. In standard chain-of-thought prompting, the model directly generates an answer a conditioned on the prompt and question. Self-consistency decomposes this into a two-stage process: first, generate a reasoning path r_i (a sequence of tokens representing the step-by-step reasoning), and then generate the final answer a_i conditioned on that reasoning path. The key structural assumption is that r_i → a_i — the reasoning path determines the answer, and the answer is a deterministic function of the path (the answer is the conclusion reached by the reasoning chain).

Formally, self-consistency introduces r_i as a latent variable and applies marginalization over it:

$P(a \mid \text{prompt}, \text{question}) = \sum_{r_i} P(a \mid r_i, \text{prompt}, \text{question}) \cdot P(r_i \mid \text{prompt}, \text{question})$

where a is a candidate final answer from the fixed answer set A, r_i is the i-th reasoning path sampled from the language model's decoder, P(r_i | prompt, question) is the model's probability of generating that specific reasoning path, and P(a | r_i, prompt, question) is the probability that reasoning path r_i leads to answer a.

What it computes: the total probability mass the language model assigns to answer a, summed over all possible reasoning paths that could lead to that answer. Rather than conditioning on a single reasoning path (as greedy decoding does), this formulation considers all paths the model can generate and weights them by their probability, then sums the probability of all paths that converge on the same answer.

Why this form: the marginalization directly encodes the paper's core intuition — a complex reasoning problem admits multiple valid reasoning paths that all lead to the correct answer. If the model's probability distribution over reasoning paths assigns non-trivial probability to several different correct paths (even if no single one dominates), the marginalized probability of the correct answer can be high even when the maximum-likelihood path (greedy decoding) is incorrect. The latent variable formulation makes explicit what greedy decoding implicitly does: greedy decoding conditions on only the single most probable r_i (the arg max over the latent variable), discarding all information from other reasoning paths. Self-consistency approximates the full marginalization by Monte Carlo sampling: rather than summing over all possible reasoning paths (intractable), it samples m paths from the decoder and approximates the sum.

In practice, the paper simplifies this further. Since P(a | r_i, prompt, question) is essentially 1 for the answer that r_i explicitly concludes and 0 for all other answers (the reasoning path states its conclusion unambiguously in the form "The answer is X"), the marginalization reduces to a majority vote over the sampled answers a_i:

$\text{answer} = \arg\max_{a} \sum_{i=1}^{m} \mathbb{1}(a_i = a)$

where m is the number of sampled reasoning paths (typically 40 in the experiments), a_i is the final answer extracted from the i-th sampled path, and 𝟙(a_i = a) is an indicator function that equals 1 when the sampled answer matches candidate a and 0 otherwise.

What it computes: a simple frequency count — for each unique answer that appears in any sampled path, count how many paths produced that answer, and select the answer with the highest count. This is an exact analog of majority voting: whichever answer appears most frequently across the m independent reasoning attempts is chosen as the final output.

Why this form: it is the simplest possible approximation to the full marginalization that avoids any reliance on the language model's probability calibration. Rather than using the model's own probability estimates P(r_i | prompt, question) to weight the votes (which the paper shows is unreliable because the model is poorly calibrated — see Table 1), the unweighted majority vote treats all sampled paths as equally valid draws from the model's distribution and lets the answer itself provide the signal. This is a key design choice: the paper explicitly tests weighted voting schemes (using both raw and length-normalized sequence probabilities) and finds they underperform unweighted majority vote or are statistically indistinguishable from it. The intuition is that for language models, the probability of a sequence is dominated by superficial features (token frequency, length) rather than reasoning correctness, so weighting by probability introduces noise without signal.

Answer Aggregation Strategies: Why Unweighted Majority Vote Wins

The paper empirically compares five distinct ways to aggregate answers across sampled reasoning paths (Table 1, Section 2), and the results justify the choice of simple unweighted majority voting. Each strategy represents a different answer to the question: "Given m sampled (r_i, a_i) pairs, how should we score each candidate answer a?"

Strategy 1: Weighted average (unnormalized). For each candidate answer a, compute the average of the raw (unnormalized) sequence probabilities P(r_i, a_i | prompt, question) over all paths that produced answer a. Score each answer by this average, then select the highest-scoring answer. In Table 1, this achieves 56.3% on GSM8K — essentially identical to greedy decoding at 56.5%. This failure is instructive: the model's raw sequence probability is dominated by sequence length (shorter sequences get higher probability because they involve multiplying fewer terms), so this metric preferentially selects answers from shorter reasoning paths regardless of correctness.

Strategy 2: Weighted average (normalized). Same as above, but each sequence probability is normalized by sequence length using the formula:

$P_{\text{normalized}}(r_i, a_i \mid \text{prompt}, \text{question}) = \exp\left(\frac{1}{K}\sum_{k=1}^{K} \log P(t_k \mid \text{prompt}, \text{question}, t_1, \ldots, t_{k-1})\right)$

where K is the total number of tokens in (r_i, a_i), t_k is the k-th token, and P(t_k | ...) is the model's token-level conditional probability. The expression inside the exponential is the geometric mean of the per-token probabilities — i.e., the average log-probability per token.

What it computes: for each complete sampled sequence, compute the average log-probability across all tokens (which corrects for sequence length — longer sequences are not penalized simply for having more terms), then exponentiate to convert back to a probability-like score. For each candidate answer a, average these normalized scores over all paths that produced a, and select the answer with the highest average.

Why this form: normalizing by length addresses the bias of unnormalized probabilities toward shorter sequences. However, Table 1 shows this strategy performs dramatically worse — only 22.1% on GSM8K. The paper notes (Section 2, footnote 2) that the model's normalized conditional probabilities P(r_i, a_i | prompt, question) are quite close to each other across different sampled paths — the model regards those generations as "similarly likely" regardless of their correctness. This means the language model is not well-calibrated and cannot distinguish between correct and incorrect solutions based on probability alone. Normalizing the probabilities doesn't fix the underlying calibration problem; it just removes sequence-length bias while leaving the fundamental inability to separate correct from incorrect intact.

Strategy 3: Weighted sum (unnormalized). For each candidate answer a, sum the raw (unnormalized) sequence probabilities over all paths that produced a, then select the answer with the highest sum. On GSM8K, this achieves 59.9% — a modest improvement over greedy decoding (56.5%) but far below the unweighted sum (74.4%). The sum partially corrects for the averaging problem in Strategy 1: if many paths converge on the correct answer, the sum of their probabilities can be high even if each individual probability is moderate. But the unnormalized nature still biases toward shorter sequences.

Strategy 4: Weighted sum (normalized). Sum the length-normalized probabilities (same normalization as Strategy 2) over all paths for each answer, then select the answer with the highest sum. Achieves 74.1% on GSM8K — essentially identical to the unweighted majority vote at 74.4%. The paper interprets this (Section 2) as evidence that the normalized probabilities across different paths are "quite close to each other" — when the weights are nearly uniform, the weighted sum reduces to an unweighted sum (majority vote). The weights add no information beyond the vote count.

Strategy 5: Unweighted sum (majority vote). Simply count the number of paths that produce each answer, with no probability weighting. Achieves 74.4% ± 0.1 on GSM8K. This is the method used throughout the paper's main results.

Why this strategy is chosen: it is the simplest method that matches or exceeds all weighted alternatives; it requires no access to token-level probabilities (making it compatible with API-based models where logprobs may not be available); and it is robust to the model's poor probability calibration. The paper's finding that normalized weighted sum and unweighted sum perform essentially identically is the empirical justification for dropping the weights entirely — if the weights are uninformative, using them adds computational overhead and complexity with no benefit. This also explains why sample-and-rank (which selects the single highest-probability path) underperforms dramatically compared to self-consistency (Figure 3): ranking by probability is selecting based on a nearly-random signal, while majority voting uses the signal that actually carries information — agreement across independent reasoning attempts.

The Sampling Mechanism: How Diversity Is Introduced

The diversity of reasoning paths is the essential ingredient in self-consistency. The paper's sampling procedure is designed to maximize this diversity while keeping outputs coherent enough to be parseable. The mechanism is standard stochastic decoding from the language model with two interacting controls: temperature and top-k truncation.

Temperature sampling (Ackley et al., 1985). At each token generation step, the model produces a vector of logits (raw scores) over the vocabulary. These logits are divided by a temperature parameter T before conversion to probabilities via softmax:

$P(t_k \mid \text{context}) = \frac{\exp(z_k / T)}{\sum_{j} \exp(z_j / T)}$

where z_k is the logit for token k, and T is the temperature.

What it computes: when T = 1, this is the standard softmax. When T > 1, the distribution is flattened — lower-probability tokens become more likely, increasing diversity. When T < 1, the distribution is sharpened — high-probability tokens become even more dominant, reducing diversity toward greedy-like behavior. At T → 0, this converges to greedy decoding (always selecting the arg max token). At T → ∞, this converges to uniform sampling over the vocabulary (producing gibberish).

Why this form: temperature provides a continuous knob to trade off between diversity and coherence. The paper uses T = 0.5 for UL2-20B and LaMDA-137B, and T = 0.7 for PaLM-540B and GPT-3. The higher temperature for larger models reflects that they are more confident and accurate per-sample, so more aggressive diversity (higher T) can be used without as much risk of incoherence.

Top-k sampling (Fan et al., 2018; Holtzman et al., 2018). After computing the temperature-scaled probabilities over the full vocabulary, only the k tokens with the highest probability are retained; all other tokens have their probability set to zero, and the probabilities of the remaining k tokens are renormalized to sum to 1. Sampling then proceeds over this truncated distribution.

What it computes: a hard filter on the vocabulary that eliminates the long tail of low-probability tokens. Without top-k, even with temperature scaling, the model would occasionally sample very low-probability tokens (e.g., rare words, typos, nonsensical completions) that could derail the reasoning path. Top-k prevents this by restricting the sampling to only the k most plausible next tokens.

Why this form: it is a safety mechanism that prevents the model from falling off a "quality cliff." The combination of temperature and top-k works as follows: temperature widens or narrows the probability gap between likely and unlikely tokens, while top-k hard-clips the tail. The paper uses k = 40 for all models except GPT-3 (which uses no top-k truncation, relying on temperature alone). In the ablation study (Figure 4, left), the paper shows robustness: varying T from 0.3 to 0.7, k from 20 to 40, and even replacing top-k with nucleus sampling (p = 0.9 or p = 0.95) produces very similar self-consistency curves on GSM8K. The exact sampling parameters are not critical — the method works as long as sufficient diversity is introduced to generate different reasoning paths.

Why sampling beats beam search. Table 6 provides a crucial ablation: applying self-consistency using beam search instead of sampling (i.e., taking the top m beams from beam search decoding and voting on their answers) performs worse than both greedy decoding and sampling-based self-consistency. On AQuA with 40 beams, "Self-consistency using beam search" achieves 24.2% vs. 26.9% for "Self-consistency using sampling." On MultiArith, it is 10.8% vs. 14.7%. The reason is that beam search produces low diversity — beams share the same high-probability prefix and differ only in the final few tokens (Li & Jurafsky, 2016). Self-consistency requires diversity in the full reasoning path — different approaches to the problem, different intermediate calculations, different logical structures. Beam search's output is diverse only in superficial ways, so the majority vote sees effectively the same answer multiple times with minor variations, defeating the purpose of sampling multiple paths.

Independence of samples. Each of the m sampled reasoning paths is generated independently from the same prompt. The model has no memory of previous samples (each forward pass starts fresh). This independence is important because it means the samples are uncorrelated draws from the model's distribution over reasoning paths, and the law of large numbers applies: as m increases, the frequency of each answer converges to its true probability under the model's sampling distribution. If the correct answer has higher probability mass (summed over all correct reasoning paths) than any individual wrong answer, increasing m will eventually make the correct answer the majority. This is why Figure 2 shows accuracy monotonically increasing with the number of sampled paths — more samples means a more reliable estimate of which answer has the most probability mass.

The number of samples m. The paper uses m = 40 as the default in all main experiments (Section 3.1: "we sampled 40 outputs independently from the decoder in each run"). The accuracy curves in Figure 2 show a characteristic shape: steep improvement from 1 to 10–20 paths, followed by gradual saturation from 20 to 40. The paper notes in Section 5 that practitioners can use a smaller number (e.g., 5 or 10) as a starting point to "realize most of the gains while not incurring too much cost." This is a practical concession to the method's main limitation: m = 40 costs 40× the inference compute of greedy decoding.

Task-Dependent Answer Extraction

For majority voting to work, the final answer must be extracted from each sampled reasoning path in a consistent, comparable format. The paper uses a simple, task-specific parser based on the expected output format established by the few-shot exemplars.

The expected output format. All prompts used in the paper (Appendix A.3, Tables 14–21) follow a consistent template: the exemplars show reasoning steps followed by a sentence of the form "The answer is X." where X is the final answer. This format trains the model (through in-context learning) to produce outputs with the same structure. The parser exploits this structure.

Arithmetic reasoning tasks. For GSM8K, MultiArith, AddSub, ASDiv, SVAMP, and AQuA, the parser extracts the first numerical value that appears after the model generates the trigger phrase "The answer is" (Section 2, footnote 1). For multiple-choice tasks like AQuA, the answer is typically a letter choice like "(a)", "(b)", etc., and the parser extracts this choice label. The arithmetic answers are numerical and can be compared for exact equality.

Commonsense reasoning tasks. For CommonsenseQA, StrategyQA, and ARC, the parser extracts the full string answer after "The answer is" (Section 2, footnote 1). CommonsenseQA and ARC are multiple-choice, so the answer is a choice label like "(b)" or "forest." StrategyQA produces yes/no answers. The full answer string is used for exact-match comparison in majority voting.

Parsing failure modes. The paper acknowledges that most generated outputs have a consistent format ("{Reasoning paths}. The answer is X.") if the model is prompted in this format, but does not discuss handling of unparseable outputs in detail. The implicit assumption is that with sufficient high-quality exemplars, the model reliably follows the expected format, and unparseable outputs (which would be discarded or counted as abstentions) are rare. The standard deviations reported in Table 2 (≤0.5 across all tasks, omitted from the table) suggest that the variance introduced by parsing inconsistencies is negligible relative to the sampling variance.

Why exact-match is sufficient. Unlike open-ended text generation where semantic equivalence matters, reasoning tasks have ground-truth answers that can be matched exactly: a math answer is a number, a multiple-choice answer is a predetermined label, and a yes/no answer is a binary string. The parser does not need to handle synonyms, paraphrases, or approximate matches — "18" and "$18" might need normalization (the paper implies numerical parsing handles this), but "18" and "eighteen" would be a mismatch that the method cannot resolve. This is a limitation of the approach: it relies on the model producing answers in a consistent, parseable format, and inconsistencies in formatting could fragment the vote (the same correct answer expressed two different ways would be counted as two different answers and potentially lose the majority).

Self-Consistency as "Self-Ensemble"

The paper explicitly positions self-consistency as distinct from traditional ensembling approaches (Section 1, Section 3.4). This distinction is important because it clarifies what is being aggregated and why the method works.

What self-consistency is NOT:

Model ensemble: multiple different trained models whose outputs are aggregated. Table 10 (Appendix A.1.3) shows that directly ensembling greedy-decoded outputs from multiple models (e.g., LaMDA-137B + PaLM-540B + GPT-3) achieves only 33.3% on GSM8K — far worse than self-consistency on a single model (74.4%). The reason is that weaker models drag down the performance of stronger ones in a naive majority vote. Self-consistency avoids this by using only a single model's outputs, so there is no quality disparity across "committee members" — all reasoning paths come from the same model at the same capacity level.
Prompt ensemble: the same model queried with different prompts, with outputs aggregated. Table 7 shows that ensembling across 3 different manually written prompt sets achieves 18.6% on GSM8K with LaMDA-137B, while ensembling across 40 random permutations of exemplar order achieves 19.2%. Both are marginal improvements over single-prompt greedy decoding (17.1%) and dramatically underperform self-consistency with 40 sampled paths (27.7%). Prompt ensembling only varies the surface form of the instruction — the model still takes a single reasoning path per prompt via greedy decoding. Self-consistency varies the reasoning paths themselves, which is a much richer source of diversity.
Sample-and-rank: sampling multiple sequences and selecting the one with the highest sequence probability (Adiwardana et al., 2020). Figure 3 shows this underperforms self-consistency substantially on GSM8K, MultiArith, and ARC-challenge — the probability-sorted curves are only slightly above the greedy decoding baseline, while the self-consistency curve is dramatically higher. The reason is that language model probability is a poor proxy for reasoning correctness (as discussed in the aggregation strategies section above).

What self-consistency IS: a "self-ensemble" that aggregates multiple reasoning attempts from a single model with a single prompt, where the only source of variation is the stochasticity in the decoding process (temperature sampling + top-k truncation). The model serves as its own committee — each sampled path is a different "expert" reasoning about the same problem, and they vote on the final answer. The key property that makes this work is that the variation introduced by sampling is variation in the reasoning strategy (different intermediate steps, different arithmetic decompositions, different logical chains), not merely variation in surface-level wording. Table 4 provides qualitative evidence: on a GSM8K problem, greedy decoding computes "60 - 20 = 40" (forgetting the second stop), while Sampled Path 1 computes "60 - 20 - 15 = 25" (subtracting both stops from the total) and Sampled Path 2 computes "60 - 15 = 45, then 45 - 20 = 25" (locating the second stop first, then finding the distance between stops). These are genuinely different reasoning strategies — they decompose the problem differently and use different intermediate values — yet they converge on the same correct answer (25). Wrong answers, by contrast, tend to be idiosyncratic: different mistakes produce different wrong answers, so no single wrong answer accumulates as many votes as the correct one.

Why this is called "consistency" rather than "majority": the paper emphasizes that the method works because of the agreement property — correct reasoning paths, even when diverse, tend to agree on the final answer, while incorrect paths do not agree with each other. The name "self-consistency" captures this: the model is consistent with itself across different reasoning attempts when the answer is correct, and inconsistent when it is wrong. Figure 5 provides empirical support: the consistency of the model's outputs (the percentage of sampled paths that agree with the final majority-vote answer) is correlated with accuracy — when the model is highly consistent (most paths agree), it tends to be correct; when consistency is low, accuracy is also low. This means self-consistency also functions as a form of uncertainty estimation: low consistency signals that the model is uncertain and likely wrong, providing a built-in confidence measure without any additional mechanism.

4. Key Insights and Innovations

Innovation 1: Reasoning Diversity as a Resource, Not Noise — Reframing the Role of Stochastic Decoding for Fixed-Answer Tasks

The paper's most fundamental conceptual move is to challenge a deeply ingrained assumption about decoding for reasoning tasks: that when the correct answer is fixed and unique, diversity in the decoding process is at best irrelevant and at worst harmful. Prior to this work, the dominant practice for tasks like math word problems and logical reasoning was greedy decoding — select the single most probable token at each step, producing one deterministic reasoning path (Wei et al., 2022; Chowdhery et al., 2022). The intuition behind this choice was straightforward: if the right answer is predetermined, why introduce randomness that might lead the model astray? Sampling was reserved for open-ended generation where multiple outputs are equally acceptable (Radford et al., 2019; Brown et al., 2020).

Self-consistency inverts this logic entirely. The paper's core insight is that diversity in the reasoning process is orthogonal to uniqueness of the final answer. A complex reasoning problem "typically admits multiple different ways of thinking leading to its unique correct answer" (Section 1), and the model's probability distribution over reasoning paths reflects this: there may be several distinct chains of thought that all lead to the same correct conclusion, with no single path necessarily dominating the probability mass. Greedy decoding samples only the mode of this distribution — the single most probable path — and if that path happens to be wrong (due to a local arithmetic error, a misinterpreted premise, or a flawed logical step), the entire reasoning attempt fails, even though the model possesses other, lower-probability-but-correct ways of solving the problem.

This reframing is significant beyond the performance gains because it changes what "diversity" means in the context of language model reasoning. The paper shows that temperature sampling + top-k truncation produces variation not just in surface-level phrasing but in the structure of the reasoning itself — different decompositions of arithmetic, different ordering of logical steps, different intermediate calculations (Table 4, Tables 12–13). This is fundamentally different from typical "diverse decoding" work in NLP (Li & Jurafsky, 2016; Vijayakumar et al., 2018), which treats diversity as a property of the output strings (avoiding repetition, producing lexically varied text). Here, diversity is a property of the inferential strategy, and it is valuable precisely because the correct answer is the attractor that multiple distinct strategies converge upon, while wrong answers are idiosyncratic and scattered.

The paper provides empirical evidence for this reframing through a negative result that is as important as the positive ones: beam search, which explicitly optimizes for high-probability sequences, performs worse than stochastic sampling (Table 6). On AQuA, beam search accuracy drops from 23.6% (beam size 1) to 10.2% (beam size 40) — more computation produces worse results. The reason, the authors argue, is that beam search suppresses diversity: beams share the same high-probability prefix and differ only superficially near the end, so the "multiple paths" are effectively the same path with minor variations. This diagnostic finding — that greedy-like optimization of probability hurts reasoning while stochastic diversity helps — is the paper's strongest argument that the field's prior default (greedy decoding for fixed-answer tasks) was not just suboptimal but actively harmful relative to what models can achieve.

This is a fundamental reframing, not an incremental improvement. It establishes a new design principle: for reasoning tasks, decode to maximize the probability mass of the correct answer summed over all reasoning paths, not the probability of any single path. This principle does not require new models, new training data, or new architectures — it is purely a decoding-level insight — but it changes how practitioners should think about extracting reasoning capability from language models.

Innovation 2: Majority Voting as Marginalization — Providing a Probabilistic Foundation for Answer Aggregation

At first glance, taking a majority vote over sampled answers seems like a simple, almost trivial heuristic — ensemble methods and voting schemes are common throughout machine learning. What makes the paper's treatment distinctive is not the voting mechanism itself but the probabilistic framing that justifies it as an approximation to full marginalization over reasoning paths, and the empirical analysis showing why the unweighted version is not just simpler but better than probability-weighted alternatives.

The paper formalizes self-consistency by introducing a latent variable r_i — the reasoning path — into the generation process and then marginalizing over it: the total probability of an answer a is the sum of the probabilities of all reasoning paths that lead to a. In the limit of infinite samples from the model's decoder, an unweighted majority vote over the sampled answers converges to selecting the answer that has the highest total probability mass under this marginalized distribution. This is not an ad-hoc trick; it is a principled Monte Carlo approximation to a well-defined probabilistic quantity.

Where the paper goes beyond this standard formulation is in its empirical critique of model probabilities as weights. Table 1 systematically compares five aggregation strategies: unweighted vs. weighted (by sequence probability), unnormalized vs. normalized (by sequence length), and sum vs. average. The results are striking and counterintuitive:

Unweighted majority vote achieves 74.4% on GSM8K.
Weighted sum (normalized) achieves 74.1% — statistically indistinguishable.**
Weighted average (normalized) achieves only 22.1% — dramatically worse than greedy decoding.**

The paper's diagnosis is that for large language models, the normalized conditional probabilities of different sampled reasoning paths are "quite close to each other" — the model "regards those generations as similarly likely" regardless of whether the reasoning is correct or incorrect (Section 2, footnote 2). This means the model's own probability estimates are uninformative for distinguishing correct from incorrect reasoning. Weighting by these probabilities adds noise without signal; in the case of weighted averaging, it actively hurts by dividing the vote count for popular answers by the number of paths, punishing answers that the model samples frequently.

This finding is a diagnostic contribution of significant practical importance. It means that sample-and-rank (Adiwardana et al., 2020) — the standard approach of generating multiple sequences and selecting the highest-probability one — is using a nearly-random signal for selection. Figure 3 confirms this: sample-and-rank barely improves over greedy decoding on GSM8K, MultiArith, and ARC-challenge, while self-consistency with the same number of samples achieves dramatic gains. The model cannot reliably recognize its own correct reasoning through probability; it can only produce correct answers through the statistical consensus of multiple independent attempts.

This insight also explains why prior work on verifier training (Cobbe et al., 2021) and human-annotation-based re-ranking (Thoppilan et al., 2022) was necessary: those approaches train separate models precisely because the base language model's own probability estimates are too poorly calibrated to serve as a reliability signal. Self-consistency routes around this calibration problem entirely by discarding the model's probability weights and using only the answer frequencies. The contribution is thus not just a new aggregation method but a diagnosis of a specific failure mode in language model decoding — probability miscalibration for reasoning correctness — and a solution that side-steps rather than fixes the problem.

This is a conceptual advance with practical implications: it establishes that for reasoning tasks, the signal for correctness lies in agreement across independent samples (consistency), not in the model's confidence scores (probability). This reframing anticipates later work on uncertainty estimation and confidence calibration in language models, and the paper explicitly demonstrates this connection: Figure 5 shows that consistency (the percentage of paths agreeing with the majority answer) is correlated with accuracy, providing a built-in confidence measure. When the model is inconsistent (many different answers with no clear majority), it tends to be wrong — the model can "know when it doesn't know" through its own sampling distribution.

Innovation 3: Self-Consistency as Robustness Across Scale, Prompts, and Task Types — Establishing Generality Without Specialization

A persistent challenge in few-shot prompting research is that methods often work well on specific model-task combinations but fail to generalize — a prompt that boosts performance on one model may hurt another, and gains on one benchmark may not transfer. The paper's third key contribution is the empirical demonstration that self-consistency provides consistent, substantial gains across a remarkably broad range of conditions without any task-specific tuning, establishing it as a robust general-purpose decoding strategy rather than a brittle trick.

The evidence for this generality is extensive and systematically presented:

Across model scales (four orders of magnitude). Figure 4 (right) shows self-consistency improving GSM8K accuracy across the full LaMDA model series from 1B to 137B parameters, with gains growing larger as model scale increases. On UL2-20B, self-consistency provides +3–7% across arithmetic tasks; on PaLM-540B, it provides +2–18% (Table 2). The method works even when the base model is weak (UL2-20B achieves only 4.1% greedy on GSM8K; self-consistency improves this to 7.3%) and when it is already strong (PaLM-540B achieves 56.5% greedy on GSM8K; self-consistency pushes this to 74.4%). There is no scale at which the method stops working or reverses.

Across model architectures. The four tested models span fundamentally different architectures and training paradigms: UL2-20B is an encoder-decoder trained with a mixture of denoising objectives; LaMDA-137B is a decoder-only model trained on dialog and web data; PaLM-540B is a decoder-only model trained on a high-quality filtered corpus with code; GPT-3 (Codex) is a decoder-only model fine-tuned on code. Self-consistency improves all of them. This is evidence that the method captures something fundamental about how language models distribute probability over reasoning paths, not something specific to a particular training recipe.

Across task types. The paper tests arithmetic reasoning (6 datasets), commonsense reasoning (3 datasets), symbolic reasoning (2 tasks), and standard NLP tasks where chain-of-thought sometimes hurts (5 tasks: ANLI, e-SNLI, RTE, BoolQ, HotpotQA). Self-consistency improves performance on every single one (Tables 2, 3, 5). On tasks where chain-of-thought prompting with greedy decoding underperforms standard prompting (ANLI-R1 drops from 69.1% to 68.8%; e-SNLI drops from 85.8% to 81.0%; RTE drops from 84.8% to 79.1%), self-consistency not only recovers the loss but surpasses the standard prompting baseline (78.5%, 88.4%, 86.3% respectively). This is a crucial robustness result: self-consistency makes chain-of-thought prompting safe to use as a default strategy, even on tasks where rationales might otherwise degrade performance.

Across sampling parameters. Figure 4 (left) and Figure 6 (Appendix A.1.1) show ablation across temperature (0.3, 0.5, 0.7), top-k (20, 40, no truncation), and nucleus sampling (p = 0.9, 0.95). The self-consistency accuracy curves are nearly identical across all settings. This is practically important: it means practitioners do not need to carefully tune sampling hyperparameters per task or per model — any reasonable stochastic decoding strategy works.

Across prompt variations. Table 9 (Appendix A.1.2) tests three different manually written sets of chain-of-thought prompts on PaLM-540B. Greedy decoding varies from 54.0% to 56.5% across prompt sets; self-consistency achieves 70.4% to 74.4%, with the gain over greedy decoding remaining stable at +16.4 to +17.9 percentage points. The method is robust to prompt engineering choices.

Under imperfect prompts and zero-shot conditions. Table 8 demonstrates that self-consistency works even when the few-shot exemplars contain deliberately incorrect reasoning (numbers swapped to wrong values while keeping the final answer correct) — performance drops with greedy decoding (17.1% → 14.9%) but self-consistency recovers to 23.4%. It also works with equation-only reasoning paths (no natural language) and with zero-shot chain-of-thought (Kojima et al., 2022), boosting zero-shot CoT from 43.0% to 69.2% on GSM8K.

This breadth of evidence establishes self-consistency as a general principle rather than a task-specific technique. The contribution is not just "a method that works on GSM8K" but the empirical finding that sampling diversity + answer aggregation is a universally applicable strategy for improving LLM reasoning, independent of model, task, prompt, and sampling parameters. This is an empirical contribution of substantial practical significance: it gives practitioners a single, simple, off-the-shelf decoding strategy that can be applied to any reasoning task with a fixed answer set and a chain-of-thought prompt, with confidence that it will improve performance without requiring task-specific tuning.

Innovation 4: The Consistency-Accuracy Correlation — Self-Consistency as an Uncertainty Estimation Mechanism

The paper's final distinctive contribution is the observation that self-consistency provides a built-in, zero-cost uncertainty estimate through the agreement rate among sampled reasoning paths — a finding that connects the method to the broader challenge of making language models "know when they don't know."

Figure 5 plots the relationship between consistency (the percentage of 40 sampled reasoning paths that agree with the final majority-vote answer) and accuracy on GSM8K with PaLM-540B. The result is a clear positive correlation: when consistency is high (most paths agree), the model is usually correct; when consistency is low (paths scatter across many different answers), the model is usually wrong. The paper states this plainly: "one can use low consistency as an indicator that the model has low confidence; i.e., self-consistency confers some ability for the model to 'know when it doesn't know'" (Section 3.5).

What makes this insight significant beyond the obvious is that it emerges from the same mechanism that produces the accuracy gains — no additional computation, no separate calibration model, no held-out data is required. The consistency score is a byproduct of the sampling-and-voting procedure: after counting votes for each answer, the fraction of paths supporting the winning answer is immediately available as a confidence score. This contrasts with prior approaches to uncertainty estimation in language models, which typically require either: (a) access to token-level probabilities and careful calibration (Guo et al., 2017; Jiang et al., 2021), which the paper has already shown are unreliable for reasoning correctness (Table 1); (b) training a separate confidence model; or (c) using model ensembles. Self-consistency provides uncertainty estimates using only the model's own sampling distribution, with no additional training or infrastructure.

This finding has implications beyond the scope of the paper. It suggests that consistency across samples is a more reliable signal of model confidence than the model's own probability estimates for tasks requiring multi-step reasoning. This aligns with the human cognitive science literature the paper cites (Stanovich & West, 2000): humans use agreement across different lines of reasoning as a confidence heuristic ("if I can think of multiple ways to arrive at this answer, I'm probably right"). The paper shows that this same heuristic emerges naturally from language model sampling distributions, without being explicitly programmed.

The practical value is clear: in deployment, self-consistency can simultaneously provide an answer and a confidence estimate that can be used for downstream decision-making — flagging low-confidence predictions for human review, abstaining when consistency is below a threshold, or adaptively allocating more compute (more samples) when initial consistency is low. The paper does not explore these applications in depth, but the correlation in Figure 5 establishes the foundation for them.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a broad collection of arithmetic, commonsense, and symbolic reasoning benchmarks. Arithmetic tasks include the Math Word Problem Repository (AddSub, MultiArith, ASDiv; Koncel-Kedziorski et al., 2016), AQUA-RAT (Ling et al., 2017), GSM8K (Cobbe et al., 2021), and SVAMP (Patel et al., 2021). Commonsense tasks include CommonsenseQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021), and ARC (Clark et al., 2018) with both Easy and Challenge splits. Symbolic reasoning uses two tasks from Wei et al. (2022): last-letter concatenation and coinflip, tested in an out-of-distribution setting (4-letter/4-flip examples given 2-letter/2-flip exemplars). For the "chain-of-thought hurts" analysis in Section 3.3, the paper adds standard NLP tasks: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), e-SNLI (Camburu et al., 2018), ANLI (Nie et al., 2020), and RTE (Dagan et al., 2005). By default, test splits are used when labels are available; CommonsenseQA uses the dev split, and StrategyQA uses the question-only set from BIG-bench collaboration (2021). No dataset-specific training or fine-tuning is performed — all evaluations use few-shot in-context learning only.
Base model(s). Four transformer-based language models spanning two orders of magnitude in scale are evaluated: UL2-20B (Tay et al., 2022) — an open-source encoder-decoder trained on a mixture of denoisers; GPT-3-175B (Brown et al., 2020) — accessed via the public Codex API in two versions (code-davinci-001 and code-davinci-002); LaMDA-137B (Thoppilan et al., 2022) — a dense decoder-only model trained on web, dialog, and Wikipedia data; and PaLM-540B (Chowdhery et al., 2022) — a dense decoder-only model trained on a high-quality filtered corpus of 780B tokens including webpages, books, Wikipedia, news, code, and social media. The choice of four models with different architectures, scales, and training data is deliberate: it tests whether self-consistency's benefits are an artifact of any specific model family or scale. The paper argues that PaLM-540B and GPT-3 are "representative of the capabilities of many contemporary LLMs" (Section 4), though this claim is discussed in the Critical Assessment.
Metrics. The sole metric is accuracy — the fraction of questions for which the extracted final answer matches the ground-truth answer exactly. For arithmetic tasks, answers are parsed as the first numerical value after "The answer is"; for multiple-choice tasks, the choice label is extracted; for yes/no tasks, the full answer string is compared. The accuracy of self-consistency is reported as the mean and standard deviation over 10 independent runs, where each run samples m reasoning paths independently and applies majority voting. Standard deviations are consistently small (typically ≤0.5%, often omitted from tables in the main text), indicating that the sampling variance at m = 40 is negligible relative to the effect sizes being measured. The greedy decoding baseline is deterministic (no sampling), so it has no variance.
Baselines. The primary comparison is against chain-of-thought prompting with greedy decoding (Wei et al., 2022), referred to as "CoT-prompting" throughout the paper. This is the same model, same prompts, same task — the only difference is the decoding strategy (greedy single-path vs. sampled multi-path with majority vote). Additional baselines include: sample-and-rank (Adiwardana et al., 2020) — sample m sequences and select the one with the highest sequence log-probability, evaluated on GPT-3 code-davinci-001 in Figure 3; beam search decoding — standard beam search with varying beam widths, compared in Table 6 on UL2-20B; prompt-order ensemble (Zhao et al., 2021; Lu et al., 2021) — randomly permute the few-shot exemplar order 40 times and take a majority vote over greedy-decoded answers, evaluated in Table 7 on LaMDA-137B; multi-prompt ensemble (Gao et al., 2021) — manually write 3 different sets of chain-of-thought prompts and majority-vote over greedy-decoded answers, also in Table 7; and standard prompting without chain-of-thought (Brown et al., 2020) — used in the Section 3.3 analysis where chain-of-thought sometimes hurts. The paper also reports previous state-of-the-art results from the literature (Tables 2–3), which typically involve task-specific training, fine-tuning with thousands of examples, or auxiliary verifier models.
Generation budget / compute accounting. The paper measures computation in terms of the number of sampled reasoning paths, which is the unit of comparison for all methods. For self-consistency, m paths are sampled independently from the decoder (default m = 40). For sample-and-rank, the same m paths are sampled but only the top-ranked one is used. For beam search, the budget is the number of beams. For prompt ensembles, the budget is the number of different prompts. This is a fair comparison scheme: all methods are evaluated with the same number of forward passes through the language model. However, the paper does not measure wall-clock time or FLOPs, and does not account for the fact that self-consistency's m = 40 samples cost exactly 40× the inference compute of greedy decoding at m = 1. This cost is discussed as the method's primary limitation in Section 5.
Cross-validation / statistical protocol. There is no train/validation/test split or cross-validation procedure in this paper — all results are on fixed test sets using off-the-shelf models with no fine-tuning. The only statistical protocol is the reporting of mean and standard deviation over 10 independent runs of self-consistency (each run samples m = 40 independent reasoning paths). The standard deviations are small (≤0.5% in all cases, per Section 3.2), so the paper often omits them from tables for readability and reports them only in figures (Figure 2, Figures 7–8 in the appendix). For the GPT-3 experiments, sampling parameters are reported to aid reproducibility (Section 3.1: T = 0.7, no top-k, 128 max tokens, no frequency or presence penalty). For UL2-20B, LaMDA-137B, and PaLM-540B, the sampling parameters are specified per-model (T = 0.5, k = 40 for UL2 and LaMDA; T = 0.7, k = 40 for PaLM). The prompts used are the exact same chain-of-thought exemplars from Wei et al. (2022), reproduced in full in Appendix A.3, which aids reproducibility for the two public models (UL2-20B and GPT-3) and partially for the two non-public models (LaMDA-137B and PaLM-540B).

Main Quantitative Results

Arithmetic Reasoning

Table 2 reports the complete arithmetic reasoning results across all four models and six datasets. The headline numbers on PaLM-540B — the largest model tested — are: GSM8K improves from 56.5% (greedy) to 74.4% (+17.9 absolute percentage points); AQuA from 35.8% to 48.3% (+12.5); SVAMP from 79.0% to 86.6% (+7.6); ASDiv from 74.0% to 81.9% (+7.9); MultiArith from 94.7% to 99.3% (+4.6); and AddSub from 91.9% to 93.7% (+1.8). These gains establish new state-of-the-art results on 5 of the 6 arithmetic tasks, surpassing prior methods that required task-specific training, fine-tuning with thousands of examples, or auxiliary verifier models (e.g., the prior GSM8K SoTA was 55% using GPT-3 175B fine-tuned with 7.5k examples plus a separately trained 175B verifier; Cobbe et al., 2021). Self-consistency achieves 74.4% with no fine-tuning, no auxiliary model, and no additional training data — purely through a change in the decoding strategy.

A striking pattern in Table 2 is that the absolute gains from self-consistency increase with model scale. On UL2-20B, the improvements are modest: +3.2% on GSM8K (4.1% → 7.3%), +3.3% on AQuA (23.6% → 26.9%), +6.8% on SVAMP (12.6% → 19.4%). On LaMDA-137B, the gains are substantially larger: +10.6% on GSM8K (17.1% → 27.7%), +9.1% on AQuA (17.7% → 26.8%), +14.4% on SVAMP (38.9% → 53.3%). On PaLM-540B, the gains become dramatic: +17.9% on GSM8K, +12.5% on AQuA, +7.6% on SVAMP. On GPT-3 code-davinci-002, the pattern continues: +17.9% on GSM8K (60.1% → 78.0%), +12.2% on AQuA (39.8% → 52.0%), +11.0% on SVAMP (75.8% → 86.8%). This scale-dependence is intuitive: larger models have a richer distribution over reasoning paths — they can generate many different correct ways to solve a problem, and self-consistency exploits this diversity. Smaller models have fewer correct paths in their distribution, so sampling multiple paths yields less diversity and smaller gains.

On the GPT-3 models, comparing code-davinci-001 (the earlier Codex version) with code-davinci-002 (the later version) reveals an important nuance: the absolute greedy decoding performance jumps substantially between versions (e.g., GSM8K: 14.6% → 60.1%), but the relative gain from self-consistency remains large in both cases (+8.8% on the weaker model, +17.9% on the stronger model). This suggests self-consistency is not merely compensating for a weak base model — it amplifies the capabilities of already-strong models.

Figure 2 (main text, shown for LaMDA-137B on four representative tasks) plots accuracy as a function of the number of sampled reasoning paths m from 1 to 40. The curves have a characteristic shape: steep improvement from m = 1 to m ≈ 10–20, followed by gradual saturation from m = 20 to m = 40. On MultiArith, accuracy rises from roughly 55% at m = 1 to approximately 75% at m = 40, with most gains captured by m = 20. On SVAMP, the rise is from roughly 39% to 53%. On CommonsenseQA, from roughly 58% to 62%. On ARC-challenge, from roughly 51% to 60%. The curves are consistently above the greedy decoding baseline (shown as a flat orange line) at all m > 1, and they monotonically increase — there is no point at which more samples hurt. The shaded regions show tight standard deviations, confirming the reliability of the trend. Figures 7 and 8 in Appendix A.1.1 show similar curves for all arithmetic and commonsense tasks on LaMDA-137B and PaLM-540B respectively, with the same monotonic improvement pattern.

Commonsense and Symbolic Reasoning

Table 3 reports results on commonsense and symbolic reasoning. On PaLM-540B, self-consistency improves StrategyQA from 75.3% to 81.6% (+6.3), ARC-challenge from 85.2% to 88.7% (+3.5), and provides smaller gains on tasks where greedy decoding is already near ceiling: CommonsenseQA (79.0% → 80.7%, +1.7), ARC-easy (95.3% → 96.4%, +1.1). On GPT-3 code-davinci-002, the pattern is similar: StrategyQA improves from 73.4% to 79.8% (+6.4), ARC-challenge from 83.6% to 87.5% (+3.9). The gains on commonsense reasoning are generally smaller in absolute terms than on arithmetic reasoning, which the paper does not explicitly analyze but which likely reflects the different nature of the tasks: arithmetic problems have a large space of possible computational decompositions, while commonsense multiple-choice questions have fewer genuinely distinct reasoning paths to the correct answer.

For the symbolic reasoning tasks tested in the out-of-distribution setting (4-letter concatenation and 4-coinflip, with only 2-letter/2-flip exemplars), Table 3 shows that self-consistency provides meaningful gains on PaLM-540B: letter concatenation improves from 65.8% to 70.8% (+5.0), coinflip from 88.2% to 91.2% (+3.0). On GPT-3 code-davinci-002, coinflip is already at 99.0% greedy and improves marginally to 99.5% (+0.5); letter concatenation improves from 70.4% to 73.4% (+3.0). The out-of-distribution setting is particularly challenging because the model must generalize to longer reasoning chains than it saw in the exemplars — and self-consistency still helps, suggesting the diversity of sampled paths includes some that successfully generalize.

Self-Consistency When Chain-of-Thought Hurts

Section 3.3 and Table 5 address a critical robustness question: Ye & Durrett (2022) showed that adding chain-of-thought reasoning can sometimes degrade performance compared to standard prompting without rationales on certain NLP tasks. If self-consistency amplifies chain-of-thought, does it amplify the failure mode as well, or does it recover performance?

Table 5 reports results on PaLM-540B across five tasks. For ANLI (three difficulty levels: R1, R2, R3), chain-of-thought with greedy decoding slightly underperforms standard prompting on R1 (68.8% vs. 69.1%) while improving on R2 (58.9% vs. 55.8%) and R3 (60.6% vs. 55.8%). Self-consistency dramatically outperforms both on all three: 78.5%, 64.5%, 63.4%. On e-SNLI, chain-of-thought reduces accuracy from 85.8% (standard) to 81.0% (a -4.8% degradation), but self-consistency achieves 88.4% — recovering the entire loss and surpassing standard prompting by +2.6%. On RTE, chain-of-thought drops from 84.8% to 79.1% (-5.7%), and self-consistency achieves 86.3% (+1.5 over standard). On BoolQ and HotpotQA, chain-of-thought already helps, and self-consistency provides further gains (74.2% → 78.4% and 28.9/39.8 EM/F1 → 33.8/44.6).

The key insight from Table 5 is that self-consistency makes chain-of-thought prompting a safe default strategy: even on tasks where adding rationales hurts individual greedy-decoded answers, the aggregate over diverse reasoning paths is more robust and reliably outperforms both standard prompting and greedy chain-of-thought. This is a practically important finding — it means practitioners can apply chain-of-thought + self-consistency uniformly across tasks without worrying about task-specific degradation.

Comparison to Sample-and-Rank

Figure 3 directly compares self-consistency with sample-and-rank (Adiwardana et al., 2020) on GPT-3 code-davinci-001 across three tasks: GSM8K, MultiArith, and ARC-challenge. Both methods use the same number of sampled sequences (1 to 40), but sample-and-rank selects the single sequence with the highest log-probability, while self-consistency applies majority voting over all samples.

The results are decisive. On GSM8K, sample-and-rank improves from roughly 14% (greedy) to roughly 17% at m = 40 — a +3% gain. Self-consistency reaches roughly 23% at m = 40 — a +9% gain, or 3× the improvement of sample-and-rank. On MultiArith, sample-and-rank reaches roughly 65% vs. roughly 82% for self-consistency. On ARC-challenge, sample-and-rank reaches roughly 47% vs. roughly 53%. In all cases, the sample-and-rank curve is closer to the greedy decoding baseline than to the self-consistency curve, demonstrating that sequence probability is a weak signal for reasoning correctness compared to answer agreement across diverse paths.

This comparison is central to the paper's argument: it directly tests the hypothesis that the language model's probability estimates are unreliable for identifying correct reasoning (Section 2, footnote 2) and that majority voting is a superior aggregation strategy. The results support both claims.

Comparison to Beam Search

Table 6 compares self-consistency against beam search decoding on UL2-20B, with both methods evaluated at the same beam/path counts (1, 5, 10, 20, 40). Three variants are tested: "Beam search decoding (top beam)" — standard beam search, taking the single highest-scoring beam; "Self-consistency using beam search" — taking all m beams and majority-voting over their answers; and "Self-consistency using sampling" — the standard self-consistency approach.

On AQuA, standard beam search performance degrades as beam size increases: 23.6% at beam size 1, dropping to 10.2% at beam size 40. This negative scaling is a critical diagnostic finding: beam search actively hurts reasoning performance as it explores more beams, because the additional beams are low-diversity variations on the same high-probability prefix and do not explore genuinely different reasoning strategies. Self-consistency using beam search (majority-voting over beams) improves over the single top beam — reaching 24.2% at 40 beams — but still underperforms self-consistency using sampling, which reaches 26.9% at 40 paths. On MultiArith, the pattern is similar: standard beam search peaks at 12.0% (beam size 5) and degrades to 10.5% at beam size 40; self-consistency using beam search reaches 12.3% at beam size 20 but drops to 10.8% at 40; self-consistency using sampling reaches 14.7% at 40 paths.

The paper attributes this gap to the lower diversity of beam search outputs, citing Li & Jurafsky (2016). This is a non-obvious and important finding: optimizing for sequence probability (beam search) suppresses the very diversity that makes self-consistency work. The best results come from stochastic sampling, which explores a wider variety of reasoning strategies even though individual paths may have lower probability.

Comparison to Ensemble-Based Approaches

Table 7 compares self-consistency against two few-shot ensembling strategies on LaMDA-137B: (1) prompt-order permutation — randomly permuting the exemplar order 40 times and taking a majority vote over greedy-decoded answers (Zhao et al., 2021; Lu et al., 2021); and (2) multiple sets of prompts — manually writing 3 different sets of chain-of-thought prompts and majority-voting across them (Gao et al., 2021).

On GSM8K, the baseline greedy decoding achieves 17.1%. Ensembling across 3 prompt sets achieves 18.6% (±0.5); ensembling across 40 prompt order permutations achieves 19.2% (±0.1); self-consistency with 40 sampled paths achieves 27.7% (±0.2). The ensemble approaches provide marginal gains (+1.5–2.1%) while self-consistency provides a +10.6% gain — roughly 5–7× larger. On MultiArith (baseline 51.8%): 3-prompt ensemble 57.1%, permutation ensemble 60.9%, self-consistency 75.7%. On SVAMP (baseline 38.9%): 3-prompt ensemble 42.1%, permutation ensemble 42.7%, self-consistency 53.3%. The pattern is consistent: prompt ensembling provides small, incremental improvements; self-consistency provides much larger gains.

The paper also tests model ensembling in Appendix A.1.3 (Table 10), where greedy-decoded outputs from multiple different models are combined via majority vote on GSM8K. The results are notably worse than single-model self-consistency: LaMDA-137B + PaLM-540B achieves 36.9% (vs. 74.4% for self-consistency on PaLM-540B alone); LaMDA-137B + PaLM-540B + GPT-3 achieves 33.3%. Weaker models drag down the performance of stronger models in a naive vote. Self-consistency avoids this by using only a single model's outputs.

Scalability: Number of Sampled Paths

Figure 2 (and the extended versions in Appendix Figures 7–8) provides the primary evidence on how performance scales with the number of sampled paths m. Across all tasks and all models, the relationship is monotonically increasing but with diminishing returns: the largest marginal gains occur in the first 5–10 paths, with the curve gradually flattening from 20 to 40 paths. The paper notes in Section 5 that "in most cases the performance saturates quickly" and that practitioners "can try a small number of paths (e.g., 5 or 10) as a starting point to realize most of the gains while not incurring too much cost."

This saturation behavior has practical implications for the compute-accuracy tradeoff. At m = 5, many tasks already capture the majority of the total gain achievable at m = 40. For example, on LaMDA-137B GSM8K (Figure 7 in appendix), accuracy at m = 5 is roughly 55–60% of the way from greedy decoding to m = 40 performance. On PaLM-540B GSM8K (Figure 8 in appendix), the curve is similar. This means the 40× compute cost of m = 40 can be substantially reduced (to ~5–10×) while retaining most of the benefit.

Ablation Studies and Robustness Checks

Answer aggregation strategies (Table 1): Unweighted majority vote matches or outperforms all probability-weighted alternatives. On PaLM-540B GSM8K, the five strategies achieve: weighted avg unnormalized 56.3%, weighted avg normalized 22.1%, weighted sum unnormalized 59.9%, weighted sum normalized 74.1%, unweighted sum (majority vote) 74.4%. The near-identity of weighted sum normalized (74.1%) and unweighted sum (74.4%) indicates that normalized sequence probabilities are effectively uniform across sampled paths — the weights add no information beyond the vote count. The dramatic failure of weighted average normalized (22.1%) shows that dividing by the number of paths supporting an answer punishes popular answers and produces worse-than-greedy performance. This ablation is the empirical foundation for the paper's choice of simple majority voting over more complex aggregation schemes.

Sampling strategies and parameters (Figure 4, left; Figure 6 in appendix): Self-consistency is robust to the choice of sampling hyperparameters. On GSM8K with PaLM-540B, the paper sweeps temperature values (T = 0.3, 0.5, 0.7), top-k values (k = 20, 40, no top-k), and nucleus sampling (p = 0.9, 0.95). All configurations produce nearly identical self-consistency curves across m = 1 to 40. The curves overlap tightly, with no configuration showing a systematic advantage over others. On LaMDA-137B (Figure 6), the same robustness is observed. This is practically significant: it means self-consistency does not require careful per-model or per-task tuning of sampling parameters — any reasonable stochastic decoding configuration works.

Number of sampled paths (Figures 2, 7, 8): Performance monotonically improves with more paths, but with diminishing returns. Across all tasks, models, and datasets, the accuracy curves are monotonically increasing in m. There is no observed case where more samples hurt performance (unlike beam search, where performance degrades with beam width in Table 6). The saturation point varies by task: easier tasks and larger models saturate earlier; harder tasks continue to benefit from additional samples up to 40.

Robustness to different prompt sets (Table 9): Self-consistency gains are consistent across manually written prompt variations. On PaLM-540B GSM8K, three different sets of chain-of-thought prompts yield greedy decoding accuracies of 56.5%, 54.6%, and 54.0%. Self-consistency achieves 74.4%, 72.1%, and 70.4% respectively — the absolute gains over greedy are +17.9, +17.5, and +16.4 percentage points. The gain is stable across prompt sets even though the baseline varies, indicating that self-consistency's benefit is not an artifact of a particular prompt design.

Imperfect prompts (Table 8): Self-consistency recovers performance when few-shot exemplars contain deliberate reasoning errors. When the numbers in the chain-of-thought exemplars are replaced with random values (while keeping the final answer correct), greedy decoding on LaMDA-137B GSM8K drops from 17.1% to 14.9%. Self-consistency with 40 paths recovers to 23.4% — higher than the original greedy baseline with correct prompts. This demonstrates that self-consistency provides robustness to prompt quality: even when the exemplars model flawed intermediate reasoning, the diversity of sampled paths allows the correct answer to emerge from the consensus.

Non-natural-language reasoning paths (Table 8): Self-consistency works with equation-only reasoning, but gains are smaller. On LaMDA-137B GSM8K, using prompts with only equations (e.g., "3 + 2 = 5") instead of natural language reasoning, greedy decoding achieves 5.0%, and self-consistency with 40 paths achieves 6.5% — a +1.5% gain. The gain is much smaller than with natural language reasoning paths (+10.6% from Table 2), which the paper attributes to equations being shorter and offering "less opportunity... for generating diversity in the decoding process" (Section 3.5). This is a useful diagnostic: it confirms that the diversity of reasoning (different ways of decomposing and solving the problem) is the active ingredient, not merely sampling noise. When the reasoning is compressed into short equations that admit fewer distinct valid formulations, self-consistency's advantage shrinks.

Zero-shot chain-of-thought (Table 8): Self-consistency works without few-shot exemplars. On PaLM-540B GSM8K, zero-shot CoT (Kojima et al., 2022) achieves 43.0% with greedy decoding. Self-consistency with 40 paths achieves 69.2% — a +26.2% gain. The gain in the zero-shot setting is actually larger than in the few-shot setting (+17.9% from Table 2), likely because the zero-shot baseline is weaker and leaves more room for improvement. This is an important finding for practical deployment: self-consistency can be applied even when well-crafted few-shot prompts are not available.

Combining self-consistency with other ensemble strategies (Table 11, Appendix A.1.4): Gains from prompt ensembling are additive but small relative to self-consistency itself. On PaLM-540B GSM8K, self-consistency alone achieves 74.4%. Adding 40 different prompt sets yields 75.4% (+1.0); adding 40 prompt permutations yields 73.8% (-0.6). The additional gain from prompt ensembling is negligible compared to the +17.9% gain from self-consistency over greedy decoding. This suggests that the diversity of reasoning paths within a single prompt configuration dominates the diversity from varying the prompt itself.

Consistency-accuracy correlation (Figure 5): The agreement rate among sampled paths is correlated with correctness, providing a built-in uncertainty estimate. On PaLM-540B GSM8K, the paper plots consistency (the fraction of 40 paths agreeing with the majority-vote answer) against accuracy for each question. There is a clear positive correlation: at high consistency (near 100% agreement), accuracy is near 100%; at low consistency (near 0% agreement, indicating scattered answers with no clear majority), accuracy is near 0%. The paper notes that "one can use low consistency as an indicator that the model has low confidence; i.e., self-consistency confers some ability for the model to 'know when it doesn't know'" (Section 3.5). This is a non-obvious and practically valuable finding: the same sampling procedure that produces the answer also produces a confidence score at zero additional cost.

Model scale ablation (Figure 4, right): Self-consistency benefits increase with model scale. On the LaMDA model series (1B, 2B, 5B, 10B, 20B, 50B, 100B, 137B, 200B parameters), self-consistency improves GSM8K accuracy at every scale. The absolute gain grows with model size: from near-zero at 1B to roughly +15 percentage points at 137B. Smaller models show minimal absolute gain because their base accuracy is near floor; larger models benefit more because they have a richer distribution of reasoning paths. This is consistent with the cross-model results in Table 2 and supports the claim that self-consistency amplifies existing reasoning capability.

Critical Assessment

Do the experiments actually demonstrate that self-consistency improves reasoning accuracy? Yes, overwhelmingly. The evidence spans 4 model families, 11+ datasets, 3 task categories (arithmetic, commonsense, symbolic), and 5 standard NLP tasks. The gains are large (up to +17.9% absolute on GSM8K), statistically reliable (standard deviations ≤ 0.5%), and consistent across every tested configuration. The paper demonstrates monotonic improvement with the number of sampled paths (Figures 2, 7, 8), robustness to sampling parameters (Figure 4, left), robustness to prompt variations (Table 9), and robustness to imperfect and zero-shot prompts (Table 8). The evidence is unusually thorough and leaves little doubt about the empirical claim.

However, the claim that self-consistency "achieves new state-of-the-art" requires qualification. The paper compares against prior published results (Tables 2–3), many of which use different models, different amounts of training data, or task-specific architectures. A fairer comparison would be: self-consistency on PaLM-540B vs. the best other method that could also be applied to PaLM-540B without additional training. That comparison is partially provided (vs. sample-and-rank, beam search, prompt ensembles) but does not include, for example, best-of-N with a trained verifier on the same PaLM-540B base model, which would be a stronger baseline. The paper's SoTA claim is better understood as "self-consistency on a large model achieves results competitive with or exceeding prior task-specific approaches" rather than "self-consistency is the optimal way to use PaLM-540B," since more sophisticated aggregation methods (e.g., weighted voting with a trained confidence model) were not tested.

The claim that self-consistency works because "correct reasoning processes tend to have greater agreement in their final answer than incorrect processes" is not directly tested. The paper demonstrates the outcome — majority voting improves accuracy — but does not provide a direct empirical measurement of the asymmetry between correct-path agreement and incorrect-path agreement. One could imagine computing, for each question, the fraction of correct paths that agree with each other vs. the fraction of incorrect paths that agree with each other, and showing that the former is systematically higher. Without this measurement, the hypothesized mechanism remains plausible but unverified. The qualitative examples in Tables 4, 12, and 13 are suggestive but not systematic.

The experimental design has several genuine weaknesses:

Single-run majority voting for the main results. The paper reports mean and standard deviation over 10 runs of self-consistency, but each run uses independent sampling — the standard deviation captures sampling noise, not model or prompt sensitivity. A deeper analysis of variance decomposition (how much is due to which samples are drawn vs. which questions are in the test set vs. prompt choice) is absent.
The test sets are small to moderate. GSM8K has 1,319 test examples; AQuA has 254; SVAMP has 1,000; ARC-challenge has 1,172. For tasks at the lower end, a few misclassified examples can meaningfully shift reported accuracies. The paper does not report confidence intervals based on test-set size, only standard deviation over sampling runs.
No comparison to simply increasing the temperature and taking a single sample. A natural question is: does the benefit come from majority voting, or simply from the fact that sampling at T > 0 sometimes produces a correct answer that greedy decoding misses, and you could get that benefit with m = 1 at a higher temperature? The paper does not report single-sample accuracy at the sampling temperatures used for self-consistency, which would isolate the contribution of the voting step from the contribution of stochastic decoding.
The answer parser is task-specific and failure modes are not analyzed. The paper notes that answers are parsed from the text following "The answer is" (Section 2, footnote 1), but does not report parsing failure rates, discuss edge cases (e.g., the model producing "The answer is 18 dollars" instead of "The answer is $18"), or analyze whether parsing errors systematically bias the results. Given that self-consistency relies on exact-match majority voting, even small parsing inconsistencies could fragment the vote for the correct answer.
No analysis of the diversity of the sampled reasoning paths. The paper claims diversity is essential but does not quantify it. Relevant metrics could include: the number of unique reasoning paths per question, the lexical diversity (distinct n-gram count), the number of distinct final answers produced, or the entropy of the answer distribution. Without such metrics, the mechanistic claim that "diverse reasoning paths" drive the gains — as opposed to simply sampling more draws from a noisy distribution and letting the mode emerge — is not empirically substantiated.
The models are not all publicly available. UL2-20B and GPT-3 (via API) are reproducible; LaMDA-137B and PaLM-540B are not. This limits independent verification of the largest claimed gains (e.g., +17.9% on GSM8K with PaLM-540B).
The cost in compute is acknowledged but not quantified in FLOPs or wall-clock time. The paper notes in Section 5 that self-consistency "incurs more computation cost" and suggests using fewer paths. However, a systematic efficiency analysis — e.g., plotting accuracy vs. total inference FLOPs and comparing the Pareto frontier of self-consistency against other methods — is absent. This is a significant omission given that the cost is the method's primary limitation.

Experiments that would have strengthened the paper:

A direct measurement of the agreement asymmetry between correct and incorrect reasoning paths (as discussed above).
Comparison against best-of-N with a separately trained verifier on the same base model, which would test whether the voting mechanism is superior to learned verification.
Analysis of self-consistency on the subset of questions where the greedy decoding answer is already correct — does self-consistency ever change a correct greedy answer to an incorrect one? If so, at what rate?
A systematic sweep of m from 1 to 100+ to identify where saturation genuinely occurs, rather than stopping at 40.
Experiments on models where chain-of-thought prompting with greedy decoding is near 0% accuracy, to test the paper's claim that "correct reasoning processes tend to have greater agreement" — if the model never produces a correct reasoning path (base accuracy near 0%), then all paths are incorrect, and majority voting should not help. Testing this boundary condition would strengthen the mechanistic argument.

Conditional nature of the claims:

The claims hold most strongly for: (1) large models (100B+ parameters) where the distribution over reasoning paths is rich; (2) tasks requiring multi-step reasoning with multiple valid solution strategies (arithmetic in particular); (3) tasks with unambiguous, parseable final answers; (4) settings where 5–40× inference cost is acceptable. The claims are weaker for: small models (<20B parameters) where absolute gains are modest; tasks with a single canonical reasoning path (where sampling produces only superficial variations); and latency-sensitive applications where 40 sequential forward passes are prohibitive. The paper is appropriately cautious about these boundaries in Section 5, but the main text claims (e.g., "striking margin") are primarily supported by the largest-model results and may not generalize to all deployment contexts.

6. Limitations and Trade-offs

1. Self-Consistency Multiplies Inference Cost by the Number of Sampled Paths

The assumption or constraint. Self-consistency replaces a single forward pass (greedy decoding) with m independent forward passes through the language model, where m = 40 in the main experiments. Each sampled reasoning path is generated from scratch — the model has no memory of previous samples, so there is no reuse of computation across paths. The paper is explicit about this limitation in Section 5:

"One limitation of self-consistency is that it incurs more computation cost. In practice people can try a small number of paths (e.g., 5 or 10) as a starting point to realize most of the gains while not incurring too much cost, as in most cases the performance saturates quickly (Figure 2)."

The consequence. At m = 40, self-consistency costs exactly 40× the inference FLOPs and wall-clock latency of greedy decoding (assuming no batching — the samples can be batched to reduce wall-clock time but not total FLOPs). For large models like PaLM-540B, this is a substantial cost: the paper reports in Appendix A.2 that inference on PaLM-540B takes "about 2 to 12 hours" per task (over roughly 1,000 examples) even for greedy decoding, meaning self-consistency at m = 40 would extend this to 80–480 hours per task on the described hardware (TPU v4, 192 chips). For API-based models like GPT-3, the cost multiplies directly — 40 API calls per question instead of 1.

The paper's suggestion to use fewer paths (5–10) partially mitigates this but also reduces the gain. The accuracy curves in Figure 2 show that at m = 5, roughly 55–60% of the total gain achievable at m = 40 is realized — meaning a practitioner accepting 5× cost captures a bit more than half the benefit, while capturing the full +17.9% on GSM8K requires the full 40× cost. There is no free lunch: the headline gains and the headline cost are inextricably linked.

What evidence exists in the paper. Figure 2 (and the extended Figures 7–8 in Appendix A.1.1) provides the raw data for this tradeoff, plotting accuracy vs. number of sampled paths. The curves are monotonic but concave — diminishing returns set in after ~10–20 paths. The paper does not report total FLOPs, wall-clock time, or dollar cost for any experiment, nor does it provide an efficiency frontier plotting accuracy vs. total compute. The cost discussion is limited to the one-paragraph acknowledgment in Section 5 and the brief hardware notes in Appendix A.2.

Mitigation status. The paper suggests using fewer paths in practice and proposes as future work using self-consistency to "generate better supervised data to fine-tune the model, such that the model can give more accurate predictions in a single inference run after fine-tuning" (Section 5). This is a post-hoc distillation approach: train the model to internalize the self-consistency behavior so that a single greedy-decoded output matches the quality of the majority-vote output. However, no distillation experiments are conducted in this paper, so this remains a suggestion rather than a demonstrated solution. The cost remains the most significant practical barrier to deploying self-consistency in latency-sensitive or budget-constrained settings.

2. The Method Is Fundamentally Limited to Tasks with Fixed, Parseable Answer Sets

The assumption or constraint. Self-consistency relies on exact-match majority voting over final answers extracted from the generated text. This requires that (a) answers come from a fixed, enumerable set (numbers, multiple-choice labels, yes/no), (b) answers can be reliably parsed from the generated reasoning paths via a simple pattern (e.g., "The answer is X"), and (c) the parsing produces a canonical representation suitable for exact-match comparison. The paper is explicit about this constraint in Section 2:

"One should note that self-consistency can be applied only to problems where the final answer is from a fixed answer set, but in principle this approach can be extended to open-text generation problems if a good metric of consistency can be defined between multiple generations, e.g., whether two answers agree or contradict each other."

The consequence. Self-consistency, as presented, cannot be applied to open-ended generation tasks — summarization, translation, creative writing, dialogue, code generation (where outputs are long and not trivially compared for equality), or any task where the "answer" is a free-form text without a single canonical form. This excludes a large fraction of real-world language model use cases. Even for tasks that do have fixed answers, the method is fragile to formatting inconsistencies: if the model produces "18 dollars" in one path and "$18" in another, exact-match voting would treat them as different answers and split the vote, potentially causing the correct answer to lose the majority even though every path was substantively correct. The paper does not analyze parsing failure rates or variance from parsing inconsistencies.

For the specific reasoning benchmarks tested, the fixed-answer constraint is satisfied (math answers are numeric, commonsense QA is multiple-choice, StrategyQA is yes/no). But this means the demonstrated success is bounded to a particular subset of NLP tasks — ones where the output space is small and the evaluation metric (exact match) aligns perfectly with the aggregation mechanism (majority vote). The method offers no guidance for tasks where correctness is graded on semantic similarity, factual accuracy of long-form text, or other non-exact-match criteria.

What evidence exists in the paper. Table 8 provides indirect evidence of the parsing dependency: when reasoning paths are equations only ("3 + 2 = 5"), self-consistency's gain drops from +10.6% to +1.5% on LaMDA-137B GSM8K — partly because shorter outputs provide less diversity, but also potentially because equation-formatted answers are more uniform and harder to diversify. The paper does not report any experiments on open-ended tasks where consistency would need a different operationalization. There is no ablation on answer parsing robustness (e.g., testing different parsing strategies or measuring how often the correct answer is produced but parsed inconsistently).

Mitigation status. The paper acknowledges the limitation and gestures toward a generalization — defining a consistency metric for open-text generation — but this is purely speculative. No metric is proposed, evaluated, or even sketched. The "in principle" language signals that the authors recognize this as a fundamental scope limitation rather than a minor implementation detail. No mitigation is attempted within the paper.

3. Self-Consistency Cannot Create Capability That the Base Model Lacks — It Only Amplifies Existing Capability

The assumption or constraint. Self-consistency works by sampling multiple reasoning paths from the model's own output distribution and taking a majority vote. This means the method can only select among answers that the model actually generates at some non-trivial rate. If the model never produces a correct reasoning path for a given question — i.e., its pass@k is zero or near-zero even at high k — then self-consistency provides no benefit: all sampled paths are incorrect, and majority voting over wrong answers cannot produce a correct one. The paper does not state this limitation explicitly, but it follows directly from the mechanism.

The consequence. The method fails on problems that are genuinely outside the model's competence. For small models (UL2-20B on GSM8K: greedy accuracy 4.1%, self-consistency 7.3%), the absolute improvement is small because the model's base capability is low — sampling more paths can only help if at least some paths are correct. For extremely difficult problems where even the largest models fail consistently, self-consistency provides zero gain. This is visible indirectly in the experimental results: on the hardest tasks and smallest models, self-consistency gains are modest in absolute terms (e.g., +3.2% on GSM8K with UL2-20B, Table 2; near-zero gain on letter concatenation with UL2-20B, Table 3). The relative improvement may be large, but the absolute ceiling is set by the model's underlying reasoning ability.

This limitation is practically significant because it means self-consistency is not a substitute for model scale or training improvements — it is a decoding strategy that extracts more of the capability already latent in the model. A model that fundamentally cannot solve certain classes of problems will not be rescued by self-consistency. This contrasts with methods like verifier training (Cobbe et al., 2021), which can in principle improve performance even on problems where the model's raw accuracy is zero (by training the verifier on human-labeled examples outside the model's generation distribution), though in practice such methods are also bounded by training data coverage.

What evidence exists in the paper. The scale-dependence of self-consistency gains provides indirect evidence: Figure 4 (right) shows that the absolute gain grows with model scale, from near-zero at 1B parameters to significant at 137B+. Smaller models have lower base accuracy and thus fewer correct paths in their sampling distribution, limiting what self-consistency can achieve. The symbolic reasoning tasks in the out-of-distribution setting (Table 3) show small absolute gains for smaller models: UL2-20B achieves 0.0% on letter concatenation with both greedy decoding and self-consistency — the model produces no correct paths, so no amount of sampling or voting helps. The paper does not provide a direct analysis of the relationship between base model pass@1 and self-consistency gain, which would quantify this limitation precisely.

Mitigation status. The paper does not address this limitation directly. The implicit mitigation is: use a larger model. Self-consistency is presented as a method that amplifies existing capability, not one that creates new capability. For practitioners, the practical implication is that self-consistency should be deployed on models that already have non-trivial base accuracy on the target task — otherwise, the compute spent on sampling is wasted. The paper's suggestion to use self-consistency for data generation and subsequent fine-tuning (Section 5) could address this limitation in the long run by improving the base model's capability through training, but this is not demonstrated.

4. The Mechanism Is Not Directly Validated — The Paper Does Not Measure Whether Correct Paths Actually Agree More Than Incorrect Paths

The assumption or constraint. The paper's central hypothesis, stated in Section 2, is:

"We hypothesize that correct reasoning processes, even if they are diverse, tend to have greater agreement in their final answer than incorrect processes."

The entire method rests on this asymmetry: majority voting works because the correct answer attracts more votes than any single wrong answer, which in turn requires that correct paths converge while incorrect paths diverge. If this hypothesis were false — if incorrect paths also tended to converge on the same wrong answer (e.g., due to a systematic model bias or a common failure mode) — then majority voting could amplify errors rather than correctness.

The consequence. The paper demonstrates that majority voting improves accuracy (an outcome), but does not provide direct evidence that the mechanism producing this improvement is the hypothesized agreement asymmetry. An alternative mechanism could produce the same outcome: for example, if the model's answer distribution is simply that the correct answer is the modal answer (appears more frequently than any single wrong answer, even if most paths are wrong), majority voting would select the correct answer without requiring any special agreement property among correct paths. The improvement over greedy decoding would then be explained by the fact that greedy decoding selects the single most probable path, which may not have the modal answer, while sampling reveals the true answer distribution.

Without a direct measurement of the agreement rates among correct vs. incorrect paths, the paper's explanatory claims about why self-consistency works remain plausible but unverified. This is not just a theoretical concern — it affects how one might try to improve or extend the method. If the mechanism is truly agreement-based, then increasing diversity (through higher temperature, different prompts, etc.) should help, and one should focus on generating maximally independent reasoning paths. If the mechanism is simply that the correct answer is the mode of the answer distribution, then diversity per se matters less than ensuring the mode is accurately estimated (which might be achieved more efficiently through other means).

What evidence exists in the paper. The paper provides qualitative examples (Tables 4, 12, 13) showing cases where incorrect greedy-decoded answers are overridden by a majority of correct sampled answers, which is consistent with the hypothesis but does not test it systematically. Figure 5 shows that consistency (agreement among all paths on the majority answer) correlates with accuracy, but this measures agreement on the winning answer aggregating both correct and incorrect cases — it does not separately measure agreement rates for correct paths vs. incorrect paths. The paper does not compute, for each question: "among the paths that are correct, what fraction agree with each other?" vs. "among the paths that are incorrect, what fraction agree with each other?" This pairwise or within-class agreement analysis would directly test the hypothesis.

Table 1 provides some indirect evidence: the "weighted avg (normalized)" strategy, which divides by the number of supporting paths and thus penalizes answers with many votes, achieves only 22.1% on GSM8K. This suggests that popular answers (which tend to be correct) do indeed attract more votes than unpopular answers (which tend to be incorrect) — but this only confirms that the correct answer is the mode, not that correct paths agree more with each other than incorrect paths do.

Mitigation status. The paper does not acknowledge this gap in mechanistic validation. The hypothesis is stated confidently in Section 2, and the results are interpreted through its lens, but no experiment is designed to isolate and test the hypothesized agreement asymmetry. A skeptic could argue that self-consistency works simply because sampling reveals the model's true answer distribution (in which the correct answer happens to be the mode), and that diversity of reasoning paths is incidental rather than causal. The paper provides no evidence to distinguish these interpretations.

5. Self-Consistency Does Not Address Nonsensical or Non-Factual Reasoning Paths — It Only Aggregates Answers

The assumption or constraint. Self-consistency treats all sampled reasoning paths as equally valid votes, regardless of their internal quality. The method makes no attempt to evaluate whether a given reasoning path is logically sound, factually accurate, or even coherent — it only looks at the final answer. If the model generates a path with nonsensical reasoning that happens to arrive at the correct answer (by chance, or by a flawed calculation that coincidentally yields the right number), that path contributes equally to the correct answer's vote count. Conversely, a path with flawless reasoning that makes a single arithmetic slip at the end gets its vote counted for a wrong answer. The paper acknowledges this in the Ethics Statement:

"Language models can sometimes generate nonsensical or non-factual reasoning paths, so one should use language models' outputs with extra caution."

The consequence. Self-consistency optimizes for answer accuracy, not reasoning quality. This has several downstream implications:

Rationales cannot be trusted even when the answer is correct. If self-consistency is used to generate explanations or reasoning traces for human consumption, the winning reasoning path may be internally inconsistent or factually wrong even though the final answer is right. The paper includes an example of this in Table 4: the StrategyQA example shows the model citing incorrect population figures ("Albany, Georgia has a population of about 88,000. Albany, New York has a population of about 95,000") while arriving at the correct yes/no answer. The majority vote selects the correct answer, but the supporting reasoning contains factual errors.
The method could reinforce systematic model biases. If the model has a systematic tendency to produce a particular wrong answer for a certain type of question (e.g., a stereotypical association in a commonsense task), self-consistency could amplify this bias by selecting the modal wrong answer, giving it the appearance of confidence through high consistency.
For interpretability and debugging, self-consistency obscures rather than illuminates the model's reasoning process. A practitioner who wants to understand why the model made a particular prediction gets a set of potentially conflicting reasoning paths and a vote tally, not a trustworthy explanation.

What evidence exists in the paper. The StrategyQA example in Table 4 (PaLM-540B) directly illustrates the problem: two sampled paths produce factually incorrect population numbers but arrive at the correct answer ("no"), while the greedy decoding path uses incorrect reasoning ("Prozac is an anti-depressant... The Great Depression is not a disease") but also arrives at the correct answer. The paper does not quantify how often the reasoning in the majority-voted answer is factually flawed, nor does it analyze the relationship between reasoning quality and answer correctness in the sampled paths.

The "imperfect prompts" experiment in Table 8 is tangentially relevant: when the few-shot exemplars contain deliberately incorrect intermediate reasoning (swapped numbers), self-consistency still improves answer accuracy (14.9% → 23.4%), demonstrating robustness to flawed exemplars. But this robustness also means the method is indifferent to reasoning quality — it will amplify correct answers even when the underlying reasoning is garbage.

Mitigation status. The paper acknowledges the nonsensical reasoning issue in the Discussion and Ethics Statement but proposes no mitigation. The stated future direction is "further work is needed to better ground models' rationale generations." Self-consistency as presented provides no tools for evaluating or improving reasoning quality — it is purely an answer-aggregation strategy. This is a fundamental tradeoff: the method achieves its simplicity and generality by ignoring reasoning quality entirely, but this means it inherits and potentially amplifies the base model's tendency to produce plausible-sounding but incorrect rationales.

6. The Method Has Not Been Tested on Tasks Requiring Factual Recall or Knowledge Integration — Only on Reasoning Benchmarks

The assumption or constraint. All primary experiments in the paper are on reasoning benchmarks — arithmetic word problems, commonsense QA, symbolic manipulation, and logical inference (NLI). These tasks share a common structure: given all necessary information in the prompt, apply multi-step logical or mathematical operations to derive an answer. They do not require the model to retrieve specific factual knowledge from its training data, integrate information across multiple documents, or reason about the truth of premises that must be verified against external knowledge.

The paper does include a few "standard NLP tasks" in Section 3.3 (BoolQ, HotpotQA, ANLI, e-SNLI, RTE) where chain-of-thought sometimes hurts. However, these are primarily used to demonstrate that self-consistency recovers the performance degradation from adding rationales — they are not the main evaluation. The paper does not claim that self-consistency works for all NLP tasks, but it also does not explicitly bound the task scope.

The consequence. It is unknown whether self-consistency would provide similar gains on tasks where the primary challenge is knowledge retrieval rather than multi-step reasoning from given premises. Consider closed-book question answering (e.g., "What year did the Berlin Wall fall?"), where the model either knows the fact or does not — there is arguably no multi-step "reasoning path" to diversify, and sampling multiple paths might simply produce the same fact repeatedly (if the model is confident) or scattered guesses (if it is uncertain), with majority voting adding little. Similarly, for tasks requiring integration of information from a provided context (e.g., long-document QA, summarization), the reasoning is constrained by the specific facts in the passage, which may limit the diversity of valid reasoning paths compared to self-contained math or logic problems.

The paper's strong results on arithmetic reasoning (up to +17.9%) and more modest results on commonsense reasoning (+1–6%) hint at this task-dependence. Arithmetic problems have a combinatorially large space of valid solution strategies (different orders of operations, different intermediate groupings), creating rich opportunities for path diversity. Commonsense QA is multiple-choice with fewer genuinely distinct reasoning strategies — you either know that moss is found in forests, or you don't. Symbolic reasoning tasks in the out-of-distribution setting (Table 3) show small gains for smaller models where the base capacity to generalize is limited.

What evidence exists in the paper. The Section 3.3 results on standard NLP tasks (Table 5) provide the closest evidence for tasks outside the core reasoning benchmarks. The gains there are moderate: +9.7% on ANLI-R1 (68.8% → 78.5%), +7.4% on e-SNLI (81.0% → 88.4%), +7.2% on RTE (79.1% → 86.3%), +4.2% on BoolQ (74.2% → 78.4%), +4.9 EM on HotpotQA (28.9 → 33.8). These are meaningful but smaller than the headline +17.9% on GSM8K. The paper does not analyze why gains differ across task types, nor does it test self-consistency on knowledge-intensive tasks like open-domain QA, fact verification, or entity linking. The benchmark selection is deliberately focused on reasoning, which is the paper's explicit scope ("reasoning in language models" in the title), but this means the generality of the method to non-reasoning tasks is unknown.

Mitigation status. The paper does not claim that self-consistency is universally applicable across all NLP tasks — its scope is explicitly "reasoning tasks" in the title and throughout. However, the framing in Section 1 ("a new decoding strategy... to replace the naive greedy decoding") and the broad claims in the abstract ("boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks") could be read as suggesting broader applicability than is demonstrated. The paper does not discuss which task characteristics predict large vs. small self-consistency gains, leaving practitioners to guess whether their specific use case would benefit. A systematic analysis of gain size as a function of task properties (reasoning depth, number of valid solution strategies, answer space size, reliance on factual knowledge) would clarify the boundary conditions but is absent.

7. Implications and Future Directions

How This Work Changes the Landscape

Self-consistency represents a methodological reframing rather than a paradigm shift — it does not introduce new models, new training procedures, or new architectural components, but it fundamentally changes how the field should think about extracting reasoning capability from language models at inference time. The paper's core contribution is establishing that diversity in the reasoning process is a resource to be exploited, not noise to be suppressed, even for tasks with unique correct answers. Before this work, the default assumption was that greedy decoding — selecting the single most probable token at each step — was the natural and correct choice for fixed-answer reasoning tasks (Wei et al., 2022; Chowdhery et al., 2022). Sampling was reserved for open-ended generation where multiple outputs are equally valid. Self-consistency inverts this: it demonstrates that stochastic sampling of multiple reasoning paths, followed by answer-level aggregation, produces substantially more accurate results than the single highest-probability path, and that this holds across model scales, architectures, task types, and prompt variations.

The reframing is captured in the probabilistic formulation the paper provides (Section 2): the quantity of interest is not the probability of any single reasoning path P(r_i | prompt, question), but rather the marginalized probability of the final answer summed over all paths that lead to it, Σ_{r_i} P(a | r_i) P(r_i | ...). Greedy decoding optimizes the former; self-consistency approximates the latter via Monte Carlo sampling. This shift in the optimization target — from path-level to answer-level probability — is the conceptual move that the paper contributes to the field. It implies that language model decoding for reasoning should be rethought as a density estimation problem over answers, not a sequence optimization problem over tokens.

This work also resolves a tension that was latent in the literature between two established practices. On one side, chain-of-thought prompting with greedy decoding (Wei et al., 2022) had demonstrated that eliciting intermediate reasoning steps improves performance. On the other side, Ye & Durrett (2022) had shown that adding chain-of-thought can sometimes hurt performance compared to standard prompting without rationales, creating uncertainty about whether chain-of-thought was a reliable default strategy. Self-consistency resolves this tension empirically (Section 3.3, Table 5): it not only recovers the lost performance on tasks where chain-of-thought degrades greedy accuracy (e-SNLI, RTE, ANLI-R1) but surpasses standard prompting, making chain-of-thought + self-consistency a safe default that does not require per-task validation. This is practically important: practitioners can apply the combination uniformly across tasks with confidence that it will not backfire.

A second tension the paper resolves concerns the role of language model probability estimates in evaluating output quality. Prior work on sample-and-rank (Adiwardana et al., 2020) and related re-ranking approaches implicitly assumed that the model's own sequence probabilities are informative about correctness — that the highest-probability output among a set of samples is likely to be the best one. The paper provides clear evidence that this assumption is false for reasoning tasks. Table 1 shows that weighted aggregation by sequence probability (both normalized and unnormalized) underperforms unweighted majority vote, and that the model's normalized probabilities across different reasoning paths are "quite close to each other" (Section 2, footnote 2) — the model is poorly calibrated and cannot distinguish correct from incorrect reasoning through probability alone. Figure 3 confirms this diagnostically: sample-and-rank barely improves over greedy decoding, while self-consistency with the same number of samples achieves dramatic gains. This finding redirects attention away from probability-based selection toward agreement-based selection — the signal for correctness lies in convergence across independent samples, not in the model's confidence scores.

The paper also changes the landscape of what counts as a strong baseline for reasoning tasks. Before this work, greedy chain-of-thought decoding was the standard point of comparison. After this work, any paper claiming to improve reasoning through prompting, fine-tuning, or architectural changes should compare against self-consistency as a decoding-level baseline — because self-consistency achieves substantial gains with zero training, zero auxiliary models, and zero additional data. If a proposed method cannot outperform self-consistency applied to the same base model, its added complexity is difficult to justify. This raises the bar for future work and establishes self-consistency as a standard tool in the evaluation toolkit.

Finally, the paper makes verifier training less urgently necessary for certain use cases, though it does not eliminate the need. Cobbe et al. (2021) showed that training a separate verifier model on thousands of examples substantially improves GSM8K performance. Self-consistency on PaLM-540B achieves 74.4% — exceeding the prior verifier-augmented SoTA of 55% (Table 2) — without any verifier at all. This does not mean verifiers are obsolete (self-consistency + verifier could potentially achieve even higher accuracy by weighting votes by verifier confidence rather than treating all paths equally), but it does mean that the cost-benefit calculus for training verifiers has shifted: self-consistency provides a strong, zero-training baseline that a verifier must meaningfully exceed to justify the additional data collection and training effort.

Follow-Up Research This Work Enables

Distillation of self-consistency behavior into single-pass models. The paper explicitly proposes in Section 5 that "one could use self-consistency to generate better supervised data to fine-tune the model, such that the model can give more accurate predictions in a single inference run after fine-tuning." The idea is straightforward: for each training question, run self-consistency with m = 40 to obtain the majority-vote answer, then fine-tune the base model on (question, majority-vote answer) pairs. This would amortize the 40× inference cost of self-consistency into a one-time training cost, after which a single greedy-decoded output would (ideally) match the quality of the full self-consistency procedure. The key research question is how much of the self-consistency gain survives distillation. Does fine-tuning on majority-vote answers actually teach the model to reason more reliably, or does it merely memorize the correct answers for the training questions without improving generalization? A strong follow-up would: (a) fine-tune PaLM-540B or an equivalent model on the GSM8K training set with self-consistency-generated answers, (b) evaluate on held-out test questions, and (c) measure whether the fine-tuned model's greedy decoding accuracy approaches the 74.4% achieved by self-consistency at m = 40. A critical control is comparing against fine-tuning on the same questions with ground-truth answers (standard supervised fine-tuning), to isolate whether self-consistency-generated labels provide any benefit beyond simply having more training data.

Direct measurement of the agreement asymmetry between correct and incorrect reasoning paths. The paper's central hypothesis — that "correct reasoning processes, even if they are diverse, tend to have greater agreement in their final answer than incorrect processes" (Section 2) — is stated but never directly tested. The evidence is entirely outcome-based: majority voting improves accuracy, which is consistent with the hypothesis but does not isolate the mechanism. A targeted follow-up would compute, for each question in GSM8K or a similar benchmark: (a) among all sampled paths that are correct, what fraction agree on the correct answer? (b) among all sampled paths that are incorrect, what is the size of the largest cluster of agreeing wrong answers? The hypothesis predicts that correct-path agreement rate should be high (many different correct reasoning strategies converge on the same number) while incorrect-path agreement should be low (wrong answers are idiosyncratic and scattered). If incorrect paths instead show non-trivial clustering on specific wrong answers — indicating systematic model biases or common failure modes — then the mechanism is more nuanced than the paper claims, and majority voting could potentially amplify errors for certain question types. This experiment requires no new models, only a systematic analysis of the sampling distributions the paper already collected.

Combining self-consistency with trained verifiers for weighted majority voting. The paper shows that unweighted majority vote matches or outperforms probability-weighted voting (Table 1) because the model's own sequence probabilities are uninformative. But this does not rule out weighting votes by a separately trained verifier model that is specifically optimized to distinguish correct from incorrect reasoning — the approach of Cobbe et al. (2021). A natural extension is: sample m = 40 reasoning paths via self-consistency, score each path with a trained verifier, and take a verifier-weighted majority vote rather than an unweighted one. The research question is whether verifier weighting provides gains beyond unweighted voting, and if so, how large, and at what verifier quality threshold the gains become significant. The paper's GSM8K result (74.4% unweighted) already exceeds the prior verifier-augmented SoTA (55%), but that prior result used a different base model (GPT-3 175B fine-tuned on 7.5k examples). A fair comparison would use the same PaLM-540B base model, train a verifier on PaLM-540B's own sampled outputs (not GPT-3's), and compare unweighted vs. verifier-weighted self-consistency. This would establish whether verifiers are complementary or redundant when combined with self-consistency.

Scaling analysis of self-consistency beyond m = 40. The paper's experiments stop at m = 40 sampled paths, and the accuracy curves in Figures 2, 7, and 8 show saturation behavior but have not fully plateaued at m = 40 for most tasks. A scaling study extending to m = 100, 200, or 500 would answer several questions: (a) Where does saturation genuinely occur — does performance continue to improve with hundreds of samples, or does it asymptote at m ≈ 40? (b) What is the limiting accuracy achievable by self-consistency on GSM8K with PaLM-540B — can it approach 90%+, or does it hit a ceiling determined by the fraction of questions for which the model ever produces a correct reasoning path? (c) Does the consistency-accuracy correlation (Figure 5) strengthen or weaken at very high m, and can consistency be used to dynamically determine how many samples are needed per question (early stopping when consistency exceeds a threshold)? This is primarily a compute question — the experiment requires significant inference budget but no methodological innovation — and would establish the asymptotic properties of the approach, which matter for high-stakes applications where accuracy is prioritized over cost.

Self-consistency for open-ended generation via learned consistency metrics. The paper explicitly limits self-consistency to tasks with fixed answer sets (Section 2), but gestures at extension to open-text generation "if a good metric of consistency can be defined between multiple generations, e.g., whether two answers agree or contradict each other." This is a non-trivial research direction. For tasks like summarization, the "consistency" between two generated summaries could be operationalized as: (a) ROUGE-L or BERTScore similarity between the two texts (measuring surface-level or semantic overlap), (b) entailment between the two texts (does summary A entail summary B, measured by an NLI model?), or (c) agreement on a set of automatically extracted factual claims. A concrete experiment would take a summarization benchmark (e.g., CNN/DailyMail or XSum), sample m summaries from a model like PaLM-540B, cluster them by some consistency metric, and select the centroid of the largest cluster as the output. The key comparison is whether this "self-consistent summary" achieves higher ROUGE or factuality scores than greedy decoding or best-of-N sampling. The paper's negative result on beam search for reasoning (Table 6) suggests that diversity is essential, which would favor consistency metrics that preserve cluster separation rather than collapsing diverse outputs. The risk is that for summarization, all sampled summaries may be surface-level variations of the same content, providing insufficient answer diversity for self-consistency to help — a negative result that would clarify the boundary conditions of the method.

Difficulty-conditioned self-consistency: does the gain vary predictably with problem difficulty? The paper does not analyze whether self-consistency gains are uniform across easy and hard problems, or whether they are concentrated on specific difficulty tiers. This matters for efficient deployment: if self-consistency primarily helps on medium-difficulty problems (where the model sometimes succeeds and sometimes fails, and extra samples resolve the ambiguity) but provides little gain on very easy problems (where greedy decoding is already correct) or very hard problems (where the model never produces a correct path), then the sampling budget could be allocated adaptively — m = 1 for trivially easy questions, m = 40 for borderline questions, and m = 1 or even abstention for impossible questions. A follow-up would bin GSM8K questions by the base model's greedy decoding accuracy (or by a difficulty estimator), and measure self-consistency gain as a function of difficulty bin. The paper's own Figure 5 provides a starting point: consistency correlates with accuracy, so one could use consistency after a small number of initial samples (e.g., 5) to predict whether more samples would help, and allocate budget accordingly. This would connect self-consistency to the broader literature on adaptive computation and test-time compute allocation, making the cost-accuracy tradeoff explicit and controllable.

Practical Applications and Downstream Use Cases

High-stakes reasoning evaluation where accuracy dominates cost concerns. For applications like standardized test grading, competition math problem solving, or formal verification of model outputs, the 40× inference cost of self-consistency is easily justified by the accuracy gains. On GSM8K with PaLM-540B, self-consistency provides +17.9 absolute percentage points — reducing the error rate from 43.5% to 25.6%, a ~41% relative error reduction. In contexts where a wrong answer has high cost (e.g., an automated tutoring system giving incorrect feedback to a student), this improvement is substantial. The consistency score (Figure 5) additionally provides a confidence estimate that can trigger human review for low-consistency predictions, creating a human-in-the-loop system where the model handles high-confidence cases autonomously and escalates uncertain cases.

Generating high-quality training data for model fine-tuning or distillation. The paper's suggestion in Section 5 to use self-consistency for data generation is immediately actionable. For any reasoning task where unlabeled questions are abundant but labeled answers are scarce, one can: (a) apply self-consistency with m = 40 to generate high-accuracy pseudo-labels, (b) filter to questions where consistency exceeds a threshold (e.g., >80% agreement), retaining only high-confidence labels, and (c) fine-tune a smaller or faster model on the resulting (question, majority-vote answer) pairs. This pipelines the 40× cost into a one-time data generation step, after which the fine-tuned model runs with single-pass greedy decoding at 1× cost while (ideally) retaining much of the self-consistency accuracy gain. The paper's results on UL2-20B (Table 2) show that small models benefit less from self-consistency directly (only +3.2% on GSM8K), but a small model fine-tuned on PaLM-540B's self-consistency outputs could potentially achieve accuracy far beyond what the small model can achieve on its own — a form of capability transfer from large models to small ones via decoding strategy.

Confidence estimation and selective prediction in production systems. The consistency-accuracy correlation in Figure 5 means self-consistency provides a built-in, zero-cost uncertainty estimate. In a production QA system, one can sample m = 10–20 paths, compute the majority-vote answer and the consistency score, and use the consistency score to decide whether to: (a) return the answer directly (high consistency), (b) flag the answer as low-confidence (medium consistency), or (c) abstain and escalate to a human operator (low consistency). This is more informative than token-level probability calibration, which the paper shows is unreliable (Table 1). The approach requires no additional training or calibration data — it emerges directly from the sampling procedure. For applications like medical QA, legal reasoning, or financial analysis where incorrect answers carry high risk, this built-in abstention mechanism is practically valuable and immediately deployable with any off-the-shelf language model capable of chain-of-thought prompting.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Greedy Decoding Leaves Reasoning Performance on the Table

Conflicting Design Philosophy in Language Model Decoding

Where Prior Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Probabilistic Formulation: The Latent Reasoning Path Variable

Answer Aggregation Strategies: Why Unweighted Majority Vote Wins

The Sampling Mechanism: How Diversity Is Introduced

Task-Dependent Answer Extraction

Self-Consistency as "Self-Ensemble"

4. Key Insights and Innovations

Innovation 1: Reasoning Diversity as a Resource, Not Noise — Reframing the Role of Stochastic Decoding for Fixed-Answer Tasks

Innovation 2: Majority Voting as Marginalization — Providing a Probabilistic Foundation for Answer Aggregation

Innovation 3: Self-Consistency as Robustness Across Scale, Prompts, and Task Types — Establishing Generality Without Specialization

Innovation 4: The Consistency-Accuracy Correlation — Self-Consistency as an Uncertainty Estimation Mechanism

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Arithmetic Reasoning

Commonsense and Symbolic Reasoning

Self-Consistency When Chain-of-Thought Hurts

Comparison to Sample-and-Rank

Comparison to Beam Search

Comparison to Ensemble-Based Approaches

Scalability: Number of Sampled Paths

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

1. Self-Consistency Multiplies Inference Cost by the Number of Sampled Paths

2. The Method Is Fundamentally Limited to Tasks with Fixed, Parseable Answer Sets

3. Self-Consistency Cannot Create Capability That the Base Model Lacks — It Only Amplifies Existing Capability

4. The Mechanism Is Not Directly Validated — The Paper Does Not Measure Whether Correct Paths Actually Agree More Than Incorrect Paths

5. Self-Consistency Does Not Address Nonsensical or Non-Factual Reasoning Paths — It Only Aggregates Answers

6. The Method Has Not Been Tested on Tasks Requiring Factual Recall or Knowledge Integration — Only on Reasoning Benchmarks

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Large Language Models are Zero-Shot Reasoners

STaR: Bootstrapping Reasoning With Reasoning