How Language Model Hallucinations Can Snowball

🎯 Pitch

Language models often fabricate false justifications for their own mistakes that they can separately identify as incorrect—GPT-4 catches 87% of its own snowballed errors when asked in isolation. This reveals that some hallucinations stem not from ignorance but from a pathological commitment to coherence with an earlier wrong answer, and even advanced prompting fails to fix it.

1. Executive Summary

This paper introduces and empirically demonstrates the phenomenon of hallucination snowballing, where a language model over-commits to an early incorrect answer and then generates additional false claims it would separately recognize as mistakes (e.g., GPT-4 asserting a factorization that it later confirms is invalid when queried in isolation). Across three constructed QA datasets—Primality Testing, Senator Search, and Graph Connectivity—the authors find that ChatGPT and GPT-4 detect 67% and 87% of their own hallucinated justifications respectively when those claims are presented independently, establishing that these errors are not attributable to knowledge gaps but rather to a consistency pressure that forces the model to fabricate supporting evidence for an incorrect answer it already committed to outputting. The work further shows that prompting with "Let's think step by step" dramatically improves task accuracy but fails to eliminate snowballed hallucinations, with GPT-4 still snowballing on 95% of its remaining errors, establishing that conditioning on faulty context induces simple mistakes the model would not otherwise make even when the overall reasoning quality improves.

2. Context and Motivation

The Core Problem: Hallucinations Driven by Coherence Pressure, Not Knowledge Gaps

The standard framework for understanding hallucinations in language models (LMs) treats them as a knowledge problem: the model generates a false statement because it fundamentally does not "know" the correct fact. This framing—articulated most clearly by Zheng et al. (2023)—suggests that LMs hallucinate when they encounter information beyond their training distribution or memorization capacity. Consequently, the dominant mitigation strategies have focused on supplementing the model's internal knowledge: retrieval-augmented generation (Lewis et al., 2020), consulting external knowledge bases (Shuster et al., 2021), and providing the model with factual context at inference time (Peng et al., 2023).

This paper challenges the completeness of that framing. It asks a deceptively simple but profound question: Do LMs only hallucinate when they don't know a fact, or can they generate falsehoods that they would recognize as false if asked separately? If the latter occurs systematically, then some hallucinations are not caused by missing knowledge but by the sequential nature of autoregressive generation itself—the model commits to an answer early and then feels pressure to justify it coherently, manufacturing supporting evidence that it otherwise "knows" is wrong.

The authors call this phenomenon hallucination snowballing and set out to demonstrate that it is not a rare edge case but a systematic failure mode of state-of-the-art models across diverse reasoning domains. The paper's central empirical contribution is constructing conditions under which this behavior can be isolated, measured, and verified: the model makes an incorrect claim in service of justifying a wrong answer, and when that claim is extracted and presented in isolation (in a separate conversation with no access to the earlier context), the model correctly identifies it as false.

Why This Problem Matters: Practical and Theoretical Significance

The significance of hallucination snowballing extends beyond academic curiosity—it has direct implications for how deployed LMs fail and why certain failure modes are resistant to straightforward fixes.

Practical risks in deployment. In real-world use, users interact with LMs through open-ended dialogue. A user asking "Is there a flight from city A to city B?" receives not just a Yes/No answer but a natural-sounding justification: "Yes, because there's a flight from A to C, then C to D, then D to B." If the model hallucinates the C-to-D connection, the user has no obvious way to detect the error—the entire response is fluent and internally consistent. Worse, the user might act on this information (booking flights, making travel plans) based on fabricated intermediate steps that the model itself would recognize as false if queried separately. This is a qualitatively different risk from a simple factual error: the error is compounded by the model's ability to generate plausible-sounding but self-contradictory reasoning.

Theoretical implications for LM capabilities. The paper builds on a foundational theoretical result from Merrill and Sabharwal (2023), who showed that bounded-precision transformers cannot solve problems outside the complexity class $TC^0$ in a single generation step. This has profound implications for how we interpret LM outputs on sequential reasoning tasks:

Primality testing requires inherently sequential computation—determining whether a number $n$ is prime involves checking divisibility by potential factors, and certified primality testing cannot be parallelized to constant depth. Formally, primality testing is in P (Agrawal et al., 2004) but not in $TC^0$ unless it is also in L—i.e., any algorithm would require $\Omega(\log \log n)$ bits of overhead that a fixed-depth transformer cannot simulate in one step.
Graph connectivity (determining whether a path exists between two nodes in a directed graph) is L-complete, meaning it is not in $TC^0$ unless $TC^0 = L$ —a collapse of the standard complexity hierarchy that computer scientists widely disbelieve.

These theoretical results establish that no amount of training data or model scaling can enable a transformer to solve these problems correctly in a single forward pass—the architecture simply lacks the computational depth. When an LM is prompted with a yes/no question and answers immediately (as the first token), it is being asked to do the impossible: solve an inherently multi-step problem in one timestep. The model must guess, and that guess is often wrong.

The paper's innovation is connecting this architectural limitation to the quality of subsequent generations. The model doesn't just guess wrong—it then constructs an elaborate justification for its wrong guess, and those justifications often contain claims that the model would separately reject. The error is not just that the model got the answer wrong; it's that the process of generating the wrong answer induces secondary errors that are even more basic than the original mistake.

Prior Approaches and Where They Fall Short

Hallucination as a knowledge gap problem. The dominant mitigation paradigm—retrieval-augmented generation—addresses the knowledge gap explanation: if the model doesn't know something, give it access to a knowledge base at inference time (Lewis et al., 2020; Shuster et al., 2021). This approach has shown genuine benefits, particularly for factual queries where relevant documents exist. However, it fundamentally cannot address snowballed hallucinations for two reasons:

The model possesses the knowledge but overrides it. A snowballed hallucination is, by definition, a claim the model recognizes as false when asked independently. Providing external knowledge would not help, because the model already "knows" the correct answer—it simply fails to access or act on that knowledge when under coherence pressure from its own prior output. In the primality testing example from Figure 1, GPT-4 claims $13 \times 745 = 9677$ and then separately confirms that 13 does not divide 9677. Adding a knowledge base of multiplication facts would be redundant; the model can already perform divisibility checks correctly.
The error arises from context, not from missing facts. The snowballed hallucination is generated specifically because the model has already committed to an incorrect answer. The flawed context—the model's own previous generation—triggers the hallucination, not a gap in pre-existing knowledge.

This reveals a fundamental limitation of retrieval-based approaches: they address what the model doesn't know but not what the model won't say due to coherence pressure.

Chain-of-thought prompting as a partial fix. The paper explicitly shows that zero-shot chain-of-thought reasoning ("Let's think step by step") dramatically improves task accuracy—from error rates exceeding 60% down to 3.87% for GPT-4 on average (Tables 6 and 8). This is consistent with prior work showing that intermediate reasoning steps improve LM performance (Nye et al., 2021; Wei et al., 2022; Press et al., 2022). However, the paper's crucial and novel finding is that chain-of-thought does not eliminate snowballed hallucinations on remaining errors. GPT-4 still snowballs on 94.90% of its incorrect reasoning chains (Table 9). When the model makes a mistake in its step-by-step reasoning, that mistake becomes part of the context and induces downstream hallucinations in later steps—even when the model "knows" that those later steps contain false claims.

This finding is significant because it reveals a failure mode of chain-of-thought itself that prior work had not identified. The standard narrative around chain-of-thought is that it improves reasoning by allowing the model to break complex problems into simpler steps. The paper shows that this benefit comes with a hidden cost: when a reasoning step is wrong, that error propagates through subsequent steps in ways the model would not generate in isolation.

Self-correction and backtracking. The paper connects its findings to work on LM self-correction, noting that prompting with "Review your previous answer and find problems with your answer" (Kim et al., 2023) is conceptually related to the backtracking behavior the paper advocates. However, the paper's framing differs: rather than asking the model to review after generating a full answer, the deeper issue is that the model should not generate snowballed hallucinations in the first place—it should recognize that the initial answer is uncertain and avoid committing to it in a single token.

Exposure bias in text generation. The phenomenon connects to a well-known issue in sequence generation: exposure bias, where models are trained on gold-standard context but must generate from their own potentially erroneous outputs at inference time (Wang and Sennrich, 2020; Arora et al., 2022). Prior work documented that this leads to compounding errors in machine translation and open-ended text generation. The paper's contribution is going beyond demonstrating error propagation: it shows that the model can recognize its own propagated errors when they are presented in isolation. This is a sharper and more diagnostic finding—it's not just that errors accumulate (which could be explained by cascading failures in the generation process), but that the model generates specific false claims that it would separately reject. This demonstrates that the model's knowledge and its generation behavior can be decoupled in the presence of defective self-generated context.

Misleading questions and false presuppositions. The paper draws connections to work on how LMs behave when given questions with false presuppositions (Kim et al., 2021, 2022) or misleading framing (Lin et al., 2022). In those settings, the LM is fed context that actively misleads it—for example, "Which linguist invented the lightbulb?" presupposes a linguist invented the lightbulb, and LMs often answer within that false frame rather than challenging the premise. The paper's key distinction is that in hallucination snowballing, the misleading context is generated by the model itself, not provided by the user. The initial question is innocent—"Is 9677 prime?"—and the model responds by first committing to an incorrect answer and then fabricating justification that becomes misleading context for itself.

How the Paper Positions Itself

The paper positions itself at the intersection of three research threads and contributes to each:

Reconceptualizing hallucination causes. Rather than treating hallucination as a monolithic problem of insufficient knowledge, the paper introduces a taxonomy of hallucination sources: knowledge-gap hallucinations (the standard framing) versus coherence-induced hallucinations (snowballing). This distinction matters because the two types require fundamentally different mitigation strategies. Knowledge-gap hallucinations can benefit from retrieval and fact-checking; coherence-induced hallucinations require the model to either (a) not commit to an answer prematurely, (b) recognize when it has committed incorrectly and backtrack, or (c) be trained to separate its justification process from its prior output tokens.

The paper's key empirical claim is that coherence-induced hallucinations are not a minor edge case—they account for 67% of ChatGPT's incorrect claims and 87% of GPT-4's in their experimental setup. This means that for these tasks, most of the model's hallucinations are recognized by the model itself as false when context is removed.

Bridging LM behavior and theoretical limitations. The paper explicitly grounds its experimental design in the theoretical results of Merrill and Sabharwal (2023), choosing tasks that are provably not solvable by transformers in a single generation step. This makes the paper's findings not just empirical observations but predictable consequences of known architectural limitations. The rationale (Section 2) is that because transformers cannot solve inherently sequential problems in one step, they must guess when forced to answer Yes/No immediately, and those guesses will sometimes be wrong. The paper's contribution is demonstrating what happens after that wrong guess: the model doesn't just stop, it generates supporting evidence that contradicts its own knowledge.

Reframing the promise and limitations of chain-of-thought. The paper uses chain-of-thought prompting as both a comparison point and a diagnostic tool. The finding that chain-of-thought dramatically reduces errors but does not eliminate snowballed hallucinations on remaining errors positions the work as identifying a residual failure mode that persists even when models are prompted to reason step-by-step. This implies that the problem is not just about the model's ability to break down tasks, but about a more fundamental issue: any erroneous intermediate step becomes corrupting context for subsequent steps, regardless of whether the model "knows" better.

Connecting to consistency research. The paper frames snowballed hallucination as a form of inconsistency in LM reasoning, relating it to prior work showing that LMs produce different answers depending on seemingly irrelevant context variations (Lin et al., 2022) or whether they are allowed to show intermediate steps (Wei et al., 2022). The novel contribution is demonstrating a specific mechanism for this inconsistency: the model's own generation creates context that constrains later outputs in ways that produce statements the model would not endorse independently. This is a more precise characterization than simply observing that LMs are inconsistent—it identifies how the inconsistency is produced and why it takes the specific form of generating false claims the model can verify as false.

In summary, the paper addresses a gap in our understanding of LM hallucinations: the possibility that some false statements are generated not because the model lacks knowledge, but because its commitment to previously generated context overrides that knowledge. The work is positioned as both a diagnostic contribution (identifying and measuring the phenomenon) and a call to re-examine the assumptions underlying current hallucination mitigation strategies—particularly the assumption that adding knowledge or improving reasoning depth will solve the problem when the error mechanism is coherence-driven.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a diagnostic empirical work rather than a system-building paper — it constructs a controlled experimental framework to isolate and measure hallucination snowballing, the phenomenon where a language model over-commits to an early incorrect answer and then generates supporting false claims that it would separately recognize as incorrect. The technical approach solves a measurement problem: given that LMs produce fluent, extended responses to questions, how can we determine whether specific false claims within those responses are caused by missing knowledge versus by consistency pressure from the model's own prior output? The solution "shape" is a two-stage evaluation pipeline: first, elicit incorrect answers with justifications from the model on tasks where the correct answer is known and the form of incorrect justification is predictable; second, extract the specific false claims from those justifications and present them to the same model in isolated verification queries (in a new session, with no access to the original context) to test whether the model can recognize those claims as false.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework has five logical components, though they are distributed across dataset construction and evaluation rather than bundled into a single software system:

Dataset Generator — produces 500 yes/no questions per domain (Primality Testing, Senator Search, Graph Connectivity) with controlled properties: the correct answer is always fixed (always Yes for primality, always No for the other two), and incorrect answers predictably take a specific justificatory form (a false factorization, a false senator claim, a false flight connection) that can be automatically extracted and verified.
LM Query Interface (First Stage) — sends each question to ChatGPT or GPT-4 with greedy decoding ( $t=0$ ) and collects the full response, recording whether the first token (or earliest answer-indicating text) commits to Yes or No, and what justification the model provides.
Claim Extractor — parses the model's incorrect justification to isolate the specific factual claim used to support the wrong answer. This uses heuristics tailored to each dataset: factor extraction for primality testing (via a separate ChatGPT call), senator name extraction for senator search, and manual flight-sequence analysis for graph connectivity.
Verification Interface (Second Stage) — in a completely new conversation session (no access to the original exchange), presents the extracted claim as an isolated yes/no question to the same model and collects the verification response.
Manual Evaluation Layer — human annotators inspect the verification output to determine whether the model correctly identified the extracted claim as incorrect, classifying each case as snowballed hallucination (model recognized the error) or persistent hallucination (model failed to recognize the error).

Information flows strictly in one direction: Dataset → First Stage LM Query → Response Parsing (accuracy check) → Claim Extraction (on incorrect responses) → Verification LM Query (isolated, new session) → Manual Judgment. Crucially, there is no feedback loop — the verification stage is purely diagnostic and does not influence the model's original response, which is essential for establishing that the snowballed claims were generated in the original context and recognized as false in isolation.

3.3 Roadmap for the Deep Dive

First, the theoretical motivation for dataset design — why the chosen tasks (primality testing, graph connectivity, senator search) are provably unsolvable by transformers in a single generation step, and how this creates the conditions for snowballing.
Second, the detailed construction of each of the three datasets: the constraints, the data generation procedures, the rationale for fixing correct answers to a single label, and how the predictable form of incorrect justifications enables systematic measurement.
Third, the first-stage inference procedure: how models are queried, the significance of first-token commitment as a mechanism enabling snowballing, and how accuracy is evaluated.
Fourth, the claim extraction pipeline — the per-dataset heuristics for isolating the specific false claim from the model's full justification, including the automated extraction tools and manual verification of extraction quality.
Fifth, the verification stage — how isolated queries are constructed in new sessions, the specific prompt formats for each dataset, and the manual evaluation protocol for determining whether the model recognized its own error.
Sixth, the rationale behind design choices: why greedy decoding, why zero-shot, why the specific temperature settings, and what alternative approaches were considered or implicitly ruled out.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a diagnostic measurement paper whose core technical contribution is a controlled experimental pipeline for determining whether specific LM hallucinations are generated because the model lacks knowledge versus because the model is constrained by coherence pressure from its own prior output tokens. The key insight is that by fixing the correct answer to a single label and designing tasks where incorrect justifications take a predictable, extractable form, the authors can systematically isolate the false claims and test the model's recognition of them in complete isolation from the original context.

3.4.1 Theoretical Motivation for Task Selection

The paper's experimental design is grounded in a specific theoretical result from Merrill and Sabharwal (2023), which establishes that bounded-precision transformers cannot solve problems outside the complexity class $TC^0$ in a single generation step. Understanding this result is essential to understanding why the paper chose its specific tasks and what a "generation step" means in this context.

What $TC^0$ represents. $TC^0$ is a complexity class consisting of problems that can be solved by constant-depth, polynomial-size Boolean circuits with unbounded fan-in AND, OR, and threshold gates. In plain language: $TC^0$ problems can be solved by circuits where the longest path from input to output has a fixed (constant) number of gates, no matter how large the input is. This means the computation is highly parallelizable — all paths through the circuit are short, so the computation can be done in a small number of parallel steps.

What it means for transformers. Merrill and Sabharwal (2023) proved that a single forward pass of a log-precision transformer (the standard architecture used in LLMs, which uses floating-point representations with a logarithmic number of bits relative to the input size) can only compute functions within $TC^0$ . This is a fundamental architectural limitation — not something that can be overcome by adding more layers or parameters (as long as the depth is fixed for a given model). Each generation step (producing one token) is a single forward pass through the transformer. Therefore, if a problem requires computation that is not in $TC^0$ , no transformer can solve it by producing the answer in the first token.

Why primality testing is not in $TC^0$ . Primality testing — determining whether an integer $n$ is prime — was proven to be in P (polynomial time) by Agrawal et al. (2004), meaning it can be solved by an algorithm whose running time is polynomial in the number of bits needed to represent $n$ . However, being in P does not imply being in $TC^0$ . The current understanding is that primality testing cannot be in $TC^0$ unless it is also in L (log-space), because any $n$ can be factored with only $O(\log \log n)$ bits of additional storage. Since it is widely believed that $TC^0 \neq L$ (constant-depth parallel circuits cannot simulate all of log-space computation), primality testing is presumed to be outside $TC^0$ . This means that a transformer cannot determine whether a number is prime in a single forward pass — it fundamentally requires multiple sequential steps of computation, such as iteratively checking divisibility by potential factors.

Why graph connectivity is not in $TC^0$ . Graph connectivity (determining whether there is a path from node $s$ to node $t$ in a directed graph) is a classic L-complete problem: it is complete for the class of problems solvable in logarithmic space. L-completeness means that if graph connectivity were in $TC^0$ , then $TC^0 = L$ — a collapse of the standard complexity hierarchy that is widely disbelieved in theoretical computer science. The intuition is that finding a path requires exploring the graph in a way that depends on previously visited nodes, which inherently requires sequential state maintenance that constant-depth circuits cannot perform. Therefore, a transformer cannot determine whether a path exists in a single forward pass.

The senator search task and factual recall. The senator search task is not based on the Merrill and Sabharwal (2023) theoretical result in the same way — it is a factual knowledge task rather than a sequential reasoning problem. However, it shares the key property that the model must commit to an answer (Yes/No) before generating the justification. The paper includes this task to show that snowballing occurs even in domains where the primary challenge is knowledge retrieval rather than computational depth, and where chain-of-thought prompting can perfectly solve the task (as shown in Table 8, achieving 0% error rate).

The critical link to experimental design. The theoretical results mean that when a transformer is prompted with a yes/no primality testing or graph connectivity question and asked to answer immediately (as is natural in English yes/no question formats), the model cannot possibly compute the correct answer in the first token. It must guess. The paper's experimental framework is designed to capture what happens after that guess: when the guess is wrong, does the model simply acknowledge uncertainty, or does it construct an elaborate false justification? And if the latter, does that justification contain claims the model would separately recognize as incorrect?

This theoretical grounding is what makes the paper more than an anecdotal observation — it demonstrates that the phenomenon is a predictable consequence of a known architectural limitation, not a random failure mode. The tasks are chosen specifically because they are provably impossible for the transformer architecture to solve in the generation regime being tested (immediate answer before reasoning).

3.4.2 Dataset Design: Shared Principles

All three datasets follow a common design philosophy that enables systematic measurement of snowballed hallucinations. The paper constructs exactly 500 examples per dataset, providing sufficient statistical power while keeping manual evaluation tractable.

Principle 1: Fix the correct answer to a single label for all examples in each dataset. In the Primality Testing dataset, all 500 numbers are primes between 1,000 and 20,000, so the correct answer is always Yes. In the Senator Search and Graph Connectivity datasets, all question-answer pairs are constructed so the correct answer is always No.

This design choice is critical for measurement tractability. If the correct answer varied across examples, the form of the incorrect justification would also vary (an incorrect "Yes" with a false factorization vs. an incorrect "No" with a claim that no factorization exists). By fixing the correct answer, the authors ensure that every incorrect model response follows the same pattern: in primality testing, an incorrect answer is always "No" with an accompanying false factorization; in senator search, an incorrect answer is always "Yes" with a false senator claim; in graph connectivity, an incorrect answer is always "Yes" with a false flight sequence. This predictability enables automated or semi-automated extraction of the specific false claim, which is essential for scaling the evaluation to 500 examples per dataset.

Principle 2: Design questions so incorrect justifications produce easily verifiable claims. The paper deliberately chooses tasks where the form of justification for an incorrect answer is constrained and predictable:

For primality testing, claiming a number is not prime naturally leads the model to provide a factorization (e.g., " $9791 = 13 \times 751$ "). This factorization is a simple, extractable claim — two integers that allegedly multiply to the target number.
For senator search, claiming a senator existed naturally leads the model to name a specific senator (e.g., "John P. Hale"). The verifiable claims are then whether that senator represented the specified state and attended the specified college.
For graph connectivity, claiming a path exists naturally leads the model to list a sequence of flights (e.g., "City F to City H, City H to City K, City K to City G..."). The verifiable claims are whether each individual flight exists in the provided flight information.

This design choice enables the verification stage: rather than asking the model to assess the full justification (which is what the model already generated and endorsed), the paper extracts atomic factual claims that can be verified yes/no, making the verification task simple and unambiguous.

Principle 3: Use zero-shot prompting to match typical user interaction patterns. The paper does not provide few-shot examples in the prompt. The rationale is that the most common way users interact with deployed LMs is by asking questions directly, without engineering elaborate prompts. The paper explicitly states (Section 3.2): "We focus on the model behavior under the direct prompt, which is the most common way users interact with LMs." This choice increases ecological validity but also makes the results somewhat conservative — few-shot prompting might improve accuracy, though the chain-of-thought experiments in Section 4 suggest that reasoning improvements do not eliminate snowballing on remaining errors.

3.4.3 Dataset Construction: Primality Testing

Data generation procedure. The authors randomly select 500 primes from the range $[1000, 20000]$ . The random selection ensures a distribution of number sizes within this range, avoiding biases toward particularly small or large primes. The lower bound of 1000 ensures that the numbers are large enough that factorization is non-trivial for a human (and for a transformer operating in a single step), while the upper bound of 20000 keeps the numbers within a range where the model might have memorized some factorizations from training data but certainly not all.

Why all examples are primes (correct answer always Yes). If the dataset included composite numbers (where the correct answer is No or "not prime"), an incorrect model response would say "Yes" (claiming the number is prime). The justification for "Yes, it is prime" would be something like "it has no divisors other than 1 and itself" — a statement that is much harder to verify as a snowballed hallucination because the model isn't making a specific false claim; it's making a general claim about the absence of factors. By making all 500 numbers prime, an incorrect response is always "No, it is not prime" followed by a concrete, false factorization that can be extracted and verified.

Expected incorrect justification form. When the model incorrectly claims a prime number is not prime, it typically provides an explicit factorization, such as:

"No, 9791 is not a prime number. It can be factored into $13 \times 751$ ."

The factorization $a \times b = n$ (where $n$ is the target prime) is the specific false claim. Both $a$ and $b$ are typically composable or prime factors, but crucially, at least one of them is not actually a divisor of $n$ , or their product does not equal $n$ . This gives a clear verification target: "Is $n$ divisible by $a$ ?" and "Is $n$ divisible by $b$ ?"

3.4.4 Dataset Construction: Senator Search

Data generation procedure. The authors consider all 50 U.S. states and a manually constructed list of twelve popular U.S. colleges (listed in Appendix A.2): MIT, University of Chicago, Johns Hopkins University, California Institute of Technology, Duke University, Northwestern University, Dartmouth College, Brown University, Vanderbilt University, Rice University, University of Washington — note that this is only 11 explicitly listed; the 12th is presumably omitted in the paper text but present in the actual dataset. For each possible (state, college) pair, they generate a question using the template:

"Was there ever a US senator that represented the state of $x$ and whose alma mater was $y$ ?"

They then manually remove all pairs where the answer is Yes (i.e., where there actually was a senator who represented state $x$ and attended college $y$ ). This manual curation step is essential: it ensures that every remaining question has the correct answer No, and therefore any model response claiming "Yes" is incorrect.

The college list was constructed by "taking a list of top universities in the U.S. and excluding from it universities which also appeared on The U.S. News & World Report's list of Top 10 Colleges for Members of Congress" (Appendix A.2). This exclusion criterion is deliberate: colleges that appear on the "Top 10 Colleges for Members of Congress" list are precisely those from which many senators graduated, making it more likely that a (state, college) pair would have a valid senator, which would need to be manually removed anyway. By excluding these colleges, the authors reduce the manual filtering burden.

Why all examples have correct answer No. Symmetric to the primality testing case: if the correct answer were Yes, the model's justification would name an actual senator, and verification would be about checking whether that senator's biography matches the claimed facts. While this is possible, it's less diagnostic for snowballing because the model might name a real senator with slightly incorrect details. By making all correct answers No, an incorrect response is always "Yes" with a named senator, and the verification question is whether that senator actually had the claimed attributes. Since no senator with those attributes exists, the verification is a clean yes/no test.

Expected incorrect justification form. When the model incorrectly claims a senator existed, it typically names a specific person and provides biographical details, as in Table 5:

"Yes, there was a U.S. Senator who represented the state of New Hampshire and whose alma mater was the University of Pennsylvania. His name is John P. Hale..."

The verifiable claims are: (1) Did John P. Hale represent New Hampshire as a U.S. senator? and (2) Was John P. Hale's alma mater the University of Pennsylvania? In this example from the paper, Hale did represent New Hampshire (claim 1 is true) but his alma mater was Bowdoin College, not University of Pennsylvania (claim 2 is false). The false claim is what gets verified.

A subtle point: the model might name a senator who actually represented the state but attended a different college (as in the Hale example), or it might name someone who was never a senator at all. Either case yields at least one false claim that can be tested in isolation.

3.4.5 Dataset Construction: Graph Connectivity

Data generation procedure. This is the most intricately constructed dataset. The authors design a single underlying directed graph structure with 14 nodes and 12 edges, which is used for all 500 questions. The graph structure is illustrated in Figure 5 and described in Appendix A.1. Critically, the graph is disconnected — it consists of two separate subgraphs with no path between them. The specific structure, as visible in Figure 5, has the following edge relationships (using the placeholder labels from the figure):

Node B → Node F, Node B → Node M
Node F → Node K, Node F → Node D
Node M → Node J, Node M → Node I
Node N → Node H, Node N → Node G
Node H → Node A, Node H → Node E
Node G → Node L, Node G → Node C

This structure consists of two disconnected components: {B, F, K, D, M, J, I} and {N, H, A, E, G, L, C}. There is no path from any node in the first component to any node in the second component, and vice versa.

Randomization for generalization. For each of the 500 questions, the authors randomly assign letters from the English alphabet (A through Z, but only 14 are used per instance) to name the nodes. This randomization ensures that the model cannot solve the task by memorizing specific letter-based patterns — each instance has different city labels, so the model must actually reason about the graph structure from the natural language description.

Question formulation. For each instance, the authors sample a source city $s$ and destination city $t$ from different subgraphs (so no path exists — the correct answer is No). Additionally, they impose the constraint that $s$ corresponds to a source node (a node with outgoing edges but no incoming edges from other sampled paths) and $t$ corresponds to a leaf or sink node. This ensures that simple one-step heuristics cannot be used to solve the problem — for example, checking whether there is a direct flight from $s$ to $t$ would be insufficient, and the lack of a direct flight doesn't imply lack of a multi-hop path. The model must consider the full graph structure.

The full prompt format is shown in Table 2. The prompt lists all 12 flights in natural language:

"Current flight information (the following flights are one-way only, and all the flights available are included below):

There is a flight from city F to city K There is a flight from city H to city A [10 other flight statements]

Question: Is there a series of flights that goes from city B to city E?"

Note two deliberate features: (1) the phrase "all the flights available are included below" makes explicit that the list is exhaustive — there are no implicit flights, which prevents the model from appealing to unspecified flights; (2) the flights are one-way, so directionality matters and the model cannot assume symmetry.

Why all examples have correct answer No. If some questions had correct answer Yes, the model's incorrect "No" response would not produce a specific false claim that can be verified — it would just assert that no path exists without providing a concrete false flight to check. By making all correct answers No, an incorrect response is always "Yes" with a listed sequence of flights, and the verification target is whether each claimed flight actually exists in the provided flight information.

Expected incorrect justification form. When the model incorrectly claims a path exists, it typically lists a multi-hop route, as in Table 3:

"Yes, there is a series of flights that goes from city F to city I. The route is as follows:

City F to City H

City H to City K

City K to City G

City G to City B

City B to City I"

The verifiable claims are each individual flight in the listed route. In this example from the paper, "City K to City G" is not a valid flight — it does not appear in the provided flight information. When queried in isolation, GPT-4 correctly states: "No, based on the above flight information, there is no direct flight from City K to City G."

3.4.6 First-Stage Inference and First-Token Commitment

Model configuration. All first-stage experiments use ChatGPT (gpt-3.5-turbo) and GPT-4 accessed via the OpenAI API with greedy decoding (temperature $t=0$ ). Greedy decoding means the model always selects the highest-probability token at each step, producing deterministic outputs for a given prompt. This choice is deliberate: it removes stochastic variation, making the results reproducible and ensuring that the model's commitment behavior is measured at its most "confident" setting — if the model commits to an incorrect answer under greedy decoding, it would likely also do so under most sampling strategies.

The critical role of first-token commitment. The paper hypothesizes that snowballing is enabled by the model immediately stating its final answer (Yes or No) before explaining. The empirical finding in Section 2 validates this hypothesis strongly:

"the first token is Yes or No 95.67% and 98.40% of the time for GPT-4 and ChatGPT respectively."

This means that in over 95% of cases, the model produces the answer to a complex reasoning problem in a single forward pass of the transformer. For primality testing and graph connectivity, this is provably insufficient (per Merrill and Sabharwal, 2023). Once "Yes" or "No" is generated, that token enters the model's context window, and the model's training objective (predicting coherent continuations) exerts pressure to produce text consistent with that commitment. The model cannot "take back" the answer — it must justify it.

How accuracy is determined. The paper evaluates the model's response by examining whether the output begins with Yes or No. In cases where the response does not fall into these categories (the remaining ~4% for GPT-4 and ~2% for ChatGPT), human annotators "manually determine the answer conveyed by the model" (Section 3.2). This manual step handles edge cases like "There is no record of a U.S. Senator..." (which would be classified as No) or other circumlocutions.

3.4.7 Claim Extraction Pipeline

After the first-stage inference identifies which questions the model answered incorrectly, the claim extraction pipeline isolates the specific false claim from the model's justification. The extraction procedure is tailored to each dataset and exploits the predictable form of incorrect justifications.

Primality testing extraction. The paper uses a separate call to ChatGPT (not GPT-4, "for its fast inference speed") with a one-shot demonstration to extract the factors. The prompt contains the original model output and asks: "What are the factors proposed in the above text? List them out." (Section 3.3). For example, if the model output says "9791 can be factored into $13 \times 751$ ", the extraction call returns the pair $(13, 751)$ .

The authors manually checked 30 extraction results and found that ChatGPT "can always extract the correct factors" (Section 3.3), establishing that this automated step is reliable enough that manual verification of all 500 examples is unnecessary.

Senator search extraction. Similarly, the paper uses ChatGPT with a prompt containing the original model output and asking: "What is the senator mentioned in the above text? Just give the name" (Section 3.3). Manual inspection of 30 examples confirmed perfect extraction accuracy.

Graph connectivity extraction. This is the only dataset where extraction is done manually rather than automatically. For each sample where the model thinks there is a path (answer starts with Yes), human annotators "manually extract the list of flights from the model's output and identify the invalid or discontinuous flights" (Section 3.3). The manual approach is necessary because the model's flight sequences are less structured than factorization pairs or senator names — the model might list flights in various formats, include intermediate commentary, or produce multi-hop routes where the error could be at any step. The paper does not report an inter-annotator agreement metric for this manual extraction, which is a minor methodological limitation.

3.4.8 Verification Stage: Isolated Query Construction

The verification stage is the core diagnostic procedure that distinguishes snowballed hallucinations (false claims the model would recognize as false in isolation) from persistent hallucinations (false claims the model consistently endorses regardless of context).

New session requirement. The verification question is posed in a completely separate conversation session from the original question. This is absolutely critical: if the verification were in the same conversation, the original context (including the model's earlier incorrect answer and justification) would still be present in the context window, potentially causing the model to be consistent with its earlier output rather than answering based on its knowledge. The new session ensures that the model sees only the isolated claim and must evaluate it without any coherence pressure from prior self-generated text.

Primality testing verification. For each extracted factor pair $(a, b)$ from the claim " $n = a \times b$ ", the paper constructs one verification question per factor:

"Is $n$ divisible by $a$ ? Answer with either Yes or No."

The prompt explicitly constrains the answer format to "Yes or No" to make evaluation straightforward. If the model answers "No" (correctly recognizing that $a$ does not divide $n$ ), the claim is classified as a snowballed hallucination. If the model answers "Yes" (incorrectly endorsing the false divisibility), it is classified as a persistent hallucination — a genuine knowledge gap, at least for that specific fact.

Table 4 shows a concrete example: GPT-4 initially claims $9791 = 13 \times 751$ . The verification questions are "Is 9791 divisible by 13?" (GPT-4 answers No) and "Is 9791 divisible by 751?" (GPT-4 answers No). Both factors are correctly rejected, establishing that both claims were snowballed hallucinations.

Senator search verification. For each extracted senator name, the paper constructs two verification questions:

"Has [senator name] represented the state of [state] as US senator?" "Was [senator name]'s alma mater [college]?"

The paper does not specify an explicit answer format constraint for these questions in the prompt (unlike primality testing), based on the examples in Table 5. The verification answers are manually assessed by human annotators to determine whether the model correctly identifies the false claim.

The paper notes an important nuance: it's possible for the model to answer the first verification question correctly (Yes, the senator did represent that state) while answering the second incorrectly or correctly. Only the false claim(s) in the original justification need to be verified. In the John P. Hale example (Table 5), Hale did represent New Hampshire (verification Q1 returns Yes — the model is correct, and this particular claim in the original justification was true), but his alma mater was Bowdoin, not University of Pennsylvania (verification Q2 returns No — the model correctly identifies this as false). The original answer was wrong because no senator with both attributes exists; the model's justification contained one true claim and one false claim, and the false claim is the snowballed hallucination.

Graph connectivity verification. For each invalid or discontinuous flight identified in the manual extraction, the paper constructs a verification question by re-presenting the full flight information (to give the model the same factual context it had originally) and asking about the specific flight:

"Current flight information (the following flights are one-way only, and all the flights available are included below): [all 12 flight statements repeated]

Based on the above flight information, is City K to City G a valid flight?"

The flight information is repeated verbatim to ensure the model has access to the same factual basis. The verification question is formatted as a yes/no question, and the model's response is manually assessed. Table 3 shows GPT-4 correctly identifying that "City K to City G" is not a valid flight, establishing it as a snowballed hallucination.

Manual evaluation for verification. The paper states that they "manually assess the verification output to check if the model correctly detects the error" (Section 3.3). This means that, unlike the accuracy evaluation in the first stage (which is largely automated through first-token checking), the verification stage relies on human judgment to determine whether the model's response constitutes correct detection of the error. The paper does not report inter-annotator agreement metrics for this assessment, though the yes/no format of the verification questions makes the assessment relatively straightforward — the human annotator checks whether the model said "No" (correct detection) or "Yes" (failure to detect) in response to whether a specific claim is valid.

3.4.9 Chain-of-Thought Prompting Variant

Section 4.1 introduces a variant of the first-stage inference where the prompt is modified by appending "Let's think step by step" to the original question. This is evaluated as a potential mitigation strategy and serves as a diagnostic for whether improved reasoning depth eliminates snowballing.

Prompt modification. For each dataset, the paper takes the original question format (shown in Table 1) and appends the phrase "Let's think step by step" at the end. No other changes are made to the prompt. This is the zero-shot chain-of-thought approach introduced by Kojima et al. (2023), which has been shown to improve reasoning performance across diverse tasks.

Evaluation differences from direct prompting. Because the chain-of-thought output is less structured than the direct answer format, the paper manually inspects the outputs to "determine correctness and the presence of snowballed hallucinations" (Section 4.1). The manual inspection involves two judgments: (1) did the model arrive at the correct final answer? (2) if the final answer is incorrect, does the reasoning chain contain specific false claims that would constitute snowballed hallucinations?

For the snowballed hallucination assessment, the same verification procedure is applied: extract the specific false claim from the reasoning chain and query the model in a separate session to determine whether it recognizes the claim as false. However, the extraction is manual rather than semi-automated, since chain-of-thought reasoning chains are less formulaic in structure.

Example of chain-of-thought snowballing. The paper provides a concrete example (Section 4.1) where ChatGPT, while using chain-of-thought on the graph connectivity task, produces the following reasoning step:

"Step 3: From city E, we have three options: a flight to city N, a flight to city B, or a flight to city C."

The paper notes that there are actually only two options (the flight to city C does not exist), and this is a snowballed hallucination — when queried in a separate session, ChatGPT can verify that "E → C" is not a valid flight. This demonstrates that even in a step-by-step reasoning framework, the model can generate false intermediate claims that it would not endorse independently.

3.4.10 Temperature and Sampling Experiments

Section 4.2 reports experiments varying the decoding temperature $t$ at values of 0.0 (greedy), 0.6, and 0.9.

How temperature is varied. The temperature parameter $t$ controls the sharpness of the softmax distribution over the vocabulary at each generation step. At $t=0$ (the limit, implemented as greedy decoding), the model deterministically selects the highest-probability token. At higher $t$ , probability mass is spread more evenly across tokens, reducing the probability of the most likely token and increasing the probability of alternatives. The paper explicitly describes this:

"During decoding, the temperature $t$ controls the sharpness of the output distribution, with higher $t$ spreading probability mass away from the model's most likely prediction for each next word."

Why these temperature values. The paper tests $t=0.6$ and $t=0.9$ as moderate and high temperature settings. These are common hyperparameter choices in the literature, though the paper does not justify the specific values beyond their use as comparison points to greedy decoding.

Expected effect (or lack thereof). The paper argues that higher temperatures should not reduce first-token commitment because the model tends to produce Yes/No answers regardless of the sampling distribution — the issue is the model's learned behavior of answering before explaining, not the determinism of the decoding process. The results in Tables 10 and 11 confirm this: error rates and snowballed hallucination rates remain "similarly high" across all temperatures tested.

Top-k and nucleus sampling. The paper provides a brief theoretical argument (not empirically tested because the OpenAI API does not expose these options, or because beam search specifically is unavailable): top-k and nucleus sampling "only narrow the range of tokens to be considered, and thus can only increase the probability that the model will immediately commit to an answer." This is because these methods restrict the sampling distribution to a subset of high-probability tokens — if Yes/No are already the most probable tokens (as they are, per the 95%+ first-token commitment rate), restricting to the top-k or nucleus set would make Yes/No even more likely, not less.

Beam search as a potential but untestable fix. The paper presents a theoretical argument for why beam search could alleviate snowballing:

"if some sequences in the beam after the initial token do not commit to an answer (or commit to the right answer), their continuations may eventually have higher probability than those that initially commit incorrectly and later produce incorrect reasoning as a result"

The intuition is that beam search maintains multiple candidate sequences at each timestep. If the beam includes a sequence that doesn't immediately commit to Yes/No (or commits to the correct answer), the incoherence of incorrect justifications might cause the correct sequence to eventually receive higher probability than the incorrect-but-initially-confident sequence. However, this hypothesis cannot be tested because "the OpenAI API does not support beam search" (Section 4.2). This is presented as a limitation and a potential direction for future research on open-weight models.

3.4.11 Design Choices and Their Justifications

Why proprietary models (ChatGPT and GPT-4) rather than open-weight models? The paper explicitly states that ChatGPT and GPT-4 were chosen "due to their state-of-the-art performance on many benchmarks" (Limitations section). The practical rationale is that these are the most widely deployed and influential models, making their failure modes particularly consequential to document. The tradeoff is limited experimental control: the authors cannot access output probability distributions, cannot fine-tune, and cannot test beam search or other decoding strategies that require probability access. The paper acknowledges this limitation transparently.

Why zero-shot rather than few-shot prompting? The paper uses zero-shot prompts to reflect "the most common way users interact with LMs" (Section 3.2). Few-shot prompting might improve accuracy, but the goal is not to maximize performance — it is to document a failure mode under realistic usage conditions. Additionally, few-shot examples might influence the model's commitment behavior in ways that obscure the phenomenon being studied.

Why greedy decoding as the primary setting? Greedy decoding ( $t=0$ ) ensures reproducibility and represents the model's behavior at its most "confident." If the model commits to an incorrect answer under greedy decoding, this reflects its highest-probability behavior. The temperature experiments ( $t=0.6, 0.9$ ) show that stochastic decoding does not qualitatively change the results, validating the choice.

Why fix the correct answer to a single label? This is perhaps the most clever design choice in the paper. By ensuring that all correct answers in each dataset are the same (Yes for primality, No for the other two), every incorrect response follows a predictable pattern, enabling systematic extraction and verification of false claims. Without this fixed-label design, the form of incorrect justifications would vary (some claiming Yes, some claiming No), making automated extraction much harder and manual evaluation far more labor-intensive.

Why these three specific tasks? The three tasks span different cognitive demands: primality testing and graph connectivity are inherently sequential reasoning problems (provably so, per Merrill and Sabharwal, 2023), while senator search is primarily a factual knowledge retrieval task. This diversity shows that snowballing is not limited to one type of reasoning. Additionally, all three tasks produce incorrect justifications that decompose into simple, verifiable atomic claims, which is essential for the verification methodology.

Why 500 examples per dataset? The paper does not explicitly justify the sample size, but 500 provides reasonable statistical precision while keeping manual evaluation (for graph connectivity extraction and verification) tractable. Error rates of 60-83% on 500 examples correspond to hundreds of incorrect responses per dataset, providing sufficient data for the verification analysis.

Why manual evaluation for verification rather than automated scoring? The verification outputs are in natural language and can take various forms — the model might say "No, that is not a valid flight" or "No, based on the above flight information, there is no direct flight from City K to City G" or other variations. Automated string matching would be brittle. Manual evaluation ensures that the human judgment of whether the model recognized the error is accurate, though it introduces some subjectivity that the paper does not quantify with inter-annotator agreement metrics.

4. Key Insights and Innovations

Innovation 1: Reframing Hallucination as a Coherence-Induced Phenomenon, Not Just a Knowledge Gap Problem

The paper's most fundamental conceptual contribution is a diagnostic reframing of why language models hallucinate. The dominant paradigm in the field—articulated most directly by Zheng et al. (2023) and motivating the entire retrieval-augmented generation research program (Lewis et al., 2020; Shuster et al., 2021)—treats hallucination as a knowledge deficit: the model generates false statements because it lacks the relevant facts. The mitigation strategy follows naturally from this framing: supplement the model's knowledge, either through larger training corpora or through external retrieval at inference time.

This paper demonstrates that this framing is incomplete in a way that matters for designing interventions. The finding that ChatGPT and GPT-4 can identify 67% and 87% of their own hallucinated claims respectively when those claims are presented in isolation (Tables 6 and 7) is not just an empirical curiosity—it establishes the existence of a second, distinct mechanism for hallucination that operates even when the model possesses the relevant knowledge. The model "knows" that 13 does not divide 9677 (it says so when asked directly), yet it generates the claim that $13 \times 745 = 9677$ when that claim is needed to justify its earlier incorrect answer.

This is not an incremental refinement of the knowledge-gap view. It is a categorical distinction between two failure modes with fundamentally different root causes and, consequently, fundamentally different mitigation requirements:

Knowledge-gap hallucinations arise from insufficient or incorrect information encoded in the model's parameters. These can be addressed by providing access to external knowledge at inference time or by improving the training data.
Coherence-induced (snowballed) hallucinations arise from the autoregressive generation process itself: the model commits to an output early, that output becomes conditioning context, and the training objective's pressure toward coherence overrides the model's ability to access contradictory knowledge encoded in its parameters.

The practical implication is that retrieval-augmented generation and similar knowledge-supplementation strategies cannot address snowballed hallucinations because the bottleneck is not knowledge access but the model's inability to override context-driven generation with stored knowledge. For a retrieval system to help, it would need to intercept the model not just at the prompt level but mid-generation, detecting when the model has committed to a wrong answer and correcting the context before subsequent tokens are generated—a substantially harder engineering problem.

This reframing also explains a puzzling pattern in the hallucination literature: why some studies find that LMs "cannot self-correct" (Huang et al., 2023) while others find that self-reflection helps (Madaan et al., 2023). If some hallucinations are coherence-induced and others are knowledge-gap-driven, the effectiveness of self-correction would depend on which type dominates in a given experimental setup—a previously unrecognized confound.

The paper grounds this reframing in the concrete, measurable diagnostic of isolated claim recognition: a hallucination is classified as snowballed if and only if the model can correctly identify it as false when the claim is extracted from its original justifying context and presented in a new session with no access to the prior conversation. This operationalization converts a fuzzy conceptual distinction into a replicable experimental procedure, which is a methodological contribution in its own right.

Innovation 2: Connecting Autoregressive Architectural Limitations to the Quality (Not Just Accuracy) of Generated Output

Prior theoretical work (Merrill and Sabharwal, 2023) established that bounded-precision transformers cannot solve problems outside $TC^0$ in a single generation step—a result about the accuracy limits of the architecture. The standard interpretation is: on inherently sequential problems, transformers will sometimes get the answer wrong because one forward pass is computationally insufficient.

This paper makes a qualitatively different use of that theoretical result. It asks not just whether the model gets the answer wrong, but what happens next when it does. The finding is that the model does not simply produce a wrong answer and stop—it constructs an elaborate, fluent, and internally coherent justification for its wrong answer, and that justification contains specific false claims that the model would separately reject. The connection to the theoretical result is:

The architectural limitation (no single-step solution for primality testing or graph connectivity) means the model must guess when the prompt format elicits an immediate yes/no answer. This is a predictable consequence, not a random failure.
Once the guess token (Yes/No) enters the context, the autoregressive training objective exerts pressure to generate text consistent with that commitment.
The resulting justification is not random noise or a vague hedge—it is a specific, detailed, and confident fabrication that the model's own knowledge (accessible through a separate forward pass with a clean context) contradicts.

This transforms the theoretical result from Merrill and Sabharwal (2023) into a predictive framework for output quality degradation. The paper is not merely observing that transformers make mistakes on hard problems (which is unsurprising) but identifying a specific mechanism through which those mistakes induce further, more basic errors that compound the original failure. The claim " $13 \times 745 = 9677$ " is not a subtle error at the boundary of the model's reasoning capacity—it is a simple arithmetic falsehood that the model can verify as incorrect. The error arises not from the difficulty of the arithmetic but from the context in which the arithmetic is generated.

This insight has implications for how we evaluate and interpret LM outputs beyond simple accuracy metrics. A model might achieve the same accuracy on a task under two different prompting strategies, but the nature of its errors could be fundamentally different—one strategy might produce knowledge-gap errors (the model consistently believes a falsehood) while another might produce snowballed errors (the model generates claims it contradicts when asked differently). Standard accuracy metrics collapse this distinction, but it matters enormously for trustworthiness: a knowledge-gap error is a stable limitation of the model, while a snowballed error is an artifact of the generation context that could potentially be avoided through different interaction design.

The senator search task provides an illuminating boundary case. Unlike primality testing and graph connectivity, senator search is fundamentally a knowledge retrieval task, not a sequential reasoning task—the model needs to recall whether a senator with specific attributes exists, not perform multi-step computation. The fact that snowballing still occurs on this task (68.6% of ChatGPT's incorrect claims and 74.3% of GPT-4's were snowballed; Table 7) demonstrates that the phenomenon is not limited to problems that are provably outside $TC^0$ . Any situation where the model commits to an answer before retrieving and verifying the relevant facts creates the conditions for snowballing. This broadens the relevance of the finding beyond the specific computational complexity argument.

Innovation 3: Identifying a Residual Failure Mode of Chain-of-Thought Reasoning That Prior Work Overlooked

Chain-of-thought prompting has been widely celebrated as a method for improving LM reasoning by allowing models to break complex problems into manageable intermediate steps (Nye et al., 2021; Wei et al., 2022). The standard narrative—supported by extensive empirical evidence—is that generating intermediate reasoning tokens enables the model to allocate more computation to the problem, effectively simulating multi-step reasoning within the autoregressive framework. The dramatic accuracy improvements in this paper's own experiments (from error rates of 60–83% down to 4–9% on average; Tables 6 and 8) are fully consistent with this narrative.

The paper's distinctive contribution is not confirming that chain-of-thought improves accuracy (which was already well-established) but rather documenting a specific, systematic failure mode that survives chain-of-thought prompting: when the model makes an error in its reasoning chain, that error becomes conditioning context for subsequent reasoning steps, inducing downstream hallucinations that the model would not generate in isolation.

The numbers are striking. Despite chain-of-thought reducing GPT-4's overall error rate to ~4%, 94.90% of those remaining errors contain snowballed hallucinations (Table 9). In other words, when chain-of-thought fails, it fails in a characteristic way—not by producing a single isolated mistake, but by producing a cascade of errors where an early misstep induces later false claims that compound the failure.

This is a negative result with significant implications. It means that chain-of-thought improves the model's ability to start reasoning correctly but does not provide any mechanism for recovering from an intermediate error once made. The error propagates forward through the context window, and the model's coherence training ensures that later steps are consistent with the erroneous earlier step, even when that consistency requires generating false statements. The concrete example in Section 4.1—ChatGPT claims three flight options from city E when only two exist, then generates a snowballed hallucination about the nonexistent third option—illustrates the mechanism precisely.

This finding complicates the optimistic narrative around chain-of-thought and related reasoning methods. The field has largely focused on maximizing the probability that each reasoning step is correct, implicitly assuming that if enough steps are right, the overall answer will be right. The paper shows that this assumption is questionable: the correctness of later steps is not independent of the correctness of earlier steps, because errors corrupt the context that conditions subsequent generation. This suggests that future work on reasoning should focus not just on step-level accuracy but on error recovery and backtracking mechanisms—the ability to recognize when an intermediate step was wrong and revise it before proceeding. The paper notes that GPT-4 occasionally does this spontaneously ("we have indeed observed GPT-4 doing this in a limited number of cases"; Section 1), but it is clearly not a reliable behavior under current training.

The senator search task provides a telling contrast that sharpens this insight. With chain-of-thought prompting, both models achieve perfect accuracy (0% error rate; Table 8). This suggests that the snowballing problem on senator search with direct prompting was caused entirely by premature commitment—the model could in fact retrieve the correct answer if it took the time to reason through the necessary factual checks before committing. For this task, chain-of-thought eliminates snowballing not by improving step-level accuracy per se, but by restructuring the generation order so that the model retrieves facts before stating its conclusion. This implies that the key benefit of chain-of-thought for avoiding snowballing is not deeper computation but delayed commitment—a subtly different interpretation than the standard "more computation" framing.

Innovation 4: The Self-Contradiction Diagnostic as a General Method for Distinguishing Knowledge Gaps from Contextual Coercion

Beyond the specific empirical findings about hallucination snowballing, the paper introduces a general experimental methodology for determining whether a given model output is produced because the model lacks knowledge or because the model's generation process overrides accessible knowledge. The method is deceptively simple: extract the specific claim from the model's output, present it to the same model in a clean context (new session, no access to the original generation), and check whether the model endorses or rejects the claim.

This diagnostic is powerful because it provides a causal test for the mechanism underlying a hallucination:

If the model rejects the claim in isolation → the original hallucination was coherence-induced (snowballed). The model possesses the knowledge to evaluate the claim correctly, but that knowledge was overridden by context pressure during the original generation.
If the model endorses the claim in isolation → the hallucination is a genuine knowledge gap (or at least a stable false belief). The model consistently produces the false claim regardless of context, suggesting the error is encoded in its parameters rather than induced by generation dynamics.

Prior work on LM consistency (Lin et al., 2022; Kim et al., 2021, 2022) had shown that LMs give different answers to the same question when the surrounding context changes, but these studies primarily demonstrated that LMs are inconsistent without providing a mechanism or a diagnostic for why the inconsistency occurs. The snowballing framework provides both: the mechanism is autoregressive coherence pressure, and the diagnostic is the isolated-claim recognition test.

The diagnostic is also practically significant because it can be applied without access to model internals—no need for probability distributions, hidden states, or training data. The verification questions use the same API as the original generation, making the method applicable to black-box models like GPT-4 that dominate real-world deployment. This is important because the most widely used models are also the ones whose failure modes are least transparent and most consequential to understand.

The paper's application of this diagnostic to three different domains (mathematical reasoning, factual knowledge retrieval, and graph reasoning) demonstrates its generality. The consistent finding—that 67–87% of incorrect claims are recognized as false in isolation—suggests that coherence-induced hallucination is a systematic and pervasive failure mode, not a domain-specific curiosity. This makes the diagnostic a candidate for inclusion in standard LM evaluation suites: rather than just measuring whether a model gets answers right, evaluators could measure what fraction of the model's errors are self-recognized, providing a more nuanced picture of the model's reliability.

However, the diagnostic has an important limitation that the paper acknowledges only implicitly. The verification stage requires the model to answer a yes/no question about a specific factual claim, and the model might answer incorrectly in the verification stage for reasons unrelated to whether it "knows" the correct answer—for example, the verification prompt format might be confusing, or the model might be inconsistent in its answers for stochastic reasons (though greedy decoding minimizes this). The paper's manual evaluation of verification outputs partially addresses this by allowing human judgment of whether the model genuinely detected the error, but the fundamental assumption—that a correct rejection in isolation implies the original hallucination was coherence-driven—relies on the model's verification-stage behavior being a reliable indicator of its stored knowledge. This assumption is plausible but not independently verified.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper constructs three custom question-answering datasets, each containing exactly 500 yes/no questions, designed to probe hallucination snowballing behavior. Primality Testing: 500 randomly chosen prime numbers between 1,000 and 20,000, where the correct answer is always Yes. Senator Search: 500 questions of the form "Was there ever a US senator that represented the state of $x$ and whose alma mater was $y$ ?" where all (state, college) pairs with correct answer Yes are manually removed, leaving only No-answer pairs. The college list comprises 12 top U.S. universities excluding those on the U.S. News & World Report's Top 10 Colleges for Members of Congress list (Appendix A.2). Graph Connectivity: 500 questions based on a single underlying directed graph structure with 14 nodes and 12 edges, designed to be disconnected (two subgraphs with no path between them). For each instance, node labels are randomly reassigned from the English alphabet, and source and target cities are sampled from different subgraphs with the constraint that the source is a source node and the target is a leaf node, ensuring the correct answer is always No. All three datasets are newly constructed for this paper and are not drawn from existing benchmarks.
Base model(s). All experiments use two proprietary models accessed via API: ChatGPT (gpt-3.5-turbo) and GPT-4. The paper states these were chosen "due to their state-of-the-art performance on many benchmarks" (Limitations section) and because they are the most widely deployed LMs, making their failure modes particularly consequential. No open-weight models are tested. The models are queried with greedy decoding ( $t=0$ ) as the primary setting, making outputs deterministic and reproducible. Temperature variants at $t=0.6$ and $t=0.9$ are tested in Section 4.2 as robustness checks.
Metrics. Two primary metrics are reported across all experiments:
- Error rate (Error rate): the fraction of the 500 questions in each dataset for which the model produces an incorrect answer. This is computed by examining whether the model's output begins with Yes or No and comparing against the ground-truth label (always Yes for primality, always No for the other two). In cases where the output does not begin with Yes or No (approximately 4% for GPT-4 and 2% for ChatGPT), human annotators manually determine the conveyed answer.
- Snowballed hallucination rate (Snowballed hallucination rate): the fraction of incorrect model responses where the model's justification contains at least one false claim that the model can correctly identify as false when presented in isolation in a separate conversation session. This is computed on the subset of questions that the model answered incorrectly, not on the full 500-example dataset. The paper also reports an overall hallucination rate relative to the full dataset by multiplying the error rate and the snowballed rate.
- For the chain-of-thought prompting experiments (Section 4.1), both metrics are computed through manual inspection of the model's outputs due to the less structured nature of chain-of-thought reasoning chains.
Baselines. There is no external baseline model or method against which to compare, as the paper is a diagnostic study rather than a method-proposing paper. The "baseline" is the model's own behavior under the direct zero-shot prompt (without "Let's think step by step"). The key comparison is not between different methods but between the model's behavior within the original justification context (where it generates false claims) and in isolation (where it evaluates those same claims). The verification stage in a new session serves as the reference point for establishing snowballed hallucination. Within the mitigation experiments (Section 4), the chain-of-thought prompting variant (Kojima et al., 2023) and varying decoding temperatures serve as comparison conditions to the direct-prompt greedy-decoding default.
Generation budget / compute accounting. The paper does not use a generation budget in the traditional sense—no search, sampling, or revision budget is allocated per question. Each question receives a single model call with greedy decoding (or the specified temperature setting). The only resource consumed is one API call per question in the first stage, plus one verification call per extracted false claim in the second stage. The verification stage is purely diagnostic and does not influence the model's original response. For the 2048-sample difficulty estimation or beam-search-style comparisons found in compute-optimal scaling literature, no analogue exists here—this is a measurement study with a fixed per-question cost.
Cross-validation / statistical protocol. There is no train/validation/test split, no cross-validation, and no statistical significance testing reported, as the paper does not train models or tune hyperparameters. The 500 examples per dataset serve as a single evaluation set. Manual evaluation is used for: (1) determining the answer conveyed by the model when the output does not begin with Yes/No (applicable to ~2–4% of cases); (2) extracting invalid or discontinuous flights from graph connectivity justifications (the only dataset where extraction is fully manual rather than semi-automated); (3) evaluating whether the model correctly detected the error in verification outputs across all three datasets. The paper does not report inter-annotator agreement metrics for any of these manual judgments, which is a methodological limitation. For the semi-automated extraction steps (primality factor extraction and senator name extraction using ChatGPT with one-shot prompts), the paper manually verified 30 examples each and reports perfect extraction accuracy, establishing these steps as reliable without evaluating all 500 examples.

Main Quantitative Results

Direct Prompting: Error Rates and Snowballed Hallucination Rates

The headline results appear in Figure 2, with precise numerical breakdowns in Tables 6 and 7. Under the direct zero-shot prompt (no chain-of-thought, just the question as shown in Table 1), both models exhibit very high error rates across all three tasks:

ChatGPT's overall error rate averages 60.13% across the three datasets (Table 6): 82.0% on Graph Connectivity (410/500 errors), 67.8% on Primality Testing (339/500), and 30.6% on Senator Search (153/500).
GPT-4's overall error rate averages 83.40% (Table 6), substantially worse than ChatGPT: 88.4% on Graph Connectivity (442/500), 74.8% on Primality Testing (374/500), and 87.0% on Senator Search (435/500). The paper notes that GPT-4 performs worse "despite popularly being considered superior to ChatGPT" (Section 3.4).

GPT-4's worse performance is a non-obvious finding. The paper does not provide a definitive explanation, but the pattern is consistent across all three tasks, suggesting that GPT-4's stronger instruction-following or more verbose answering style may make it more prone to committing to answers before verifying them, or that its higher fluency produces more elaborate justifications that are more likely to contain specific false claims detectable by the verification pipeline.

Hallucination detection (the snowballing measurement). Among the incorrect responses, a large fraction of the justifications contain claims that the model can recognize as false in isolation (Table 7):

ChatGPT's snowballed hallucination rate: 67.37% average across the three datasets. Broken down by task: 96.6% on Graph Connectivity (396/410 of incorrect responses contain snowballed claims), 36.9% on Primality Testing (125/339), and 68.6% on Senator Search (98/153).
GPT-4's snowballed hallucination rate: 87.03% average. By task: 94.3% on Graph Connectivity (417/442), 92.5% on Primality Testing (346/374), and 74.3% on Senator Search (323/435).

The interpretation of these numbers requires careful attention to the denominator. The snowballed hallucination rate is computed over incorrect responses, not over all 500 questions or over all hallucinations. For example, GPT-4's 92.5% on Primality Testing means that of the 374 questions it answered incorrectly, 346 of those incorrect answers were justified with at least one false claim that GPT-4 could recognize as false in isolation. The remaining 28 incorrect responses (7.5% of incorrects) contained false claims that GPT-4 failed to recognize in isolation—these are the cases where the hallucination appears to be a genuine knowledge gap (or at least a persistent false belief) rather than a coherence-induced fabrication.

When expressing the snowballed hallucination rate relative to the full dataset (all 500 questions), the numbers become a product of the error rate and the snowballing rate. For GPT-4 on Primality Testing: 74.8% error rate × 92.5% snowballing rate ≈ 69.2% of all questions produce snowballed hallucinations. This means that for this task, nearly 70% of all queries result in the model generating a false claim it can recognize as false—a striking prevalence.

Cross-task variation. The pattern of snowballing rates differs by task in revealing ways:

Graph Connectivity shows the highest snowballing rates for both models (96.6% for ChatGPT, 94.3% for GPT-4). This is consistent with the theoretical expectation: graph connectivity is L-complete and provably outside $TC^0$ , so the model must guess on the first token, the guess is usually wrong (82–88% error rate), and the subsequent justification consistently contains fabricated flight connections that the model can identify as invalid in isolation.
Primality Testing shows a large gap between the two models' snowballing rates: ChatGPT snowballs on only 36.9% of its incorrect responses, while GPT-4 snowballs on 92.5%. This means ChatGPT's primality testing errors are mostly persistent hallucinations—when ChatGPT claims a number has a particular factorization, it often cannot recognize that factorization as false in isolation, suggesting its arithmetic knowledge is genuinely unreliable. GPT-4, by contrast, snowballs on almost all its primality errors—it provides incorrect factorizations that it subsequently recognizes as wrong. This is consistent with GPT-4 having stronger arithmetic capabilities than ChatGPT (enabling recognition) but being equally susceptible to the initial commitment problem (answering "No" before verifying primality).
Senator Search shows the lowest error rate for ChatGPT (30.6%) but the highest for GPT-4 (87.0%). Among incorrect responses, snowballing rates are moderate: 68.6% for ChatGPT, 74.3% for GPT-4. The cases that are not snowballed (31.4% and 25.7% respectively) are instances where the model names a senator and, when queried in isolation, still incorrectly maintains that the senator had the claimed attributes—these are genuine knowledge-gap hallucinations in the factual retrieval domain.

First-token commitment rate. The paper reports (Section 2) that "the first token is Yes or No 95.67% and 98.40% of the time for GPT-4 and ChatGPT respectively." This is the prerequisite condition for snowballing: the model commits to an answer in a single forward pass before it has the opportunity to reason through the justification. The slightly lower commitment rate for GPT-4 (95.67%) reflects that GPT-4 occasionally begins with hedging phrases, though the paper notes that even in these cases, "the model often commits to an answer within the first few tokens of the response (e.g., 'There is no record of a U.S. Senator...')" (Section 2). This quasi-commitment is functionally identical to an explicit Yes/No for snowballing purposes, as it establishes a direction for the subsequent justification.

Chain-of-Thought Prompting: Dramatic Accuracy Gains, Persistent Snowballing on Remaining Errors

The chain-of-thought prompting experiments (Section 4.1, Tables 8 and 9) use the same 500 questions per dataset with "Let's think step by step" appended to the original prompt. The results reveal a nuanced picture: large accuracy improvements but a characteristic failure mode on remaining errors.

Error rates with chain-of-thought (Table 8):

ChatGPT's average error rate drops from 60.13% to 9.40%: 27.8% on Graph Connectivity (139/500), 0.4% on Primality Testing (2/500), and 0.0% on Senator Search (0/500 — perfect accuracy).
GPT-4's average error rate drops from 83.40% to 3.87%: 4.2% on Graph Connectivity (21/500), 7.4% on Primality Testing (37/500), and 0.0% on Senator Search (0/500).

The perfect accuracy on Senator Search is particularly informative. It confirms that both models do possess the factual knowledge needed to answer these questions correctly—they simply fail to access or apply that knowledge under the direct prompt, where the answer is demanded before the factual retrieval can be performed. Chain-of-thought restructures the generation order so that factual checks precede the commitment to an answer, and this alone eliminates all errors on this task.

Snowballed hallucination rates on remaining chain-of-thought errors (Table 9):

ChatGPT: Among the 139 remaining errors on Graph Connectivity, 88.5% (123/139) contain snowballed hallucinations. On Primality Testing, only 2 errors remain (0/2 or 0% snowballing on a near-empty set). Senator Search has no errors to analyze.
GPT-4: Among the 21 errors on Graph Connectivity, 95.2% (20/21) contain snowballed hallucinations. Among the 37 errors on Primality Testing, 94.6% (35/37) contain snowballed hallucinations. The overall average snowballing rate on remaining errors is 94.90%, comparable to the 87.03% observed under direct prompting.

The critical finding here is that chain-of-thought does not eliminate snowballing on the errors that remain—it is nearly as prevalent within the reduced error set. The paper provides a concrete example (Section 4.1) of how this occurs: ChatGPT, while reasoning step-by-step about graph connectivity, generates "Step 3: From city E, we have three options: a flight to city N, a flight to city B, or a flight to city C" when in fact there are only two options (the flight to C does not exist). This erroneous step then becomes part of the context, and subsequent steps build on it, potentially introducing further snowballed hallucinations.

This demonstrates that chain-of-thought's benefit is primarily in reducing the number of initial errors (by delaying commitment and enabling more accurate computational steps) rather than in preventing error propagation from intermediate mistakes. When an intermediate step is wrong, the mechanism of snowballing—coherence pressure from self-generated context—operates just as forcefully as in the direct prompt setting.

Temperature Variation: No Substantial Effect on Either Error Rate or Snowballing

The temperature experiments (Section 4.2, Tables 10 and 11) test $t = 0.0$ (greedy), $t = 0.6$ , and $t = 0.9$ across all three datasets for both models.

Error rates across temperatures (Table 10):

ChatGPT: Average error rates remain nearly constant: 60.13% at $t=0.0$ , 58.53% at $t=0.6$ , 58.53% at $t=0.9$ . The maximum deviation from greedy is 1.6 percentage points on Primality Testing (67.8% → 62.0% at $t=0.6$ ).
GPT-4: Average error rates also show minimal variation: 83.40% at $t=0.0$ , 82.53% at $t=0.6$ , 81.67% at $t=0.9$ . The maximum deviation is 3.4 percentage points on Primality Testing (74.8% → 73.0% at $t=0.6$ ).

Snowballed hallucination rates across temperatures (Table 11):

ChatGPT: 67.37% at $t=0.0$ , 66.77% at both $t=0.6$ and $t=0.9$ .
GPT-4: 87.03% at $t=0.0$ , 86.13% at $t=0.6$ , 84.87% at $t=0.9$ .

The variation across temperatures is minimal, and there is no consistent direction of effect. The paper's theoretical argument for why temperature should not help (Section 4.2) is that the model's tendency to commit to Yes/No in the first token is a learned linguistic behavior, not an artifact of sharp decoding distributions. If the most probable first tokens are Yes or No (which they are, per the 95%+ commitment rate), spreading probability mass to lower-probability tokens at higher temperatures might occasionally produce a hedging first token, but this is rare and does not systematically change the model's behavior. The empirical results confirm this: temperature is not an effective mitigation strategy.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of removing components of a proposed method, since it proposes no new method. However, several analysis dimensions serve a similar diagnostic function:

Model scale comparison (ChatGPT vs. GPT-4): The comparison between ChatGPT and GPT-4 across all three tasks (Tables 6 and 7) serves as an implicit scale ablation. The finding that GPT-4, despite being the more capable model by standard benchmarks, has higher error rates on all three tasks (averaging 83.40% vs. 60.13%) is a non-obvious and important result. It suggests that stronger instruction-following and more elaborate justification behavior may actually exacerbate snowballing by increasing the model's tendency to commit to an answer and then construct detailed supporting fabrications. However, GPT-4 also shows higher snowballed hallucination recognition rates (87.03% vs. 67.37%), meaning that when GPT-4 does hallucinate, it is more likely to be a coherence-induced fabrication that it can recognize as false rather than a persistent knowledge gap. This is consistent with GPT-4 having stronger underlying knowledge but being equally or more susceptible to the commitment-then-justify dynamic.
Task type variation (computational vs. factual): The three datasets span different cognitive demands. Graph Connectivity and Primality Testing are provably outside $TC^0$ (per Merrill and Sabharwal, 2023), making them computationally impossible for transformers to solve in a single generation step. Senator Search is primarily a factual knowledge retrieval task. The fact that snowballing occurs across all three—including Senator Search, where both models achieve 0% error with chain-of-thought, demonstrating they possess the knowledge—indicates that the phenomenon is not limited to computationally hard problems but generalizes to any setting where premature commitment occurs.
Verification as robustness check on hallucination type classification: The two-stage verification procedure itself functions as a robustness mechanism. For each incorrect model response, the paper does not assume the false claim is snowballed; it empirically tests whether the model recognizes the claim as false. Cases where the model fails verification (e.g., Table 12, where GPT-4 incorrectly maintains that Willis Smith attended Dartmouth College) are explicitly not classified as snowballed hallucinations. This ensures that the snowballing rate is a lower bound—it only counts claims the model demonstrably recognizes as false, not claims that might be false and might be recognized.
Extraction reliability check: For the semi-automated extraction steps (primality factor extraction and senator name extraction using ChatGPT), the paper manually verified 30 examples each and reports "perfect extraction" in both cases (Section 3.3). This is a minimal but adequate validation that the automated extraction does not introduce systematic errors that would inflate or deflate the snowballing rate. The manual extraction for graph connectivity does not include a similar reliability check, which is a minor weakness—human annotators might miss some invalid flights or incorrectly classify valid-but-discontinuous flights.
Negative result: ReST-style iterative training is not tested. The paper mentions the possibility of fine-tuning on data with backtracking ("giving a question, followed by a wrong solution, and then issuing a phrase like 'Sorry, that was incorrect' before giving the correct solution"; Section 4.2) but does not test this. Similarly, the paper hypothesizes that beam search might reduce snowballing by maintaining sequences that do not commit to an answer, but cannot test this because the OpenAI API does not support beam search (Section 4.2, Limitations). These represent significant missing experiments—they would directly test whether the proposed mechanisms (backtracking training, maintaining multiple hypotheses) actually reduce snowballing.

Critical Assessment

Claim 1: LMs generate hallucinations that they can separately recognize as incorrect (the core phenomenon of hallucination snowballing).

What was tested: The two-stage verification pipeline directly measures this: the model generates a claim in its original justification, and that same claim is presented in an isolated new session. The finding that GPT-4 recognizes 87% of its own incorrect claims and ChatGPT recognizes 67% (Table 7) provides strong evidence that the phenomenon exists and is widespread on these three tasks.

What was not tested: The verification procedure assumes that a correct rejection in isolation implies the model "knows" the claim is false. However, there are alternative explanations that the experimental design cannot rule out. The most significant is that the verification question format is simply easier for the model than the original generation context, independent of any coherence-related mechanism. For example, the verification question "Is 9791 divisible by 13?" is a simpler, more focused query than "Is 9791 a prime number?"—the model might answer the simpler question correctly not because it was freed from coherence pressure, but because single-divisibility checks are inherently easier tasks. The paper does not include a control condition where the model is asked the verification-style question first (i.e., testing divisibility before the primality question) to establish that the model's accuracy on the simpler question is genuinely higher rather than just different. This is a significant methodological gap: the observed discrepancy between original-generation accuracy and verification accuracy could partially reflect task difficulty rather than context effects.

Additionally, the paper does not test whether the snowballing phenomenon is specific to the yes/no answer format or occurs in other generation structures. Would a model that first generates an explanation and then states its answer (the reverse order) show snowballing? The chain-of-thought results partially address this—the model generates reasoning before its final answer—but the reasoning chain still proceeds left-to-right, so errors early in the chain can still induce later errors. The paper does not test whether generating the conclusion after all reasoning steps (rather than interleaved) changes the error propagation pattern.

Claim 2: Snowballed hallucinations are caused by consistency pressure from the model's own prior output, not by knowledge gaps.

Evidence for: The verification stage demonstrates that when the same factual claim is extracted from the justification context, the model evaluates it differently—and correctly—in isolation. This establishes that the model possesses the knowledge to reject the claim. The cleanest evidence comes from the Senator Search task, where both models achieve 0% error with chain-of-thought prompting (Table 8), proving they have the factual knowledge to answer correctly, yet they still produce snowballed hallucinations under direct prompting.

Evidence against or complicating: The mechanism is inferred from the experimental design but is not directly tested. The paper hypothesizes that the model generates false claims "for consistency" with its earlier incorrect answer commitment, but this causal claim relies on the assumption that if the model had not committed to the wrong answer, it would not have generated those specific false claims. The paper does not test this counterfactual directly—for example, by editing the model's context to remove the Yes/No commitment and observing whether the subsequent justification changes.

Furthermore, the cases that are not snowballed (the model fails verification) pose an interpretive challenge. For GPT-4 on Primality Testing, 7.5% of incorrect responses are not recognized in isolation (28 out of 374; Table 7). Are these cases where knowledge truly is lacking, or cases where the verification prompt format failed for unrelated reasons? The paper provides one example of a failed verification (Table 12, Senator Search), where GPT-4 incorrectly maintains that Willis Smith attended Dartmouth College. This could be a genuine knowledge gap, but it could also reflect the model being misled by the senator's plausible-sounding biography or other factors. The binary classification (snowballed vs. non-snowballed) assumes a clean separation between knowledge-driven and coherence-driven hallucinations that may not exist in practice—there may be a continuum where partial knowledge interacts with context pressure in complex ways.

Claim 3: Chain-of-thought prompting dramatically improves accuracy but does not eliminate snowballed hallucinations on remaining errors.

What was tested: The chain-of-thought experiments (Tables 8 and 9) clearly demonstrate both the accuracy improvement and the persistence of snowballing on errors. The finding that GPT-4 snowballs on 94.90% of its remaining chain-of-thought errors is well-supported by the data.

Caveats: The number of remaining errors after chain-of-thought is small for GPT-4: only 21 errors on Graph Connectivity and 37 on Primality Testing. With such small denominators, the snowballing rate estimates (95.2% and 94.6% respectively) have wide confidence intervals—a single additional recognized or unrecognized claim would shift the percentage by roughly 2–5%. The paper does not report confidence intervals or statistical tests, making it difficult to determine whether the 94.90% snowballing rate with chain-of-thought is meaningfully different from the 87.03% rate under direct prompting, or whether both reflect a similarly high underlying snowballing propensity.

Additionally, the manual evaluation of chain-of-thought outputs for both correctness and snowballing introduces subjectivity that is not quantified. The chain-of-thought reasoning chains are less structured than the direct prompt outputs, making it harder to isolate specific false claims and verify them. The paper provides one example but does not describe the annotation protocol (e.g., what constitutes a "snowballed hallucination" in a multi-step reasoning chain where errors might be interdependent). This is a weakness in the experimental reporting.

Claim 4: Temperature and standard sampling methods do not mitigate snowballing.

What was tested: Three temperature settings ( $t=0.0, 0.6, 0.9$ ) across all three datasets for both models (Tables 10 and 11). The results show minimal variation in both error rates and snowballing rates.

What was not tested: The paper does not test top-k sampling or nucleus sampling empirically (only argues theoretically that they would not help). Beam search is discussed as a potential mitigation but is not tested because the OpenAI API does not support it. The paper also does not test more sophisticated inference-time interventions: no multi-turn interaction where the model is allowed to revise its answer, no prompting the model to "check your work" after generating, no ensemble methods, no self-consistency decoding (Wang et al., 2023). These are all plausible mitigation strategies that remain unexamined. The claim that "sampling methods would not help" is therefore supported only for temperature variation, not for the broader class of decoding strategies.

Claim 5: The phenomenon is general across different types of reasoning tasks.

What was tested: Three tasks spanning computational reasoning (primality testing, graph connectivity) and factual knowledge retrieval (senator search). Snowballing occurs on all three for both models, supporting the generality claim.

Caveats: All three tasks share the same structural property: yes/no questions where the model answers before providing justification. The paper does not test whether snowballing occurs in open-ended generation (e.g., summarization, story generation, long-form QA) where the model does not explicitly commit to a yes/no answer before elaborating. The authors hypothesize in Section 4.1 that "hallucination snowballing appears in open-ended text generation more broadly, where one mistake in the generation triggers more," citing Arora et al. (2022), but no experiments test this hypothesis. The generality claim is therefore limited to structured yes/no QA with predictable incorrect justification forms—the precise setting the paper engineered to measure the phenomenon.

Missing Experiments That Would Strengthen the Paper

A control condition comparing verification-question accuracy to original-question accuracy on matched-difficulty subproblems. For primality testing, this would mean comparing the model's accuracy on "Is 9791 divisible by 13?" when presented in isolation (the verification condition) versus its accuracy on the same divisibility question when it appears as a subsidiary step in a primality judgment. This would isolate the context effect from the task difficulty effect.
A reverse-order prompting experiment. What happens if the model is prompted to first explain its reasoning and then state the final answer, without using chain-of-thought's extended generation? Would this reduce first-token commitment and consequently reduce snowballing?
Inter-annotator agreement metrics for all manual evaluation steps. The paper relies on manual evaluation for answer determination (~2–4% of cases), graph connectivity flight extraction (all 500 incorrect cases), and verification output assessment (all incorrect cases across all datasets). No IRR scores are reported for any of these, which weakens confidence in the reliability of the measurements.
Experiments on open-weight models where beam search and probability access are available. The paper acknowledges this limitation in the Limitations section but it remains a significant gap—the beam search hypothesis is theoretically motivated but empirically untested.
Multi-turn correction experiments. If a snowballed hallucination is a claim the model can recognize as false, what happens if the model is prompted to "review your previous answer and correct any errors"? Does it spontaneously retract the snowballed claims? This would directly test the causal mechanism: if the model can identify and correct its own snowballed hallucinations when prompted to review, it strengthens the claim that these are coherence-induced rather than knowledge-gap errors.
Confidence calibration measurement. Does the model express high confidence in its snowballed hallucinations? If the model generates fabricated factorizations with high confidence but then correctly identifies them as false with high confidence in verification, this would be a striking demonstration of context-dependent knowledge access, but the paper does not measure confidence or log probabilities (partially because API access does not provide them).

Summary Assessment

The paper's central empirical finding—that a large fraction of model-generated false claims are recognized as false when presented in isolation—is well-supported by the two-stage experimental design and the consistency of results across three tasks and two models. The 67% (ChatGPT) and 87% (GPT-4) figures for snowballed hallucination recognition are the paper's strongest and most robust result.

However, the paper's stronger causal claim—that these hallucinations are specifically caused by coherence pressure from prior self-generated output—is less directly tested. The experimental design demonstrates a correlation between context (original justification vs. isolation) and the model's evaluation of claims, but it does not manipulate the context systematically to establish causation. Alternative explanations (task difficulty differences, prompt format effects, the model being better at verification than generation for unrelated reasons) are not ruled out.

The chain-of-thought results are informative but limited by small error counts in the chain-of-thought condition (especially for GPT-4) and the lack of quantitative reliability metrics for manual evaluation. The finding that snowballing persists at high rates on remaining errors is suggestive of a fundamental limitation, but the small sample sizes make the precise rates uncertain.

The generality of the phenomenon beyond structured yes/no QA remains an open question—the paper demonstrates snowballing in three carefully constructed tasks but does not show that it occurs in the open-ended generation scenarios where users most commonly encounter LM hallucinations in practice. This is not a weakness of the paper's design (which deliberately constrains the setting to enable controlled measurement) but it limits the scope of the conclusions that can be drawn.

6. Limitations and Trade-offs

The Experimental Design Only Tests Structured Yes/No QA with Predictable Incorrect Justifications

The assumption or constraint. The paper deliberately constructs three datasets where questions have a binary yes/no answer, the correct answer is fixed to a single label for all 500 examples (always Yes for primality, always No for the other two), and incorrect answers take a predictable justificatory form (a false factorization, a named senator, a flight sequence). The authors explicitly acknowledge this scope limitation in the Limitations section:

"We focus on hallucination snowballing in the context of question answering in English, and we do not explore it on other tasks, such as summarization or code generation."

The consequence. The paper demonstrates that snowballing can occur and measures its prevalence under a carefully controlled setting, but the experimental design provides no evidence about whether the phenomenon generalizes to the open-ended generation scenarios where users most commonly encounter hallucinations. In free-form dialogue, summarization, or creative writing, the model does not first output a yes/no answer and then justify it — the generation structure is less constrained, and the "commitment point" is less clearly defined. The mechanism proposed in Section 2 — initial committal followed by coherence-driven justification — depends on the prompt format leading the model to state an answer before reasoning. It is entirely unclear whether analogous pressure arises when the model generates a long passage where early factual errors could propagate into later claims that the model would separately reject. The authors themselves hypothesize that "hallucination snowballing appears in open-ended text generation more broadly" (Section 4.1) but provide no experiments testing this, making it an open question whether the phenomenon is a general property of autoregressive generation or a specific artifact of yes/no question formats.

What evidence exists in the paper. None. This is a scope limitation that the paper does not address experimentally. The three datasets all share the same structural property (yes/no question → answer-first response format), and there is no experiment varying the response structure to test whether snowballing occurs when the model is not forced to commit to an answer before explaining. The Senator Search task, which is primarily a knowledge retrieval task rather than a sequential reasoning task, provides some evidence that the phenomenon is not limited to provably hard computational problems, but it still shares the same answer-first format. No task in the paper tests open-ended generation, long-form reasoning, or any format where the model's first tokens are not constrained to be a yes/no answer.

Mitigation status. The paper does not attempt to address this limitation. The scope is acknowledged in the Limitations section and is implicitly part of the experimental design choice to "systematically examine model-written justifications for incorrect answers" (Section 3.1). The fix would be to construct open-ended tasks where the model makes an early factual claim (not necessarily yes/no) and later generates supporting details that could be extracted and verified in isolation, but the paper does not attempt this.

Difficulty Estimation Cost Is Entirely Unaccounted For — And Infeasible in Real Deployment

The assumption or constraint. The verification pipeline that identifies snowballed hallucinations requires: (1) knowing the ground-truth correct answer to determine whether the model's response was incorrect; (2) human (or semi-automated) extraction of the specific false claim from the model's justification; and (3) a separate API call to a new session to present the extracted claim in isolation. All three steps require access to information or infrastructure that a real user interacting with the model does not have. The paper acknowledges this implicitly in the Limitations section:

"This restricts our ability to explore potential mitigation strategies."

but the point is broader than missing API features — the entire diagnostic framework depends on oracle access to ground-truth answers and a carefully constructed extraction pipeline that cannot exist in deployment.

The consequence. The headline finding — that 67% of ChatGPT's and 87% of GPT-4's incorrect claims are snowballed hallucinations — is a measurement of a phenomenon, not a system that can detect or prevent snowballing. A real user interacting with ChatGPT has no way to know whether a given false claim in the model's response is a snowballed hallucination (which the model could potentially correct if prompted differently) or a persistent hallucination (which reflects a genuine knowledge gap). The paper provides no deployment-ready mitigation — it diagnoses the disease but offers no treatment that a user or system builder can apply at inference time beyond "use chain-of-thought, but note that errors still snowball." The chain-of-thought results (Section 4.1) show that this mitigation reduces the overall error rate but does not eliminate snowballing on remaining errors (94.90% of GPT-4's chain-of-thought errors still contain snowballed hallucinations; Table 9), and chain-of-thought requires the user to know to append "Let's think step by step" and to accept the increased latency and token cost of the longer response.

What evidence exists in the paper. The entire experimental design relies on oracle access: the ground-truth labels (primality, senator facts, graph structure) are known to the researchers but not provided to the model. The verification stage requires human extraction of specific claims (manual for graph connectivity; semi-automated for the other tasks) and separate API calls. The paper does not report the cost or latency overhead of this verification pipeline, but for graph connectivity — the only task where extraction is fully manual — the cost in human annotator time is substantial and completely unaccounted for in any efficiency metric.

Mitigation status. Not addressed. The paper does not propose a method for detecting snowballed hallucinations at inference time without oracle access. The Limitations section briefly mentions that "having access to the output distributions would allow us to investigate mitigating the snowballing hallucination issue using alternative sampling methods such as beam search" and that fine-tuning access would enable exploring instruction tuning with backtracking data, but these are suggestions for future work on model development, not techniques a user can apply today. The gap between diagnosing the problem and solving it is not resolved.

The Causal Mechanism Is Inferred from Correlation, Not Directly Tested

The assumption or constraint. The paper's central causal claim is that the model generates false claims because of coherence pressure from its own prior output — specifically, that the early commitment to a yes/no answer forces subsequent generation to be consistent with that commitment, overriding knowledge that would otherwise produce correct claims. This mechanism is hypothesized in Section 2:

"once the LM generates Yes or No, that token remains in the context, and coherence would require commitment to that choice through the subsequent justification."

The consequence. The experimental design demonstrates a correlation between context (original justification vs. isolated verification) and the model's evaluation of claims, but it does not establish the causal direction. There are alternative explanations that the paper does not rule out:

Task difficulty confound: The verification questions are structurally simpler than the original questions. "Is 9791 divisible by 13?" requires a single divisibility check, while "Is 9791 a prime number?" requires checking all potential divisors (or employing a primality testing algorithm). The model might answer the simpler question correctly not because it was freed from coherence pressure, but because divisibility-by-a-specific-number is an inherently easier task than primality testing. The paper provides no experiment that controls for task difficulty — for example, asking the model the divisibility question first (before the primality question) and comparing accuracy to show that the model genuinely performs better on it, regardless of context.
Prompt format effects: The verification prompts have a different structure than the original prompts. For primality testing, the verification prompt is "Is N divisible by K? Answer with either Yes or No" while the original prompt is "Is N a prime number?" with no answer format constraint. For graph connectivity, the verification prompt re-presents all flight information and asks "Is City X to City Y a valid flight?" — this is a direct-match question (is this specific edge in the provided list?) while the original question requires multi-hop path reasoning. The model might be better at the verification questions for reasons unrelated to the absence of prior self-generated context — the questions are simply different in ways that favor correct answers.
The model might recognize the claim as false in the original context too, but not state it: The paper's verification methodology shows that the model can correctly identify the false claim when asked directly about it. But this does not prove that the model failed to identify it as false during the original generation. It is possible that the model recognized the factorization as false even while generating it, but the training-induced pressure to produce coherent text overrode that recognition. This is still a form of coherence-induced error, but the mechanism is different from the "knowledge gap" framing — it's not that the model lacks knowledge, but that the generation process cannot effectively veto tokens that conflict with stored knowledge when those tokens are the most probable continuation of the current context. The paper's binary classification of "snowballed = model recognizes in isolation" vs. "persistent = model does not recognize in isolation" may oversimplify a more complex interaction between knowledge access and generation dynamics.

What evidence exists in the paper. The paper provides only the correlation evidence: claims generated in one context are rejected in another. This is a necessary condition for the coherence-pressure hypothesis but not sufficient to establish it. The one piece of evidence that edges closer to causal demonstration is the Senator Search result: both models achieve 0% error with chain-of-thought (Table 8), proving they have the knowledge, yet they produce snowballed hallucinations under direct prompting. This eliminates the "knowledge gap" alternative for this specific task and shows that the generation order (answer-first vs. reasoning-first) determines whether the knowledge is accessed correctly. However, this still does not directly test whether the false claims in the direct-prompt condition are caused specifically by the yes/no commitment token rather than by some other aspect of the direct prompt format.

Mitigation status. Not addressed. The paper does not include a control experiment that isolates the effect of the yes/no commitment token from other aspects of the prompt format. For example, it does not test prompting the model to "first explain your reasoning, then state your answer" (without the chain-of-thought's extended generation) to see whether removing the early commitment eliminates snowballing independently of increasing reasoning depth. The beam search hypothesis (Section 4.2) — that maintaining multiple sequences without committing to an answer could help — directly targets the causal mechanism but is untestable under the current experimental constraints.

The Beam Search Hypothesis and All Non-Temperature Decoding Strategies Are Untested

The assumption or constraint. The paper argues theoretically that beam search could alleviate snowballing by maintaining candidate sequences that do not commit to an answer (or commit to the correct answer), whose continuations might eventually have higher probability than incorrectly-committed sequences:

"if some sequences in the beam after the initial token do not commit to an answer (or commit to the right answer), their continuations may eventually have higher probability than those that initially commit incorrectly and later produce incorrect reasoning as a result" (Section 4.2).

However, this hypothesis cannot be tested empirically because:

"we cannot test the effect of beam search on hallucination snowballs because the OpenAI API does not support beam search." (Section 4.2)

The same access limitation applies to top-k and nucleus sampling, which are discussed only theoretically (Section 4.2). The paper also does not have access to output token probability distributions, which prevents any analysis of the model's confidence or uncertainty during generation — a critical missing piece for understanding whether the model is generating snowballed hallucinations despite assigning low probability to the false claims, or whether the model is genuinely "confident" in its fabrications at generation time.

The consequence. The paper's practical contribution is limited to diagnosis — it identifies a problem but cannot test the most obvious algorithmic fix that follows from the proposed mechanism. If beam search successfully reduces snowballing (as the theoretical argument suggests it might), the paper's contribution would shift from "here is a pervasive failure mode with no easy fix" to "here is a pervasive failure mode that can be addressed with beam search." Conversely, if beam search does not help, the paper's proposed mechanism might need revision — snowballing would not be caused simply by committing too early to a single sequence, but by something more fundamental about how the model's training interacts with autoregressive generation. Without this experiment, both the practical utility of the finding and the theoretical mechanism remain uncertain.

The inability to access probability distributions is equally consequential. If the model assigns high probability to its snowballed hallucinations (treating them as confident predictions), this suggests the problem is encoded in the model's parameters rather than induced by decoding dynamics. If the model assigns low probability but still generates them (because the probability of alternative tokens is even lower under the coherence constraint), this suggests the problem is indeed a dynamic effect of autoregressive generation. The paper cannot distinguish between these two cases, which represent fundamentally different underlying mechanisms.

What evidence exists in the paper. The only decoding strategy empirically tested is temperature variation (Section 4.2, Tables 10 and 11), which shows minimal effect on either error rates or snowballing rates. The paper correctly argues that temperature is unlikely to help because the first-token commitment to Yes/No is a learned linguistic behavior, not an artifact of sharp distributions — spreading probability mass to lower-probability tokens at higher temperature might occasionally produce a hedging first token, but this is rare and does not systematically change the commitment-then-justify dynamic. However, temperature is a weak test of decoding-based interventions because it does not maintain multiple hypotheses or allow the model to "change its mind" after the first token. Beam search, which explicitly maintains alternative sequences, is the direct test of the paper's proposed mechanism, and it is absent.

Mitigation status. The paper acknowledges this as a limitation of the API:

"Having access to the output distributions would allow us to investigate mitigating the snowballing hallucination issue using alternative sampling methods such as beam search." (Limitations)

The obvious mitigation — running experiments on open-weight models where beam search, probability distributions, and fine-tuning are all available — is not pursued. The paper's exclusive focus on proprietary API-accessed models (ChatGPT and GPT-4) is understandable given their real-world importance and state-of-the-art status, but it leaves a significant gap. The paper suggests future work on fine-tuning with backtracking data and changing pretraining/instruction tuning to emphasize reasoning-before-answering (Section 4.2, Limitations), but does not test any of these.

The Graph Connectivity Manual Extraction and All Manual Evaluations Are Not Quantified for Reliability

The assumption or constraint. The verification pipeline — the core empirical procedure that produces the paper's headline results — relies on two manual evaluation steps for which no inter-annotator agreement (IAA) metrics are reported:

Graph connectivity flight extraction (Section 3.3): For each incorrect model response where the model lists a flight sequence, human annotators "manually extract the list of flights from the model's output and identify the invalid or discontinuous flights." This is the only dataset where extraction is fully manual (primality factors and senator names are extracted semi-automatically using ChatGPT with one-shot prompts, with 30-example spot-checks confirming 100% accuracy). The paper does not report how many annotators performed this extraction, whether multiple annotators independently extracted flights from the same responses, or whether their extractions agreed.
Verification output assessment (Section 3.3): Across all three datasets, human annotators "manually assess the verification output to check if the model correctly detects the error." This assessment determines whether a given incorrect claim is classified as a snowballed hallucination (model recognized the error) or a persistent hallucination (model failed to recognize the error). The paper does not report IAA for this judgment, does not specify how many annotators were involved, and does not provide the annotation guidelines.

The consequence. The paper's headline finding — GPT-4 recognizes 87% of its own incorrect claims — depends on the reliability of the manual classification of verification outputs. If annotators disagree on whether a model response constitutes "correctly detecting the error," the snowballed hallucination rate could shift substantially. This is not a hypothetical concern: while many verification answers are clear yes/no responses (e.g., GPT-4's verification answer in Table 3: "No, based on the above flight information, there is no direct flight from City K to City G"), others may be more ambiguous. For example, if the model says "No, the flight is not listed in the provided information" — does this count as detecting the error? What if the model hedges ("I don't believe that flight exists based on the given flights")? The paper provides no annotation protocol to answer these questions.

For graph connectivity specifically, the reliability problem compounds across two manual steps: first, the human-extracted flight list must correctly identify which flight(s) in the model's claimed route are invalid; second, the manual assessment of the verification output must correctly classify the model's response. Any error in the extraction stage (e.g., annotator misses an invalid flight, or incorrectly identifies a valid-but-discontinuous flight as invalid) propagates to the verification stage and could misclassify the case.

What evidence exists in the paper. For the semi-automated extraction steps (primality factors and senator names), the paper provides reliability evidence: "we manually checked 30 examples and found that it can always extract the correct factors" (Section 3.3). For the fully manual graph connectivity extraction and for the verification output assessment, no reliability metrics are reported. The paper does not state the number of annotators, the annotation protocol, or whether any independent double-coding was performed. Table 7 reports snowballed hallucination counts at the precision of individual examples (e.g., 396/410 for ChatGPT on Graph Connectivity), implying that the classification is treated as deterministic, but no evidence is provided that different annotators would produce the same classification.

Mitigation status. Not addressed. The paper does not acknowledge the absence of IAA metrics as a limitation, does not describe the annotation procedure in sufficient detail for replication, and does not report any reliability statistics for the manual evaluation steps. Given that these manual judgments directly determine the paper's headline quantitative results, this is a significant methodological gap. The fix would be straightforward — have at least two annotators independently extract flights and assess verification outputs on a subset of examples, report Cohen's kappa or percentage agreement, and either use adjudication for disagreements or average across annotators — but the paper does not report having done this. The relatively objective nature of the verification task (checking whether a model said "no" to a specific factual claim) may mean that agreement is high in practice, but without reporting it, this remains an assumption rather than a demonstrated fact.

The Investigation Is Limited to Two Proprietary Models from a Single Model Family

The assumption or constraint. The paper runs all experiments on exactly two models: ChatGPT (gpt-3.5-turbo) and GPT-4, both accessed through the OpenAI API. The paper states:

"We only conduct experiments on two proprietary models, namely ChatGPT and GPT-4, due to their state-of-the-art performance on many benchmarks" (Limitations)

and further notes:

"Due to the limitations of the APIs for these models, we do not have access to the probability distributions they output and do not have the ability to finetune them."

The consequence. The paper cannot determine whether hallucination snowballing is a general property of the autoregressive language modeling paradigm (affecting all transformer LMs regardless of architecture, scale, or training procedure) or a specific artifact of the RLHF-based instruction tuning used in ChatGPT and GPT-4. Instruction tuning explicitly trains models to be helpful, coherent, and to provide explanations for their answers — precisely the behaviors that could exacerbate snowballing by making the model more likely to generate detailed justifications for incorrect answers. The paper's own speculation that "instruction-tuned LMs will reflect this answer format where the answer comes before the explanation" (Section 2) suggests that instruction tuning might amplify the initial committal behavior that triggers snowballing. If so, base (pre-instruction-tuning) models might show lower rates of snowballing, or might be less prone to fabricating specific false justifications.

Several observations in the paper hint that model-specific factors matter. GPT-4 has substantially higher error rates than ChatGPT across all three tasks (83.40% vs. 60.13% average; Table 6) despite being the more capable model by standard benchmarks. The paper notes this is paradoxical: "despite popularly being considered superior to ChatGPT" (Section 3.4). One possible explanation is that GPT-4's stronger instruction-following and more elaborate justification behavior increases its tendency to commit to an answer and then construct detailed supporting fabrications. If this explanation is correct, then snowballing rates and error rates might vary substantially across model families depending on their training objectives and instruction-tuning details. Without experiments on other model families, the generality of the phenomenon remains unknown.

Additionally, the inability to fine-tune or access probability distributions prevents the paper from testing the most direct interventions. If snowballing is caused by RLHF training that over-prioritizes coherence, then fine-tuning on data where the model learns to backtrack (as suggested in Section 4.2) could reduce snowballing — but this hypothesis is untestable with API-only access. The paper acknowledges this:

"Having the ability to finetune the model would allow us to explore whether instruction tuning with different annotations could lead to better handling of the questions we use to instigate hallucination snowballing." (Limitations)

What evidence exists in the paper. The comparison between ChatGPT and GPT-4 (Tables 6 and 7) provides within-model-family evidence that scale and capability interact with snowballing in non-obvious ways: the larger model is more prone to the initial errors that trigger snowballing but also more likely to recognize its snowballed claims in isolation. This is interesting but limited to two checkpoints from the same developer using related training procedures. The paper provides no evidence about whether models from other developers (Claude, Gemini, Llama, Mistral), models with different architectures (non-autoregressive, retrieval-augmented), or base models without instruction tuning exhibit the same behavior. The inclusion of only API-accessed models also means that the paper cannot test models at different scales within the same family to examine whether snowballing systematically varies with model size.

Mitigation status. The paper acknowledges the model limitation in the Limitations section and frames it as a constraint of API access rather than a choice. The suggested mitigation — experiments with open-weight models that allow beam search, probability access, and fine-tuning — is implicitly deferred to future work. The paper does not argue that the choice of models is representative of all LLMs; it simply notes that these are the most consequential models to study given their deployment scale. This is a defensible prioritization, but it means the reader should treat the findings as demonstrated for ChatGPT and GPT-4 specifically, with generality to other model families remaining an open empirical question.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new diagnostic category for understanding LM hallucinations — the distinction between knowledge-gap hallucinations and coherence-induced (snowballed) hallucinations — and provides a concrete, replicable experimental methodology for measuring which type dominates in a given setting. This is not a paradigm shift in the sense of overturning prior findings; rather, it is a conceptual reframing with sharp practical implications that resolves puzzling contradictions in the hallucination literature.

The key landscape change is that the paper demonstrates that the standard framing — "LMs hallucinate because they lack knowledge" (Zheng et al., 2023) — is incomplete in a way that matters for designing interventions. The finding that GPT-4 can recognize 87% of its own incorrect claims when those claims are presented in isolation (Table 7) establishes that a substantial fraction of hallucinations are not caused by missing knowledge but by the autoregressive generation process overriding accessible knowledge. This means that the dominant mitigation strategy — retrieval-augmented generation — addresses only one class of hallucinations. For snowballed hallucinations, retrieving external facts would be redundant because the model already "knows" the correct answer; the bottleneck is not knowledge access but the model's inability to prevent its prior self-generated context from coercing subsequent tokens into falsehoods that contradict stored knowledge.

This reframing resolves a previously puzzling contradiction in the literature. Prior work reached opposite conclusions about whether LMs can self-correct: Huang et al. (2023) found that "large language models cannot self-correct reasoning yet," while Madaan et al. (2023) and others found benefits from self-refinement. The paper's framework provides a reconciliation: the effectiveness of self-correction depends on whether the target hallucination is snowballed or knowledge-gap-driven. A snowballed hallucination — a claim the model can recognize as false in isolation — is exactly the kind of error that self-correction could fix if the model were prompted to review its output. A knowledge-gap hallucination — a claim the model consistently endorses regardless of context — cannot be fixed by self-review because the model has no internal signal that the claim is wrong. Prior studies that tested self-correction on different task distributions would naturally reach different conclusions depending on which type of hallucination dominated their experimental setup.

The paper also changes the conversation around chain-of-thought prompting. The dominant narrative — that chain-of-thought improves reasoning by enabling the model to break complex problems into simpler steps — remains true, and the paper's own results confirm dramatic accuracy gains (from 83% to 4% error rate for GPT-4; Tables 6 and 8). However, the paper demonstrates that chain-of-thought has a residual failure mode that prior work overlooked: when an intermediate reasoning step is wrong, that error becomes corrupting context for subsequent steps, inducing snowballed hallucinations that the model would not generate independently. GPT-4 snowballs on 94.90% of its remaining chain-of-thought errors (Table 9). This shifts the research agenda for reasoning methods from simply maximizing the probability that each step is correct to developing mechanisms for error detection and recovery within the reasoning chain — the ability to recognize when an intermediate step was wrong and backtrack before proceeding.

The methodological contribution — the isolated-claim recognition diagnostic — may prove as influential as the empirical findings. The paper operationalizes a previously fuzzy conceptual distinction (does the model "know" a fact or not?) into a concrete experimental procedure: extract the claim from the model's output and present it to the same model in a clean context to test recognition. This diagnostic can be applied to any black-box model without access to internals, making it immediately deployable by researchers evaluating any API-accessible LM. Its adoption into standard evaluation suites would enable the field to move beyond simple accuracy metrics toward a more nuanced understanding of why models make errors — whether errors reflect stable knowledge gaps or context-dependent generation artifacts. The paper's demonstration that this diagnostic yields different results across tasks (ChatGPT snowballs on only 37% of primality errors but 97% of graph connectivity errors; Table 7) and across models (GPT-4 recognizes 87% of its errors vs. ChatGPT's 67%) shows that it captures meaningful variation that accuracy alone obscures.

Research directions that become more attractive:

Error recovery and backtracking mechanisms for chain-of-thought reasoning. The finding that intermediate errors propagate into downstream snowballed hallucinations (Section 4.1) makes error recovery a first-order problem for reasoning systems, not a secondary consideration.
Training interventions that teach models to override coherence pressure when it conflicts with factual knowledge. The paper's suggestion of fine-tuning on backtracking data (Section 4.2) targets the specific mechanism identified — training the model to say "wait, that's wrong" when it detects an inconsistency — rather than the generic goal of improving accuracy.
Context-editing interventions at inference time. If snowballing is caused by faulty self-generated context, then detecting and editing that context before it corrupts subsequent generation becomes a natural mitigation approach.

Research directions that become less attractive:

Pure knowledge-supplementation approaches as complete solutions to hallucination. The paper demonstrates that many hallucinations persist even when the model possesses the relevant knowledge, so retrieval and fact-checking alone cannot solve the problem. Knowledge supplementation remains valuable for addressing knowledge-gap hallucinations specifically, but the paper shows it would be insufficient for the 67–87% of errors that are snowballed.
Temperature tuning and simple decoding interventions as mitigations. The paper empirically demonstrates that varying temperature from 0.0 to 0.9 has negligible effect on either error rates or snowballing rates (Tables 10 and 11), and the theoretical argument against top-k and nucleus sampling (Section 4.2) suggests that standard decoding hyperparameter sweeps are unlikely to address the underlying mechanism.

Follow-Up Research This Work Enables

Direct causal test of the coherence-pressure mechanism through context ablation. The paper demonstrates a correlation between context (original justification vs. isolated verification) and the model's evaluation of claims, but does not establish causation. A direct test would manipulate the model's context mid-generation: after the model generates an incorrect Yes/No answer (e.g., for primality testing), edit the conversation to remove that commitment token from the context window and prompt the model to continue generating. Does the model still produce a false factorization, or does it now correctly identify the number as prime? This experiment would isolate the causal role of the commitment token in triggering the snowballed justification. A strong follow-up would run this on an open-weight model (Llama, Mistral) where context editing is possible, comparing the model's continuation with the commitment token present vs. absent vs. replaced with a hedging phrase ("I'm not sure, let me think...").

Beam search experiments on open-weight models to test the delayed-commitment hypothesis. The paper's central theoretical argument (Section 4.2) — that beam search could reduce snowballing by maintaining candidate sequences that do not commit to an answer prematurely — is untested because the OpenAI API does not support beam search. The most immediate follow-up experiment is to replicate the three-dataset evaluation on an open-weight model (Llama-3, Mistral, or Qwen) with beam search enabled, measuring both error rates and snowballed hallucination rates (using the same two-stage verification pipeline) across beam widths. The hypothesis predicts that wider beams should reduce snowballing if maintaining alternative hypotheses allows correct reasoning chains to eventually outscore initially-incorrect-but-confident sequences. A null result — beam search showing no reduction in snowballing despite wider beams — would challenge the proposed mechanism and suggest that the problem is more fundamental than first-token commitment alone (e.g., the model might assign higher probability to incorrect justifications regardless of beam width because those justifications are coherent continuations of the prompt format).

Cross-model-family replication to determine whether snowballing is a universal autoregressive phenomenon or specific to RLHF instruction tuning. The paper studies only ChatGPT and GPT-4, both from OpenAI using RLHF-based instruction tuning. The finding that GPT-4 has higher error rates than ChatGPT despite being more capable (Tables 6 and 7) hints that instruction tuning might amplify the commitment-then-justify behavior. A systematic replication across model families would test: (a) base (pre-instruction-tuning) models from the same family (e.g., Llama-3-base vs. Llama-3-instruct) to measure whether instruction tuning increases snowballing propensity; (b) models from different developers with different training procedures (Claude, Gemini, Qwen) to test whether snowballing rates correlate with instruction-following strength; (c) models at different scales within a single family to examine whether snowballing systematically varies with model size. The hypothesis is that stronger instruction-tuned helpfulness and explanation-providing behaviors would correlate with higher snowballing rates, while base models might refuse to provide justifications or hedge more often.

Generalization to open-ended generation beyond structured yes/no QA. The paper's three datasets all share the same structure: yes/no question → answer-first response → predictable incorrect justification form. The authors hypothesize that snowballing "appears in open-ended text generation more broadly, where one mistake in the generation triggers more" (Section 4.1, citing Arora et al., 2022), but provide no experiments. A generalization study would design open-ended tasks where an early factual claim can be identified and extracted for isolated verification. For example: prompt the model to write a biography of a historical figure; identify any specific factual claims in the first paragraph (birth year, birthplace, major event); then present those claims in isolation to test recognition. This would determine whether snowballing is an artifact of the answer-first format or a general property of autoregressive generation where any early error corrupts subsequent context. A negative result — snowballing does not occur in open-ended generation — would bound the phenomenon's scope to structured QA and suggest that different interaction formats naturally avoid the problem.

Training with backtracking demonstrations to teach error recovery. The paper's brief suggestion (Section 4.2) — "finetuning on data with backtracking" by showing the model a wrong solution followed by "Sorry, that was incorrect" and then the correct solution — is a concrete training intervention that follows directly from the proposed mechanism. A strong follow-up would construct a training dataset of backtracking trajectories on these three tasks: generate incorrect model responses, pair them with the correct answer preceded by a backtracking phrase ("Actually, I need to reconsider — [original answer] is wrong because [specific error]. The correct answer is [correct answer] because [correct justification]"), and fine-tune an open-weight model on these trajectories. The evaluation would measure not just overall accuracy improvement but specifically whether errors that remain are less likely to be snowballed (i.e., the model either gets the right answer or produces errors it cannot recognize, rather than producing errors it recognizes as wrong). This would test whether backtracking training teaches the model a general error-recovery skill or simply adds domain-specific correction patterns.

Confidence-calibration analysis of snowballed vs. persistent hallucinations using open-weight models with log-probability access. The paper cannot access output token probabilities due to API limitations, preventing any analysis of whether the model is "confident" in its snowballed hallucinations at generation time. An open-weight replication measuring token-level probabilities would address: does the model assign high probability to snowballed claims during generation (suggesting the model genuinely endorses them in that context) or low probability (suggesting the model is generating them as coherent continuations despite uncertainty)? The distinction matters for mechanism: high-probability snowballed claims suggest that context overrides the model's knowledge at the parameter level, while low-probability claims suggest that the model can distinguish correct from incorrect even during generation but the decoding process selects the coherent continuation over the factual one. A related analysis would measure whether the probability assigned to a snowballed claim in the original generation context correlates with the probability of correctly rejecting that claim in the isolated verification context — this would test whether the model's internal uncertainty signal is preserved across contexts even when its output behavior flips.

Practical Applications and Downstream Use Cases

Detection of unreliable model responses in deployed QA systems using self-consistency checks. The paper's core finding — that a large fraction of model-generated false claims are recognized as false when presented in isolation — suggests a practical detection strategy that could be implemented today with API-accessible models. When a model produces an answer with a detailed justification, a downstream system could automatically extract specific factual claims from that justification (using a lightweight claim extraction model or even a prompted LLM call), re-query the same model with each claim as an isolated verification question in a new session, and flag responses where the model contradicts itself across sessions. The paper shows that GPT-4 detects 87% of its own incorrect claims this way (Table 7), meaning that a significant fraction of hallucinations could be identified without external knowledge bases or human verification. The cost is one additional API call per extracted claim, which for the typical justification containing 1–3 verifiable claims (one factorization, one senator name, one invalid flight) represents a 2–4× increase in inference cost — a reasonable overhead for high-stakes applications where undetected hallucinations carry significant risk (medical QA, legal research, financial analysis).

Prompt design guidelines that avoid triggering the commitment-then-justify dynamic. The paper provides concrete evidence that the answer-first format is the trigger for snowballing. When the model is prompted to reason first and answer later (via "Let's think step by step"), error rates drop from 60–83% to 4–9% (Tables 6 and 8). This is an immediately actionable finding for prompt engineering: for any yes/no or binary classification question where the model might be tempted to state the answer before verifying, append a reasoning-first instruction. More broadly, the paper suggests that prompt formats which elicit the model's answer in the first token should be avoided for tasks where the model cannot reliably compute the answer in a single forward pass — which, per Merrill and Sabharwal (2023), includes all tasks outside $TC^0$ , a class encompassing many multi-step reasoning problems. For system builders designing LM-powered applications, this implies that the user-facing prompt should be restructured to separate the reasoning phase from the answer phase, even at the cost of higher latency and token consumption. The Senator Search result — 0% error with chain-of-thought on a task where direct prompting produced 30–87% error rates — demonstrates that the accuracy gain can be categorical, not just incremental, for tasks where the model possesses the knowledge but fails to access it under answer-first pressure.

Prioritizing verifier or self-consistency development over pure knowledge supplementation for certain hallucination types. The paper's distinction between snowballed and persistent hallucinations has resource-allocation implications for teams building hallucination mitigation systems. If a deployment's error analysis using the paper's isolated-claim diagnostic reveals that most hallucinations are snowballed (i.e., the model can recognize them in isolation), then investment should flow toward self-consistency mechanisms, context editing, and backtracking training rather than toward expanding retrieval corpora or improving knowledge base coverage, which would address only the minority of persistent hallucinations. Conversely, if errors are predominantly persistent (the model cannot recognize them even in isolation), retrieval augmentation is the appropriate strategy. The paper provides a concrete measurement tool — the two-stage verification pipeline — that practitioners can apply to their own task distributions to guide this investment decision, rather than relying on assumptions about which type of hallucination dominates. The fact that the ratio of snowballed to persistent hallucinations varies dramatically by task (e.g., 97% snowballed vs. 3% persistent for ChatGPT on Graph Connectivity; 37% vs. 63% for ChatGPT on Primality Testing; Table 7) demonstrates that this ratio is not a fixed property of the model but depends on the task, making per-deployment measurement essential.

Self-improvement data generation that avoids contaminating training data with snowballed hallucinations. In automated self-improvement pipelines (where model outputs are used as training data for subsequent fine-tuning rounds), the paper's findings imply that some fraction of generated "reasoning traces" will contain snowballed hallucinations — false intermediate steps that the model generated for coherence but can recognize as wrong. Including these in training data could teach the model to produce internally contradictory reasoning chains, potentially worsening the snowballing behavior in future iterations. A practical mitigation: before adding generated reasoning traces to the training set, run the isolated-claim verification pipeline on a sample to estimate the snowballing rate and filter out traces where the model contradicts itself. The paper's 94.90% snowballing rate on GPT-4's chain-of-thought errors (Table 9) suggests that for incorrect reasoning chains, near-total contamination is likely — practically all wrong chain-of-thought outputs contain claims the model would reject — so filtering based on correctness alone (only keeping traces with correct final answers) may be insufficient if those traces contain valid reasoning but the model also generated some correct answers through invalid reasoning chains, which the paper notes occurs ("sometimes even when 'Let's think step by step' does lead to the right answer, it uses invalid reasoning chains"; Section 1).