Why Language Models Hallucinate

🎯 Pitch

Language models are forced to guess because training and evaluation both penalize admitting uncertainty—a model that says 'I don't know' loses on the leaderboard. Lower bounds show that even on perfect data, cross-entropy pretraining compels hallucination on rare facts, and post-training with binary grading creates a compute-optimal bluffing strategy.

1. Executive Summary

This paper analyzes why language models hallucinate—producing plausible falsehoods instead of admitting uncertainty—by tracing the statistical origins of such errors through the modern training pipeline and the incentive structures of evaluation benchmarks. The work establishes a formal reduction from generative errors to binary classification mistakes via the Is-It-Valid (IIV) problem (distinguishing valid responses from uniformly random errors), showing that calibrated base models trained with cross-entropy on even error-free data must hallucinate on facts that appear only once in training, with a lower bound of roughly the singleton rate. For post-training, the paper argues that hallucinations persist because most mainstream evaluations use binary grading that penalizes uncertainty expressions like "I don't know," creating an epidemic where compute-optimal guessing (bluffing when unsure) improves leaderboard scores while honest abstention is strictly suboptimal. A proposed mitigation—embedding explicit confidence thresholds into existing benchmark prompts—is shown to realign incentives, establishing that hallucination reduction requires modifying the scoring of primary evaluations rather than merely adding bespoke hallucination benchmarks, since the latter are drowned out by dominant binary-graded leaderboards.

2. Context and Motivation

The Core Problem: Hallucinations Persist Despite Massive Investment in Mitigation

The paper addresses a specific, stubborn gap: state-of-the-art language models continue to hallucinate—producing overconfident, plausible falsehoods—despite years of research explicitly targeting this failure mode. This is not a problem of insufficient effort. The field has developed an extensive toolkit of mitigation strategies: reinforcement learning from human feedback (RLHF, Ouyang et al., 2022), reinforcement learning from AI feedback (RLAIF, Bai et al., 2022), direct preference optimization (DPO, Rafailov et al., 2023), retrieval-augmented generation (RAG, Lewis et al., 2020; Shuster et al., 2021), repeated self-consistency checks (Manakul et al., 2023; Xue et al., 2025), and search-augmented reasoning (Nakano et al., 2021). Yet, as the paper demonstrates with concrete examples from contemporary deployed models (GPT-4o, DeepSeek-V3, Llama-4, Claude 3.7, and Meta AI), hallucinations remain pervasive even in the latest systems (Table 1, Section 1). A model like DeepSeek-V3—released in 2025 with 600B parameters and trained with reasoning-oriented RL—still cannot reliably count the number of Ds in "DEEPSEEK" when asked, returning answers ranging from 2 to 7 across independent trials.

This persistence is puzzling and practically damaging. The paper cites the 2025 AI Index Report (Maslej et al., 2025) which notes that hallucination benchmarks "have struggled to gain traction within the AI community," suggesting that the field's understanding of why hallucination persists is incomplete. If the problem were purely a matter of better data or better training techniques, the rapid pace of model improvement over 2022–2025 should have produced more dramatic reductions. The fact that it hasn't implies that hallucinations arise from structural features of the training-and-evaluation pipeline that are not addressed by the prevailing mitigation approaches.

Why the Problem Matters: Trust, Deployment, and the Limits of the Current Paradigm

The practical stakes are high, and the paper identifies several dimensions:

Trust and utility. Hallucinations "diminish their utility and trustworthiness" (Section 1, first sentence). In high-stakes applications—medical advice, legal reasoning, financial analysis—an overconfident falsehood is worse than an acknowledged uncertainty because users may act on it. The birthday example in the introduction (Adam Kalai's birthday, returned as three different incorrect dates including "01-01" and "15-06") illustrates the failure mode: the model was explicitly instructed to respond only if known, yet it nonetheless produced confident-sounding fabrications across multiple trials. This isn't a model failing to retrieve a fact—it's a model failing to recognize the boundary of its own knowledge.
The gap between capability and reliability. Modern LLMs possess enormous factual and reasoning capabilities, but reliability requires not just being right but knowing when you might be wrong. A model that answers 90% of questions correctly but fabricates confident falsehoods on the remaining 10% is far less useful than one that answers 85% correctly and expresses uncertainty on the rest—yet the latter will score lower on almost every mainstream benchmark. This misalignment between evaluation metrics and real-world utility is the socio-technical crux of the paper's argument.
Theoretical significance beyond engineering. The paper argues that hallucinations are not merely an engineering bug but a consequence of statistical learning under the standard cross-entropy objective, even with perfect, error-free training data. This is a much deeper claim than "models are trained on noisy internet text and therefore reproduce errors" (GIGO, Section 3.4). It asserts a fundamental tension between calibration and correctness—a tension that the prior literature had noted in passing but not systematically analyzed in the context of generative language models.
Scale doesn't solve it. The examples involve models spanning multiple scales, architectures, and training paradigms (GPT-4o, DeepSeek-V3, Claude 3.7, Llama-4). All exhibit the same qualitative failure mode. This suggests that scaling model size or training compute alone won't eliminate hallucination—the paper's theoretical analysis explains why: the problem stems from the optimization objectives and evaluation incentives, not from insufficient capacity.

Prior Approaches and Where They Fall Short

The paper situates itself against three broad categories of prior work, each of which it argues is insufficient:

1. Surveys of hallucination causes that are descriptive rather than explanatory. Sun et al. (2025) catalog a wide range of factors contributing to hallucination: model overconfidence (Yin et al., 2023), decoding randomness (Lee et al., 2022), snowballing effects (Zhang et al., 2023), long-tailed training samples (Sun et al., 2023), misleading alignment training (Wei et al., 2023), spurious correlations (Li et al., 2022), exposure bias (Bengio et al., 2015), the reversal curse (Berglund et al., 2024), and context hijacking (Jeong, 2024). While comprehensive, these taxonomies are post-hoc; they describe what factors correlate with hallucination but don't provide a unified statistical framework that predicts hallucination rates from first principles. The paper's innovation is to provide such a framework via the IIV reduction, which subsumes many of these factors (arbitrary facts, poor models, computational hardness, distribution shift) under a single formal structure.

2. Post-training mitigation techniques that don't address the incentive problem. RLHF, DPO, and similar alignment methods can reduce specific types of hallucinations—the paper acknowledges they have been shown to reduce conspiracy theories and common misconceptions (Ouyang et al., 2022; OpenAI, 2023a; Costello et al., 2024). However, these methods operate within an evaluation ecosystem that systematically penalizes the expression of uncertainty. A model trained with DPO to be "helpful, harmless, and honest" may learn to avoid obviously harmful content while still fabricating plausible-sounding answers to factual questions, because the alternative—saying "I don't know"—would lower its score on the binary-graded benchmarks against which it is ultimately measured. The paper's key insight is that post-training is fighting an uphill battle against the gradient induced by the evaluation regime itself.

An important nuance: the paper cites Gekhman et al. (2024) who show that fine-tuning on novel information can initially decrease hallucination rates, only for them to later increase. This suggests that the relationship between post-training and hallucination is non-monotonic and potentially fragile. The paper's framework explains why: fine-tuning shifts the model's distribution toward higher confidence on training examples, which reduces hallucination on in-distribution queries but may increase overconfidence on out-of-distribution ones—especially if the evaluation metric rewards confident output regardless of correctness.

3. Hallucination-specific evaluations that fail to gain adoption. Numerous dedicated hallucination benchmarks exist (Ji et al., 2023; Tian et al., 2024; Bang et al., 2025; Hong et al., 2024). Some—like those measuring calibration error or semantic entropy (Farquhar et al., 2024)—are methodologically sophisticated. Yet the paper observes that these evaluations "have struggled to gain traction within the AI community" (Maslej et al., 2025, cited in Section 2). The diagnosis is structural: a small number of hallucination evaluations cannot counteract the incentive created by the larger ecosystem of primary benchmarks (GPQA, MMLU-Pro, SWE-bench, MATH, IFEval, etc.) that all use binary grading. If an aligned model that never hallucinates loses 5 points on MMLU-Pro because it abstains on uncertain questions, while a hallucinating model gains those 5 points by guessing, the hallucinating model will dominate the leaderboard. Adding more hallucination evaluations doesn't fix this—the dominant evaluations still reward guessing.

The paper is careful to distinguish its critique from the claim that hallucination evaluations are flawed. Rather, the problem is that the ratio of hallucination-aligned to hallucination-rewarding evaluations is too small for the former to influence model development. This is a socio-technical argument: the solution is not to design better hallucination metrics (though that remains valuable) but to modify the existing primary evaluations so that they stop penalizing uncertainty.

4. Prior theoretical work on hallucination inevitability. The most directly related prior work is by Kalai and Vempala (2024), who connected the Good-Turing missing mass estimator (Good, 1953) to hallucination rates, showing that calibrated language models must hallucinate on facts that appear only once in training data. The current paper explicitly builds on and extends this result. The key extensions are:

The IIV reduction provides a more general framework that connects hallucinations to binary classification errors in the Is-It-Valid problem, covering not only arbitrary facts (the setting of Kalai and Vempala, 2024) but also poor models, computational hardness, and distribution shift within a unified analysis. The earlier work is shown to be a special case of the IIV reduction (Theorem 2).
Incorporation of prompts and abstentions. Kalai and Vempala (2024) analyzed unconditional text generation without prompts. The current paper extends the analysis to include prompts (context c from distribution μ) and valid abstention responses like "I don't know" (IDK). This is essential because real-world hallucinations occur in prompted settings and because the IDK option is central to the post-training argument about evaluation misalignment. The paper strengthens the earlier bound to handle this richer setting (Theorem 1 vs. Corollary 1).
The post-training analysis is entirely new. Kalai and Vempala (2024) focused on pretraining; the current paper adds a socio-technical analysis of why post-training fails to eliminate hallucinations, grounded in the observation that binary-graded evaluations make guessing optimal (Observation 1). This bridges the gap between statistical inevitability (pretraining errors) and persistent deployment failures (post-training persistence).

Other theoretical work in the vicinity includes Hanneke et al. (2018), who analyze an interactive learning algorithm that queries a validity oracle to agnostically train a hallucination-minimizing language model—their method is statistically efficient but computationally intractable, making it a theoretical existence result rather than a practical proposal. Kalavasis et al. (2025) and Kleinberg and Mullainathan (2024) formalize a trade-off between consistency (avoiding invalid outputs) and breadth (generating diverse, valid content), showing that any model generalizing beyond its training data must either hallucinate or suffer mode collapse. These results are complementary to the current paper: they establish impossibility for broad classes of languages, while the IIV reduction provides quantitative lower bounds on hallucination rates for specific fact types.

How This Paper Positions Itself

The paper's positioning is distinctive in three ways:

First, it offers a unified theoretical framework (the IIV reduction) that explains multiple hallucination mechanisms through a single lens. Rather than treating arbitrary facts, poor model expressiveness, computational hardness, and GIGO as separate phenomena requiring separate explanations, the paper shows they are all manifestations of high misclassification rates in the IIV binary classification problem, which translates to high generative error rates via Theorem 1. This is more than a conceptual unification—it provides quantitative lower bounds (e.g., err ≥ 2·erriiv - k/K - δ) that relate measurable quantities (calibration error δ, valid-to-error set size ratio k/K) to hallucination rates.

Second, it reframes hallucination reduction as a socio-technical problem rather than a purely technical one. The technical observation that binary grading makes guessing optimal (Observation 1) is trivial—it follows immediately from the definition of binary scoring. The contribution is recognizing that this trivial fact, combined with the dominance of binary-graded benchmarks in the evaluation ecosystem, creates an "epidemic" that cannot be solved by better post-training alone. The paper argues that the route to hallucination reduction runs through modifying the scoring of existing benchmarks (by embedding explicit confidence thresholds into prompts and penalizing incorrect answers), not through designing yet more hallucination-specific evaluations that sit alongside unreformed binary benchmarks.

Third, it connects the pretraining and post-training analyses through a narrative of inevitability and persistence. The pretraining analysis shows that hallucinations are inevitable for calibrated base models trained with cross-entropy on realistic data (Theorem 2: err ≥ sr - 2/min_c|E_c| - ∆). The post-training analysis shows why these errors survive the pipeline despite deliberate mitigation efforts: the evaluation environment rewards the very overconfidence that generates them. Taken together, these two analyses explain both why hallucinations arise and why they are so hard to get rid of—a more complete picture than either pretraining-only or evaluation-only accounts.

The paper is careful to delineate its scope. It does not claim that all errors are hallucinations (the analysis applies to a defined error set E ⊂ X of plausible falsehoods), nor does it claim that search, reasoning, or RAG are useless for hallucination reduction. It explicitly notes (Section 5) that "Search (and reasoning) are not panaceas"—RAG may reduce some hallucinations but does not eliminate the incentive problem because "the binary grading system itself still rewards guessing whenever search fails to yield a confident answer." The framework is meant to characterize the residual hallucination rate that persists after other mitigation techniques are applied, arising from the fundamental statistical pressures of the training and evaluation pipeline itself.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a statistical account of why hallucinations arise and persist, not a system or algorithm. It solves the puzzle of why language models produce plausible falsehoods instead of admitting uncertainty by establishing a mathematical reduction linking generation to classification, then analyzing the incentive structure of evaluation benchmarks.

3.2 Big-Picture Architecture (Diagram in Words)

The analysis has two sequential stages connected by a shared formalism:

The Is-It-Valid (IIV) Classification Problem — a binary classification task where a learner must distinguish valid responses (+) from errors (−). This serves as the bridge: any language model can be converted into an IIV classifier via probability thresholding.
The Reduction from Generation to Classification — a mathematical inequality (Theorem 1) showing that the error rate of any base language model ε is lower-bounded by roughly twice the misclassification rate of its induced IIV classifier, minus a small correction. This converts well-understood classification error sources into predictions about generative errors.
Error Factor Decomposition — mapping classical binary classification failure modes (insufficient data for complex concepts, poor model expressiveness, computational hardness, distribution shift, GIGO) onto language model hallucinations via the IIV reduction.
Post-Training Incentive Analysis — a game-theoretic observation (any binary-graded evaluation makes uncertainty expressions strictly suboptimal) combined with an empirical audit of 10 major benchmarks showing they overwhelmingly use binary grading. This explains why post-training fails to eliminate hallucinations: the evaluation ecosystem systematically rewards guessing.
Proposed Mitigation: Explicit Confidence Targets — modifying existing benchmark prompts to penalize incorrect answers relative to abstention, with the penalty threshold stated explicitly so that the optimal strategy (answer iff confidence exceeds threshold) is computable by both model and evaluator.

3.3 Roadmap for the Deep Dive

First, the formal error set framework (Section 3, foundational definitions): what counts as an error, what counts as valid, and how the language model's error rate is defined. This establishes the ontology used throughout.
Second, the Is-It-Valid (IIV) classification problem and the threshold rule that converts any language model into an IIV classifier. This is the central reduction mechanism.
Third, Theorem 1 and Corollary 1: the mathematical inequality linking generative error rate to IIV misclassification rate, including the role of calibration error δ and why it is small after cross-entropy training. This is the paper's core theoretical engine.
Fourth, the application to three error regimes: arbitrary facts (Theorem 2, using the Good-Turing missing mass), poor models (Theorem 3, connecting to agnostic learning error opt(G)), and computationally hard problems (Observation 2, encryption example). Each maps a known classification failure mode onto hallucination.
Fifth, the post-training analysis: Observation 1 (why binary grading makes abstention suboptimal), the meta-evaluation of 10 benchmarks (Table 2), and the explicit confidence target proposal. This transitions from "why hallucinations exist" to "why they survive post-training" to "how to change the evaluation ecosystem."

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical analysis paper whose core idea is that language model hallucinations can be understood as errors in a derived binary classification problem (the IIV problem), establishing both lower bounds on inevitable hallucination rates during pretraining and an explanation for why post-training fails to eliminate them due to evaluation misalignment.

The Error Set Framework: Defining Hallucination Formally

Before any analysis can proceed, the paper must establish what counts as an error, since hallucination is not a single well-defined category. The framework is set up with careful attention to scope and boundaries.

The universe of plausible strings. The paper restricts attention to a set X of plausible strings — strings that a language model might realistically output (Section 3). This excludes nonsensical strings (gibberish, random character sequences) which state-of-the-art models rarely generate. The set X is partitioned into two disjoint, nonempty subsets:

V: the set of valid examples — responses that are factually correct, logically consistent, and appropriate.
E = X \ V: the set of errors — plausible but incorrect responses.

For hallucination specifically, E is "the set of plausible generations containing (one or more) plausible falsehoods" (Section 3.1, "Hallucination errors"). The paper notes a subtlety: an alternate definition of hallucination is "generations that are not grounded in the training data (or prompt)." The analysis applies to this definition as well because the training data are assumed to contain only valid examples (p(V) = 1), so any generated error e ∈ E is necessarily ungrounded.

The role of plausibility. By restricting to X = V ∪ E and excluding nonsense strings N, the analysis focuses on the hard problem: distinguishing truth from plausible-sounding falsehood. If a model were only evaluated on whether its outputs are grammatical and superficially reasonable (as opposed to factually correct), it would score perfectly on all of X — the challenge is discriminating E from V within X. The paper states that the analysis can be extended to include nonsense strings by defining err := p̂(N ∪ E) and setting D(N) = 0 in the IIV distribution (Section 5, "Plausibility and nonsense"), but this extension is not needed for the main results.

Prompts and responses. In the general setting (Section 3.2), each example x is a pair (c, r) where c is a prompt drawn from distribution μ over a set of prompts C, and r is a response. The training distribution p(r | c) gives the conditional probability of response r given prompt c, extended to a joint distribution by p(c, r) := μ(c)p(r | c). The noiseless training assumption means p(E) = 0: every training example is valid. This is the setting that makes the lower bounds meaningful—even with perfect training data, the model must still err.

The singleton IDK response. The framework explicitly accommodates an "I don't know" (IDK) response as a valid output: "a language model which always outputs IDK also avoids errors" because IDK is assumed to be a member of V (Section 3). This is crucial because it prevents the trivial "solution" of always outputting IDK from being considered successful — such a model has zero error rate but fails at density estimation, the actual goal of language model training. The tension is precisely that good density estimation (matching the training distribution p) forces the model to output non-IDK responses that carry hallucination risk.

What the framework omits. Several limitations are noted (Section 5): (a) open-ended generations are simplified by defining any response containing at least one falsehood as an error, ignoring the possible continuum of partial errors; (b) latent context (user intent not captured by the prompt) cannot be represented, so errors that arise from ambiguous prompts are outside the framework; (c) degrees of uncertainty and hedging are collapsed into the binary correct/incorrect/IDK trichotomy, which the paper calls a "false trichotomy" that is nevertheless an improvement over the "false dichotomy" of correct/incorrect that dominates current evaluation.

The Is-It-Valid (IIV) Classification Problem

The central insight of the paper's pretraining analysis is a reduction from generative error to binary classification error. The reduction proceeds by defining a derived classification problem and then showing that a language model's generative error rate is lower-bounded by its performance on this derived problem.

IIV as a supervised learning task. The Is-It-Valid problem is defined as follows (Section 3.1):

Input space: the set of plausible strings X = E ∪ V.
Target function f: X → {+, −}: f(x) = + iff x ∈ V (the string is valid), f(x) = - iff x ∈ E (the string is an error). This is the "true unknown ground-truth labeling" that a classifier must learn.
Training distribution D: a 50/50 mixture of valid examples drawn from p and uniformly random errors drawn from E. Formally:

$D(x) := \begin{cases} p(x)/2 & \text{if } x \in V \\ 1/(2|E|) & \text{if } x \in E \end{cases}$

where p is the (assumed noiseless) pretraining distribution over valid examples, and |E| denotes the size of the error set (assumed finite for simplicity).

What it computes: the distribution D produces a training example by first flipping a fair coin: heads → sample a valid example from the real language distribution p, tails → sample a uniformly random error from E. Each example is labeled + (valid) or - (error). This constructs a balanced classification problem where the learner sees roughly half valid outputs and half errors during training.

Why this form: the 50/50 mix is a design choice that gives equal weight to detecting errors (false positives) and recognizing valid outputs (false negatives). If the mixture were skewed toward valid examples (say, 99% valid), a trivial classifier that always outputs + would achieve 99% accuracy while being useless. The balanced mixture forces the classifier to actually learn the distinction. The uniform sampling of errors 1/|E| means that at test time, the classifier sees errors drawn uniformly at random from the error set — exactly the distribution a language model would encounter if it generated uniformly among plausible but incorrect outputs. This is a modeling assumption: real language models don't sample errors uniformly, but the uniform distribution provides a worst-case baseline because it makes errors maximally hard to distinguish from valid outputs (no patterns to exploit).

The IIV misclassification rate err_iiv: the standard test error of the induced classifier under distribution D:

$err_{iiv} := \Pr_{x \sim D}[\hat{f}(x) \neq f(x)]$

where ŷ(x) is the classifier's predicted label. This is the probability that a random test example (drawn from the 50/50 mixture) is misclassified by the induced classifier.

The Threshold Rule: Converting a Language Model into an IIV Classifier

Any base language model p̂ (a probability distribution over X) is converted into an IIV classifier f̂ via a single-threshold rule (Section 3.1, Equation 2):

$\hat{f}(x) := \begin{cases} + & \text{if } \hat{p}(x) > 1/|E| \\ - & \text{if } \hat{p}(x) \leq 1/|E| \end{cases}$

where p̂(x) is the language model's assigned probability to string x, and |E| is the size of the error set.

What it computes: for any candidate string x, the classifier looks at the language model's estimated probability p̂(x). If p̂(x) exceeds the threshold 1/|E| (the probability a uniform distribution over errors would assign to any single error), the classifier predicts + (valid). Otherwise, it predicts - (error). In the prompted version (Section 3.2), the rule is generalized to ŷ(c, r) = + iff p̂(r | c) > 1/min_c |E_c|, where min_c |E_c| is the minimum size of the error set across all prompts c.

Why this threshold: 1/|E| is the probability mass assigned to any single error under a uniform distribution over E. A perfectly calibrated language model (matching the training distribution p where p(V) = 1) would assign probability 0 to all errors and positive probability to valid outputs. A random, uninformative model would assign roughly equal probability to all strings. The threshold 1/|E| is the indifference point: if the model assigns more probability to a string than a uniform prior over errors, it "prefers" that string over random noise, suggesting it has learned something about validity. Choosing a different threshold would trade off false positives against false negatives; the paper later shows (Theorem 4) that the optimal threshold may depend on the number of choices C. For the general analysis, 1/min_c |E_c| is chosen because it is the most conservative threshold — it is the strictest possible (requiring the highest probability to count as positive) across all prompts, minimizing false positives on errors.

In the prompted setting (Section 3.2). The threshold becomes 1/min_c |E_c| (the minimum error set size across all prompts) rather than 1/|E|. This is because different prompts may have different numbers of possible erroneous responses (e.g., a multiple-choice question with 4 options has |E_c| = 3, while a free-form birthday query might have thousands). Using the minimum ensures the threshold is valid across all prompts — it is always at most 1/|E_c| for any c. The generalization also extends the IIV distribution: errors are sampled as (c, r) where c ∼ μ and r is uniformly random in E_c.

Theorem 1: The Generative Error Rate Lower Bound

The core mathematical relationship is expressed in Theorem 1 (and its unprompted special case, Corollary 1), which lower-bounds the language model's generative error rate in terms of its IIV misclassification rate.

Statement (Corollary 1, unprompted case):

$err \geq 2 \cdot err_{iiv} - \frac{|V|}{|E|} - \delta$

where:

err := p̂(E) is the language model's generative error rate — the total probability mass it assigns to error strings E (Equation 1). This is what we ultimately want to minimize.
err_iiv is the IIV misclassification rate (defined above).
|V|/|E| is the ratio of the number of valid examples to the number of error examples.
δ := |p̂(A) - p(A)| is a calibration error term, where A := {x ∈ X | p̂(x) > 1/|E|} is the set of strings for which the model's probability exceeds the threshold.

What the bound computes: the inequality provides a lower bound on the generative error rate err in terms of three quantities that are either measurable or theoretically analyzable: the IIV misclassification rate err_iiv (how well the model can distinguish valid from error strings), the relative size of the valid and error sets |V|/|E|, and the calibration error δ (how closely the model's aggregate probability above threshold matches the true data distribution's aggregate probability above threshold). If err_iiv is large (the IIV classification problem is hard) and δ is small (the model is well-calibrated), then err must be at least roughly 2·err_iiv.

In operational terms: suppose we have a concept — like "is this a correct birthday for a specified person?" — where errors outnumber valid statements by a factor of 364 to 1 (for each person, there is one correct date and 364 incorrect dates, plus IDK). Then |V|/|E| ≈ 2/364 is tiny. If the IIV misclassification rate err_iiv is, say, 0.4 (meaning the model can only correctly classify validity 60% of the time on this concept), and the calibration error δ is 0.01, then the generative error rate on birthday prompts will be at least 2·0.4 - 2/364 - 0.01 ≈ 0.79. That is, the model will hallucinate on at least ~79% of birthday queries where the fact was not memorized from training.

Why the factor of 2: the constant 2 arises because errors in the IIV distribution D are equally weighted (50% of samples), while the generative error rate err is the model's total probability mass on E, which can be at most 1. If the IIV misclassification rate were 0.5 (chance level for a balanced binary problem), the bound says err ≥ 1 - k/K - δ, meaning the model would have to hallucinate on almost everything (minus the fraction of valid outputs it accidentally gets right). The factor of 2 is tight: for large |E| and small δ, if err_iiv is near 0.5 (i.e., the classification problem is essentially unlearnable), then err could be near 1 (the model hallucinates on almost all outputs). This is exactly what happens for arbitrary facts that appear only once in training.

The prompted generalization (Theorem 1):

$err \geq 2 \cdot err_{iiv} - \frac{\max_c |V_c|}{\min_c |E_c|} - \delta$

where max_c |V_c| is the maximum number of valid responses for any single prompt (typically 1 for factual queries, or 2 if IDK is counted as valid), min_c |E_c| is the minimum error set size across prompts, and δ := |p̂(A) - p(A)| where A := {(c, r) ∈ X | p̂(r | c) > 1/min_c |E_c|}.

Key property: this bound holds for any base model p̂ — it is a universal lower bound, not an empirical observation about specific architectures. The only assumption about the model is that it is a probability distribution. The bound therefore applies equally to transformer-based models, n-gram models, or any future architecture that learns a distribution over text. This architecture-independence is one of the paper's core contributions: hallucinations are shown to be inherent to the statistical learning problem itself, not an artifact of transformer attention patterns or autoregressive decoding.

Calibration and Why δ Is Small After Cross-Entropy Pretraining

The term δ in the bound measures a specific form of miscalibration: the difference between the model's aggregate probability mass assigned to strings with p̂(x) > 1/|E| and the true data distribution's aggregate probability mass assigned to those same strings.

Definition and interpretation:

$A := \{x \in X \mid \hat{p}(x) > 1/|E|\}$

$\delta := |\hat{p}(A) - p(A)|$

where p̂(A) is the total probability the model assigns to the set A (summing p̂(x) over all x with p̂(x) > 1/|E|), and p(A) is the total probability the training distribution assigns to that same set.

What δ measures: it is the mismatch between the model's "above-threshold mass" and the true data's "above-threshold mass." If the model assigns, say, 30% of its total probability to strings with individual probability exceeding 1/|E|, but the true data distribution only has 25% of its mass in that set, then δ = 0.05. This is a calibration error at a single threshold t = 1/|E|, which is weaker than standard calibration measures like Expected Calibration Error (ECE) that integrate over all thresholds t ∈ [0, 1].

Why δ is small after pretraining. The paper provides a justification via the cross-entropy pretraining objective:

$L(\hat{p}) = \mathbb{E}_{x \sim p}[-\log \hat{p}(x)]$

Consider a rescaled version of the model that multiplies all probabilities above threshold by a factor s > 0 and renormalizes:

$\hat{p}_s(x) \propto \begin{cases} s \cdot \hat{p}(x) & \text{if } \hat{p}(x) > 1/|E| \\ \hat{p}(x) & \text{if } \hat{p}(x) \leq 1/|E| \end{cases}$

The derivative relationship. The paper shows that δ equals the magnitude of the derivative of the cross-entropy loss with respect to the scaling factor s, evaluated at s = 1:

$\delta = \left|\frac{d}{ds}L(\hat{p}_s)\big|_{s=1}\right|$

What this means operationally: if δ were non-zero (say, positive), then slightly rescaling the above-threshold probabilities by some s ≠ 1 would reduce the cross-entropy loss. But if the model p̂ is at a local minimum of the cross-entropy objective — as it should be if training converged — then the derivative must be zero, implying δ ≈ 0. For any model class flexible enough to approximate such a rescaling (which modern neural networks certainly can), gradient-based optimization should drive δ to be small. The paper states: "For any class of language models powerful enough to approximate such simple rescaling, local optimization should yield small δ."

Empirical support. The paper cites Figure 2 (GPT-4 calibration histograms from OpenAI, 2023a), showing that pretrained models are well-calibrated before reinforcement learning. Post-trained models, which deviate from the cross-entropy objective in favor of RL-based optimization, show larger calibration errors. This is important because the lower bound err ≥ 2·err_iiv - k/K - δ becomes stronger (tighter) when δ is small — calibration (small δ) forces hallucination.

The calibrated-vs-non-hallucinating tension. The paper's argument implies that a model can avoid hallucinations (err ≈ 0) only by being miscalibrated (δ large) or by having solved the IIV classification problem perfectly (err_iiv ≈ 0). Since the latter is impossible for arbitrary facts (Section 3.3.1), and the former is incompatible with good cross-entropy training (which minimizes δ), well-trained base models must hallucinate at a rate bounded below by the irreducible IIV error. This is the core of the "calibrated language models must hallucinate" result.

Arbitrary Facts: The Singleton Rate Lower Bound (Theorem 2)

The most concrete application of the IIV reduction is to epistemic uncertainty — facts for which the training data contain insufficient information to distinguish truth from falsehood. The paper formalizes this in the Arbitrary Facts model (Definition 1).

Definition 1 (Arbitrary Facts). The model abstracts away all linguistic variation to focus on the core statistical challenge:

There is a fixed prompt distribution μ(c) and, for each prompt c, a set of possible responses R_c (excluding IDK) and a probability α_c ∈ [0, 1] of answering rather than abstaining.
For each prompt c, a single correct answer a_c ∈ R_c is chosen uniformly at random, independently across prompts. This random choice represents epistemic uncertainty — from the learner's perspective before seeing data, any response in R_c is equally likely to be correct.
The training distribution is: p(a_c | c) = α_c (the correct answer is generated with probability α_c) and p(IDK | c) = 1 - α_c (the model abstains otherwise). Thus E_c = R_c \ {a_c} (all incorrect answers are errors) and V_c = {a_c, IDK} (the correct answer and abstention are both valid).
The model is assumed to be noiseless: no incorrect answers ever appear in training. This is a best-case assumption for the learner — any real training data with errors would produce higher hallucination rates.

What this models: birthday queries (where R_c is the set of 365 possible dates), dissertation titles, or any fact where the answer is arbitrary relative to the available training information. The key property is that the true answer a_c is uniformly random and independent across prompts, so seeing the answer for one prompt provides zero information about the answer for another prompt. This makes the IIV classification problem fundamentally hard — the VC dimension of the function class is enormous (each prompt can have an independent correct answer), so learning requires seeing essentially every prompt-answer pair in training.

The singleton rate sr (Definition 2). The singleton rate extends Alan Turing's missing-mass estimator (Good, 1953) to the setting with IDK abstentions:

A prompt c is a singleton if it appears exactly once in the N training examples {(c^(i), r^(i))}_{i=1}^N without abstention, i.e., |{i : c^(i) = c ∧ r^(i) ≠ IDK}| = 1.
S ⊆ C is the set of all singleton prompts.
The singleton rate is sr := |S|/N, the fraction of training examples that are singletons.

What sr estimates: Turing's insight was that the fraction of items appearing exactly once in a sample is (in expectation) a good estimate of the probability mass assigned to items that have never been seen. Applied to prompts: if 20% of the prompts that were answered (not IDK) in the training data appeared exactly once, then roughly 20% of the probability mass over prompts corresponds to prompts whose correct answer the model has never seen. Since the model cannot know the answer for unseen prompts (by the randomness assumption), it must guess — and guesses will be wrong with probability (|R_c| - 1)/|R_c|, which is near 1 for large response sets like birthdays (364/365).

Theorem 2 (Arbitrary Facts, lower bound). With probability ≥ 99% over the random choice of correct answers a_c and the N training examples:

$err \geq sr - \frac{2}{\min_c |E_c|} - \frac{35 + 6\ln N}{\sqrt{N}} - \delta$

where min_c |E_c| is the minimum number of incorrect responses across prompts (e.g., 364 for fixed-format birthdays).

What the bound says in plain language: the hallucination rate err (the fraction of generated responses that are errors) must be at least the singleton rate sr, minus a correction for the size of the error set (approximately 2/364 ≈ 0.005 for birthdays — negligible), minus a statistical correction that decreases as 1/√N (vanishing for large datasets), minus the calibration error δ. This means: the fraction of hallucinations is at least the fraction of training prompts that appear only once (ignoring IDK).

Example calibration. If sr = 0.20 (20% of answered training prompts are singletons), min_c |E_c| = 364 (birthdays), N = 10^{12} tokens with roughly 10^{11} answered prompts, and δ < 0.01: then err ≥ 0.20 - 0.005 - nearly0 - 0.01 ≈ 0.185. The model will hallucinate on at least ~18.5% of birthday queries.

The matching upper bound. The theorem also provides an upper bound: there exists an efficient algorithm (simple memorization of seen (c, a_c) pairs + uniform guessing on unseen prompts with correct abstention probability) achieving:

$err \leq sr - \frac{sr}{\max_c |E_c| + 1} + \frac{13}{\sqrt{N}}$

with probability ≥ 99%, and with δ = 0 (perfect calibration). For large error sets, sr/(max_c|E_c| + 1) ≈ 0, so err ≈ sr — the hallucination rate is essentially equal to the singleton rate. This shows the lower bound is tight: the singleton rate is both a lower bound on error and (asymptotically) achievable.

How the proof works (Appendix B). The proof connects the IIV misclassification rate err_iiv to the singleton rate via Lemma 2 (which adapts Good-Turing guarantees to the setting with IDK) and then applies Theorem 1 to convert IIV error to generative error. The key technical step is Lemma 1, which extends the standard Good-Turing concentration bound (Corollary 3, from McAllester and Ortiz, 2003) to account for the fact that IDK responses are collapsed and do not contribute to the singleton count. The concentration bounds use the union bound (for error probabilities) and Hoeffding's inequality (for the variance of the sum of independent random variables representing whether each unseen prompt is misclassified). The 35 + 6ln N term comes from simplifying the constants in the Hoeffding and Good-Turing bounds to a clean asymptotic form for the 99% confidence level.

What stronger statement follows compared to Kalai and Vempala (2024). The earlier work had a similar bound but "omitted prompts and abstentions." By including IDK, the current paper's bound accounts for the possibility that models can strategically abstain rather than hallucinate. The bound shows that even with this option available, the hallucination rate among non-IDK outputs must still be at least the singleton rate — abstaining on some queries does not eliminate the hallucination pressure on the queries where the model does answer. This matters because a model could achieve low overall error rate by abstaining on most queries; the paper's bound shows that the error rate conditioned on answering is forced to be high for singleton concepts.

Poor Models: The Agnostic Learning Connection (Theorem 3 and Corollary 2)

The Arbitrary Facts model assumes the target function (mapping prompts to correct answers) has no learnable structure — each prompt's answer is independent. A complementary source of hallucination is poor models: the model's architecture or training procedure cannot represent the true decision boundary, even if it were learnable in principle. The paper formalizes this through the lens of agnostic learning (Kearns et al., 1994).

Agnostic learning error opt(G). Consider a family G of candidate IIV classifiers (e.g., all functions representable by thresholding a base model from a given architecture family at some threshold t). The optimal error achievable by any classifier in G is:

$opt(G) := \min_{g \in G} \Pr_{x \sim D}[g(x) \neq f(x)] \in [0, 1]$

where f is the true IIV target function (distinguishing valid from error) and D is the IIV distribution. opt(G) captures the expressiveness limitation of the model class G: if no classifier in G can approximate the true decision boundary, then opt(G) will be large, independent of how much training data is available.

The induced classifier family. Given a language model p̂_θ parameterized by θ ∈ Θ, the family of classifiers obtained by thresholding p̂_θ at different levels is:

$G := \{g_{\theta, t} \mid \theta \in \Theta, t \in [0, 1]\}$

where each classifier is defined by g_{θ,t}(c, r) := + iff p̂_θ(r | c) > t. This family includes all possible ways to convert the language model into a binary classifier — the specific threshold t = 1/min_c|E_c| used in the reduction is just one member of this family.

Theorem 3 (Pure multiple-choice). In the special case where each prompt has exactly one correct answer (|V_c| = 1 for all c), corresponding to a standard multiple-choice exam with no IDK option:

$err \geq 2\left(1 - \frac{1}{C}\right) \cdot opt(G)$

where C = min_c |E_c| + 1 is the number of choices (the one correct answer plus |E_c| incorrect ones.

What the bound means: if no classifier in the model's representational class G can achieve better than some baseline error rate opt(G) on the IIV problem, then the language model's generative error rate err must be at least 2·(1 - 1/C)·opt(G). For a 2-choice question (C = 2), the factor is 2·(1/2) = 1, so err ≥ opt(G). For a 4-choice question, the factor is 2·(3/4) = 1.5, so err ≥ 1.5·opt(G). This is the key: the IIV error is amplified into generative error by a factor that increases with the number of choices, up to a maximum factor of 2 (as C → ∞).

Proof sketch (Theorem 4 in Appendix C). The proof is elegant. For any fixed model, consider picking a threshold t uniformly at random in [0, 1]. The expected false positive rate (classifying an error as +) is:

$\Pr_{t, x\sim D}[\hat{f}_t(x) = +, f(x) = -] = \frac{1}{2(C-1)} \cdot err$

This is because for any error response r ∈ E_c, the probability that a random threshold t falls below p̂(r|c) is exactly p̂(r | c), and summing this over all error responses and prompts gives err/(C-1) (up to the factor 1/2 from the 50/50 IIV mixture). The expected false negative rate (classifying the correct answer as -) is:

$\Pr_{t, x\sim D}[\hat{f}_t(x) = -, f(x) = +] = \frac{1}{2} \cdot err$

because the correct answer a_c is valid, and the probability that a random threshold exceeds p̂(a_c | c) is 1 - p̂(a_c | c), which sums to err across all prompts (since err = Σ μ(c)(1 - p̂(a_c | c))). Summing these gives the expected misclassification rate in terms of err, and since the minimum over thresholds is at most the expectation, there exists some t achieving at most this expected error. Rearranging gives the bound.

Corollary 2: Trigram models must hallucinate on pronoun-gender agreement. The paper provides a striking concrete example (Section 3.3.2). Consider two prompts that differ only in the gender of the subject:

c_1: "She lost it and was completely out of . . ."
c_2: "He lost it and was completely out of . . ."

with responses:

r_1: "her mind." (correct for c_1, incorrect for c_2)
r_2: "his mind." (correct for c_2, incorrect for c_1)

The prompt distribution μ is uniform over {c_1, c_2}. For this setting, |V_c| = 1 for each prompt (both valid responses contain only the correct answer, no IDK), C = 2 (two possible responses), and opt(G) = 1/2 for the trigram model family. Why? A trigram model conditions only on the previous two words — for c_1, the context for predicting the next word after "out of" is just "... out of", which is identical for both prompts ("... out of" appears in both). The trigram model therefore cannot distinguish between c_1 and c_2 — it sees the same trigram context in both cases. Since the correct response is different (r_1 vs. r_2), any trigram model must get exactly one of the two predictions wrong, giving opt(G) = 1/2. Theorem 3 then gives err ≥ 2·(1 - 1/2)·(1/2) = 1/2. The trigram model must have a generative error rate of at least 50% on these prompts, regardless of training data size.

What this illustrates beyond the specific example: the poor-model error source applies whenever the model's representational capacity is insufficient to capture the relevant distinction. Trigram models cannot capture long-range dependencies (here, the gender of the subject, which may be many words before the prediction point). Modern transformer models handle this specific example, but the same principle applies to more subtle distinctions: if the model's architecture, context window, or training procedure cannot represent the decision boundary needed to separate valid from invalid responses, then opt(G) is large and Theorem 3 forces a correspondingly high hallucination rate.

The letter-counting example revisited. The paper's introduction example — models failing to count the number of Ds in "DEEPSEEK" — is classified as a poor-model issue rather than an arbitrary-fact issue. The evidence: DeepSeek-R1, a reasoning model, reliably counts letters by spelling out the word character by character in a chain of thought. This suggests the underlying model can learn the correct function; the non-reasoning variant fails not because the training data lack the information but because the model's "fast" (non-reasoning) processing represents prompts by tokens (D/EEP/SEE/K) rather than individual characters, making the character-counting function difficult to extract. This is a representational limitation — the model has the knowledge but cannot access it efficiently without explicit reasoning steps.

Additional Error Factors: Hardness, Distribution Shift, and GIGO

The paper briefly analyzes three additional sources of classification error that translate to hallucinations via IIV:

Computational Hardness (Observation 2, Appendix D). Some IIV problems are computationally intractable — no polynomial-time algorithm can distinguish valid from error better than chance. The paper gives a stylized example using symmetric-key encryption:

Valid prompts are of the form "What is the decryption of h?" where h = e_S(r) is a ciphertext encrypting message r under secret key S.
V_c = {r, IDK} (the correct decryption is valid, as is abstaining).
E_c is the set of all messages m ≠ r — these are uniformly random guesses.

A secure encryption scheme (Definition 3) has the property that no efficient classifier can distinguish the true (ciphertext, message) pair from a random one with probability more than (1+β)/2 above chance, for some small security parameter β. Theorem 1 then implies that any language model whose induced IIV classifier cannot break the encryption must have hallucination rate at least:

$err \geq 1 - \beta - \frac{2}{|M|-1} - \delta$

What this means: for a secure cryptosystem with large message space |M| (so 2/(|M|-1) ≈ 0) and small security parameter β ≈ 0, any computationally bounded language model will have hallucination rate near 1 on encryption prompts — it will essentially always output incorrect decryptions. This is not a data limitation (there's a deterministic mapping) but a computational one: the model doesn't have the resources to invert the encryption function, and the IIV reduction shows this forces errors even though IDK is a valid response.

The paper notes that more natural examples of computationally hard problems inducing hallucinations are studied empirically by Fan et al. (2024) and Tang et al. (2025), covering problems from complexity classes like NP-hard tasks and graph computations.

Distribution Shift. When prompts at test time come from a different distribution than training prompts, IIV misclassification rates increase because the classifier encounters inputs it wasn't trained to handle. The paper gives the example: "What's heavier, a pound of feathers or a pound of lead?" — this prompt is likely rare in training and may trigger erroneous answers in models that have seen the correct scientific fact (they weigh the same) but are confused by the adversarial phrasing. This maps to the standard out-of-distribution (OOD) generalization problem in classification.

GIGO: Garbage In, Garbage Out. If training data contain errors (statements labeled as valid that are actually false), then p(E) > 0 — the training distribution itself is noisy. The IIV reduction's assumption p(V) = 1 fails, and the paper notes that "even higher error rates may be expected when the assumption fails." This is the most straightforward error source: language models trained on internet text that contains falsehoods will reproduce those falsehoods. The paper mentions that post-training can reduce certain GIGO errors (conspiracy theories, common misconceptions) through RLHF (Ouyang et al., 2022; OpenAI, 2023a; Costello et al., 2024), but the fundamental statistical pressure from other error factors (arbitrary facts, poor models) remains.

Post-Training: Why Hallucinations Survive (Observation 1)

The pretraining analysis shows that hallucinations are inevitable for calibrated base models. The post-training analysis explains why deliberate attempts to remove them fail: the evaluation ecosystem rewards hallucinatory behavior.

The formal setup (Section 4.1). For any given prompt c:

R_c is the set of all plausible responses (valid or error).
A_c ⊂ R_c is the set of plausible abstention responses (e.g., "I don't know," "I'm not sure," requests for clarification). We assume A_c ≠ ∅.
A binary grader g_c: R_c → ℝ has the property that {g_c(r) | r ∈ R_c} = {0, 1} and g_c(r) = 0 for all r ∈ A_c — abstentions always receive zero credit. This models accuracy scoring, pass/fail, or any metric that awards 1 for fully correct and 0 otherwise.
The test-taker (the language model) knows c, R_c, A_c, and that the grading is binary, but does not know which r receives g_c(r) = 1 (the correct answer). Instead, the test-taker has a posterior belief distribution ρ_c over possible binary graders.

Observation 1 (formal statement). For any prompt c and any distribution ρ_c over binary graders:

$A_c \cap \arg\max_{r \in R_c} \mathbb{E}_{g_c \sim \rho_c}[g_c(r)] = \emptyset$

What this says: the set of optimal responses under expected score never includes any abstention. The proof is trivial but the implication is profound:

Since g_c(a) = 0 for all a ∈ A_c (by definition of binary grading), the expected score of any abstention is E[g_c(a)] = 0.
Since X is finite and g_c maps somewhere to {0, 1}, there must exist some non-abstention response r* ∈ R_c \ A_c with E[g_c(r*)] > 0. If not, then E[g_c(r)] = 0 for all r, meaning no grader assigns a positive score to any response — but then g_c(r) = 0 identically, contradicting {g_c(r) | r ∈ R_c} = {0, 1}.
Therefore, argmax_{r ∈ R_c} E[g_c(r)] contains only responses with strictly positive expected score, which excludes all abstentions (score 0).

Why this is decision-theoretically optimal rather than pathological: a rational expected-utility maximizer, facing a binary grading scheme where abstention yields 0 and guessing has any non-zero probability of yielding 1, will always guess. The expected score of a guess with confidence τ (the test-taker's subjective probability that the guess is correct) is τ · 1 + (1 - τ) · 0 = τ. The expected score of abstention is 0. Since τ > 0 for any guess the test-taker considers possible, guessing strictly dominates abstention. The model doesn't need to be confident — it just needs to believe there's some chance its guess is right.

The connection to hallucination. The paper frames hallucination as an instance of this strategic guessing: "When uncertain, students may guess on multiple-choice exams and even bluff on written exams, submitting plausible answers in which they have little confidence. Language models are evaluated by similar tests. In both settings, guessing when unsure maximizes the expected score." The model's hallucinations are the analog of a student's bluffs — they are the outputs that maximize expected score under the (implicit or explicit) binary grading rubric, even when the model would ideally prefer to express uncertainty.

The Meta-Evaluation: 10 Benchmarks, 9 That Penalize Uncertainty (Table 2, Appendix F)

The paper doesn't just assert that binary grading is prevalent — it audits 10 major benchmarks that collectively dominate the evaluation landscape, analyzing how each treats abstentions.

Selection criteria. The 10 benchmarks were chosen because they appear on influential leaderboards and in model release reports. Table 2 (page 14) lists: GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH (L5 split), MuSR, SWE-bench, and HLE.

Results summary:

Binary grading with no IDK credit (8 of 10): GPQA (multiple-choice accuracy), MMLU-Pro (multiple-choice accuracy), IFEval (programmatic instruction verification — binary rubric sub-scores with no abstention option), Omni-MATH (answer equivalence grading, typically via language model judges, with no credit for IDK), BBH (exact-match or multiple-choice accuracy), MATH L5 (equivalence grading), MuSR (multiple-choice accuracy), and SWE-bench (binary: patch passes unit tests or doesn't).
Binary grading with explicit IDK as an answer option (1 of 10): HLE (Humanity's Last Exam). The paper notes HLE offers a calibration error metric, but careful analysis shows this does not count as giving IDK credit: "A model could hallucinate 100% of the time with 0 calibration error if it always generates incorrect and indicated 0% confidence in each answer." Conversely, "A model could never hallucinate and have 100% calibration error if it always generates correct answers with 0% confidence." Calibration error measures the gap between stated and actual confidence, not whether the model abstains when appropriate. For assessment of whether uncertainty is penalized, the relevant fact is that HLE's primary metric is binary accuracy.
Partial IDK credit (1 of 10): WildBench uses LM-graded rubrics on a 1–10 scale. The rubric explicitly states that a response scoring 5–6 "is fair but has some issues (e.g., factual errors, hallucinations, missing key information)," while a response scoring 3–4 "is poor and does not help the user solve the problem meaningfully." Since an "I don't know" response would likely fall in the 3–4 range (doesn't help meaningfully), it scores lower than a hallucination-containing response in the 5–6 range (at least somewhat helpful). The paper notes: "Thus, the grading may encourage guessing." This is worse than merely giving no credit — it actively penalizes IDK relative to bluffing.

Broader evidence of dominance. The paper documents the influence of these specific benchmarks:

Google's Gemini 2.5 Pro model card (2025) reports results for GPQA, MMLU, SWE-bench, HLE, and AIME (similar to MATH L5).
OpenAI has published results for GPQA, MMLU, SWE-bench verified, IFEval, MATH, and HLE across various model release reports.
Stanford's 2025 AI Index Report includes results for MMLU-Pro, GPQA, WildBench, MATH, SWE-bench, and HLE.
The curated leaderboards had 50% overlap in their top evaluations (GPQA, MMLU-Pro, IFEval appeared on both HELM Capabilities and Open LLM Leaderboard v2).

The "epidemic" framing. The paper argues this is not merely a technical oversight but an epidemic in the epidemiological sense: a condition that is widespread in the population (of evaluations) and self-reinforcing. Because all the primary benchmarks reward guessing, the competitive pressure among model developers pushes toward models that guess rather than abstain — even if individual developers would prefer to build honest models. The mechanism is game-theoretic: if model A is truthful (abstains when uncertain, never hallucinates) and model B is a bluffer (always guesses), model B will dominate on the leaderboard, making model A invisible regardless of its actual trustworthiness. Adding more hallucination-specific evaluations doesn't break this dynamic because the primary benchmarks (which developers optimize for) still reward bluffing. The paper states this explicitly: "a small fraction of hallucination evaluations won't suffice. The numerous primary evaluations must be adjusted to stop penalizing abstentions when uncertain."

A concrete illustration of the competition dynamic. Suppose Model A is an aligned model that correctly signals uncertainty and never hallucinates. Let Model B be similar to Model A except that it never indicates uncertainty and always guesses when unsure. Model B will outperform A under 0-1 scoring, the basis of most current benchmarks. This is direct: every question where Model A would have abstained (earning 0 points), Model B guesses and has some positive probability of earning 1 point. The expected score difference accumulates across all uncertain questions.

Where the meta-evaluation evidence comes from (Appendix F). The appendix provides detailed analysis of how the two major curated leaderboards (HELM Capabilities and Open LLM Leaderboard v2) selected their evaluation sets, showing that the benchmarks were explicitly chosen for quality, recency, saturation levels, and reproducibility — not for their treatment of uncertainty. Both leaderboards include MMLU-Pro, GPQA, and IFEval. HELM additionally includes Omni-MATH and WildBench; Open LLM Leaderboard adds BBH, MATH L5, and MuSR. SWE-bench and HLE are analyzed separately as independently dominant benchmarks. This comprehensive coverage establishes that binary grading is not an occasional flaw but the default design pattern across the evaluations that actually drive model development.

The Proposed Mitigation: Explicit Confidence Targets (Section 4.2)

The paper proposes modifying the instructions of existing benchmarks to state a confidence target explicitly, rather than modifying the underlying scoring function behind the scenes.

The mechanism: append a statement like the following to each evaluation prompt:

"Answer only if you are > t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points."

where t ∈ (0, 1) is a confidence threshold. The grading becomes:

Correct answer: +1 point
Abstention ("I don't know" or equivalent): 0 points
Incorrect answer: -t/(1-t) points (penalty)

What this incentive structure achieves. Under this scoring rule, the expected score of guessing (with subjective confidence τ, the model's belief that its guess is correct) is:

$\mathbb{E}[\text{score} \mid \text{guess with confidence } \tau] = \tau \cdot 1 + (1-\tau) \cdot \left(-\frac{t}{1-t}\right)$

The expected score of abstaining is 0. Guessing is better than abstaining iff:

$\tau - (1-\tau)\frac{t}{1-t} > 0 \iff \tau > t$

The optimal behavior: guess only when the model's subjective confidence τ exceeds the stated threshold t. This is behaviorally calibrated — the model doesn't need to output a numerical probability; it just needs to decide whether its confidence exceeds t and act accordingly. At t = 0, this reduces to standard binary grading ("make your best guess even if unsure"). At t = 0.5, the penalty is 0.5/0.5 = 1 point per error (balanced: guessing at 50% confidence breaks even). At t = 0.75, the penalty is 0.75/0.25 = 3 points per error (the model should guess only when ≥ 75% confident). At t = 0.9, the penalty is 0.9/0.1 = 9 points per error (the model should guess only when ≥ 90% confident).

Why state the threshold explicitly in the prompt. The paper emphasizes two reasons for making the confidence target part of the instructions rather than an implicit scoring rule known only to the evaluator:

Eliminates ambiguity and gaming. If the penalty is not stated, model developers face a tradeoff between accuracy and error rates, and "no single model will be best in general" because different developers may guess different implicit penalties. Stating the threshold makes the optimal strategy transparent to both the model and the evaluator — any model that exhibits behavioral calibration at the stated threshold will achieve its maximum possible score. A single model may be best across all thresholds if it is perfectly calibrated, but if the threshold is implicit, models can exploit the ambiguity to appear better than they are.
Supports consensus and auditing. Standardized tests that use error penalties (JEE, NEET, GATE, AMC, SAT, AP, GRE — some historically, some currently) explicitly state the penalty in test instructions, which allows test-takers to make informed decisions about guessing and enables objective grading. The analogy: "students might bicker that grading is unfair given instructions that there is an unspecified penalty for errors. Instead, specifying confidence thresholds explicitly in each problem's instructions supports objective grading even if the specific thresholds chosen are somewhat arbitrary or even random."

Why this is more effective than adding hallucination evaluations. The paper argues that adding hallucination-specific evaluations with implicit error penalties faces a fundamental problem: the accuracy-error tradeoff. A model that is more aggressively accurate (fewer errors) will necessarily answer fewer questions (more abstentions), losing points on accuracy-centric benchmarks. A hallucination evaluation that penalizes errors rewards the first model; a binary accuracy evaluation rewards the second. As long as the binary accuracy evaluations dominate in number and influence, the second model wins. By modifying existing primary evaluations to include explicit confidence targets, the paper's proposal removes the incentive conflict at its source: the same benchmarks that currently reward guessing would, after modification, reward appropriate expressions of uncertainty.

The "false trichotomy" and its justification (Section 5). The paper acknowledges that the correct/incorrect/IDK categorization is a "false trichotomy" — real-world uncertainty is continuous, and appropriate responses include hedging, omitting details, asking clarifying questions, and expressing nuanced confidence levels, not just binary guess vs. abstain. The authors defend this simplification on pragmatic grounds:

"Although the statistical ideal might be to score each evaluation just as we would like to score the language model in the downstream application, explicit confidence targets offer a practical, objective modification to mainstream evaluations, and a false trichotomy may at least offer an IDK option unlike a false dichotomy."

The argument is that moving from binary (correct/incorrect) to ternary (correct/incorrect/IDK) is the minimum viable change that breaks the guessing-optimality result (Observation 1) and can be implemented at scale by modifying prompt instructions without changing the underlying evaluation infrastructure.

Connection to prior work. The paper notes that penalties for incorrect answers have been "well-studied within hallucination research" (Ji et al., 2023), and that Wu et al. (2025) introduced "risk-informing prompts with explicit penalties" — a direct precedent. The current paper's contribution is arguing that such modifications should be applied to mainstream evaluations (SWE-bench, MMLU-Pro, etc.), not just hallucination-specific benchmarks, and that the threshold should be stated explicitly in the instructions. The strategic insight is that the adoption path runs through modifying the evaluations that developers already optimize for, not through creating new evaluations that compete with them for attention.

Summary of Design Choices and Their Justifications

Error set E restricted to plausible strings: excludes nonsense (which SOTA models rarely generate) to focus on the hard problem of distinguishing truth from plausible falsehood. The extension to include nonsense (X = N ∪ E ∪ V) is trivial but not needed for the main results.
Training distribution assumed noiseless (p(V) = 1): a best-case assumption that makes lower bounds stronger — real training data with errors would only increase hallucination rates.
IIV distribution uses 50/50 valid/error mixture: forces the classifier to learn the distinction rather than defaulting to the majority class, making err_iiv a meaningful lower bound.
IIV threshold 1/min_c|E_c|: chosen to be the most conservative threshold (minimizes false positives) and generalizes across prompts with different error set sizes. The threshold is part of the reduction, not a hyperparameter to be tuned.
Calibration bound via cross-entropy derivative: shows that δ (calibration error at threshold) is small for any model at a local minimum of the cross-entropy loss, connecting the statistical bound to the actual optimization procedure used in pretraining.
Singleton rate definition excludes IDK: adapts Good-Turing missing mass estimation to the setting where models can abstain, ensuring the lower bound applies to the error rate among answered queries rather than being trivially satisfied by always abstaining.
Confidence target stated explicitly in prompt: transforms the evaluation from an implicit (and exploitable) penalty structure to an explicit decision problem where the optimal strategy is computable by both model and evaluator. This supports objective grading and removes the ambiguity that allows models to game implicit penalties.
Ternary scoring (correct/incorrect/IDK) rather than continuous: pragmatic choice that enables immediate deployment ("false trichotomy") while being a clear improvement over the current binary false dichotomy.

4. Key Insights and Innovations

Innovation 1: Hallucination as a Classification Problem — The IIV Reduction

Before this paper, the dominant approaches to understanding hallucination were either taxonomic (cataloging observed failure modes: snowballing, exposure bias, reversal curse, etc., as in Sun et al., 2025) or algorithm-specific (analyzing errors that arise from particular architectural choices like autoregressive decoding). The field had no unified statistical framework that could explain why hallucinations arise across different architectures, training paradigms, and error types from a single principle.

This paper's fundamental conceptual move is the Is-It-Valid (IIV) reduction: the observation that generating valid text is harder than classifying whether text is valid, and that this hardness gap can be quantified mathematically (Theorem 1: err ≥ 2·err_iiv - k/K - δ). This transforms hallucination from a mysterious emergent behavior of neural networks into a downstream consequence of binary classification error — a problem class that has been studied for decades in computational learning theory (Kearns and Vazirani, 1994; Domingos, 2012).

What makes this distinctive is not the reduction technique itself (reductions between learning problems are standard in theoretical CS, e.g., Beygelzimer et al., 2016), but rather where the reduction points: it shows that the same factors that cause misclassifications in supervised learning — insufficient data for high VC-dimension concepts, poor model expressiveness (agnostic learning error opt(G)), computational intractability, distribution shift, and noisy training data — are necessary and sufficient to explain hallucinations in generative models. A practitioner who understands why a classifier fails on a particular problem now understands why a language model hallucinates on the same type of fact. This demystifies hallucination: it is not a ghost in the machine but a statistically expectable outcome of fitting a probability distribution to data where some distinctions are unlearnable.

The significance extends beyond explanation to prediction. Prior taxonomies could tell you that hallucinations happen and categorize them after the fact. The IIV reduction provides quantitative lower bounds: given a measurable quantity like the singleton rate sr (which can be estimated from training data counts) and a model's calibration error δ (measurable via auditing), you can predict the minimum hallucination rate without running the model. Theorem 2 gives this bound explicitly for arbitrary facts: err ≥ sr - 2/min_c|E_c| - O(1/√N) - δ. The bound is both a lower bound (proving inevitability) and asymptotically tight (the upper bound matches). This converts hallucination from a post-hoc empirical observation into a falsifiable prediction of the statistical learning framework — a significant advance in theoretical rigor for a problem that has largely been studied empirically.

The reduction also reconciles prior conflicting findings. Kalai and Vempala (2024) showed that calibrated language models must hallucinate on rare facts, but their analysis omitted prompts and abstentions. Hanneke et al. (2018) showed that interactive learning with a validity oracle could eliminate hallucinations in principle, but their algorithm was computationally intractable. Kalavasis et al. (2025) and Kleinberg and Mullainathan (2024) demonstrated an inherent breadth-consistency tradeoff but didn't give quantitative error bounds. The IIV reduction subsumes the first result as a special case (Theorem 2 extends the singleton-rate bound to prompted settings with IDK), explains why the second is computationally infeasible (Observation 2: if the IIV problem is computationally hard, the generative error rate is near 1), and provides the missing quantitative dimension to the third.

Innovation 2: Calibration as the Mechanism That Forces Hallucination, Not Just Correlates with It

The machine learning community has long known that modern neural networks can be miscalibrated (Guo et al., 2017) and that post-training with RLHF often worsens calibration (Figure 2, OpenAI, 2023a). The standard narrative treats calibration as a desirable property that models should strive for — a well-calibrated model is honest about its uncertainty, which should reduce overconfident errors.

This paper inverts that intuition. The core argument, developed through the δ term in Theorem 1 and its connection to the cross-entropy objective, is that calibration is precisely what forces hallucination in the presence of unlearnable distinctions. The proof: the calibration error δ at threshold 1/|E| equals the derivative of the cross-entropy loss with respect to a simple probability rescaling. A model at a local minimum of the training loss will have δ ≈ 0. When δ is small, the lower bound err ≥ 2·err_iiv - k/K - δ becomes tight: almost all of the IIV misclassification error translates into generative hallucination. A model could avoid hallucination by being miscalibrated — assigning systematically wrong probabilities that don't match the data distribution — but cross-entropy optimization actively prevents this.

This is a diagnostic reframing with sharp practical implications. Before this work, one might have thought: "To reduce hallucinations, we should improve calibration so the model knows when it's uncertain and can abstain." After this work, the correct diagnosis is: "Hallucinations arise because the model is well-calibrated on the training distribution, and that calibration forces it to assign probability mass to errors on unlearnable concepts. The path to reducing hallucinations is either to make the IIV problem learnable (by providing more data or better models) or to deliberately sacrifice calibration on unlearnable facts (by post-training that teaches the model to abstain rather than match the data distribution)."

The paper provides empirical evidence for this inversion in Figure 2 (GPT-4 before and after RLHF). The pretrained model is well-calibrated; the RLHF model is not. This is typically presented as a failure of RLHF (it damages calibration). But through the paper's lens, this miscalibration is functional: RLHF deliberately breaks the calibration that pretraining enforced, allowing the model to abstain (or express uncertainty) on queries where matching the data distribution would require guessing. The downside is that breaking calibration can also introduce overconfidence on other queries. This tension — calibration forces hallucination, but breaking calibration to reduce hallucination introduces new failure modes — is a genuinely novel framing that changes how one thinks about the alignment-calibration tradeoff.

The connection to the cross-entropy derivative (δ = |d/ds L(p̂_s)|_{s=1}) is an elegant theoretical justification but the deeper contribution is the reframing of calibration from solution to cause. This is a fundamental conceptual shift, not an incremental refinement. Prior work treated calibration as a metric to optimize; this paper treats it as a property that is in tension with hallucination avoidance under the standard pretraining objective. The implication is that pretraining alone cannot produce models that are simultaneously calibrated and hallucination-free on arbitrary facts — a form of impossibility result that was not previously articulated in these terms.

Innovation 3: The Evaluation Ecosystem as an Active Cause of Hallucination, Not a Passive Measurement Failure

The standard narrative around hallucination evaluation is: "Current benchmarks don't measure hallucination well, so we need better hallucination-specific benchmarks." This has driven the creation of numerous dedicated hallucination evaluations (Bang et al., 2025; Hong et al., 2024; Lin et al., 2022b; Farquhar et al., 2024) and survey papers calling for better metrics (Ji et al., 2023; Tian et al., 2024). The implicit assumption is that if we could just measure hallucination accurately, the problem would be solved by competitive pressure — developers would optimize for the hallucination metric and hallucinations would decrease.

This paper argues this assumption is structurally wrong. The diagnosis is not that hallucination evaluations are missing from the ecosystem — it's that they are drowned out by the dominant binary-graded benchmarks that actively reward hallucinatory behavior. Observation 1 establishes the game-theoretic core: under any binary grading scheme, guessing (which produces hallucinations when the guess is wrong) strictly dominates abstention in expectation. The proof is two lines, but the implication is devastating: a model that never hallucinates but sometimes says "I don't know" will always lose to a model that always guesses, on every binary-graded benchmark, regardless of how well-intentioned the benchmark designers were.

What makes this an innovation rather than an observation is the socio-technical diagnosis and the proposed intervention path. The paper identifies this as an epidemic in the epidemiological sense: a condition (binary grading) that is widespread in the population of evaluations and self-reinforcing through competitive pressure. Adding more hallucination evaluations is like opening a single healthy restaurant in a food desert — it doesn't change the incentives that drive everyone to eat fast food. The "numerous primary evaluations" (quoting Section 4) that dominate leaderboards and model cards exert far more influence on model development than any specialized hallucination benchmark can counteract.

The meta-evaluation in Table 2 provides the empirical grounding: 9 of 10 major benchmarks use binary grading with no IDK credit; the 10th (WildBench) gives higher scores to hallucination-containing responses than to abstentions. This is not an accident — it's a systematic pattern arising from the fact that benchmark designers optimize for clarity, reproducibility, and ease of grading, all of which push toward binary metrics. The paper's audit of how these benchmarks were selected (Appendix F, examining HELM Capabilities and Open LLM Leaderboard curation criteria) shows that treatment of uncertainty was never a selection criterion — benchmarks were chosen for quality, recency, saturation, and reproducibility, not for whether they penalize honest abstention.

The proposed mitigation — embedding explicit confidence targets into existing benchmark prompts — follows directly from this diagnosis. Rather than creating new evaluations that compete with the dominant ones (and lose, as the 2025 AI Index report documents), modify the dominant evaluations themselves so that they stop rewarding guessing. The confidence threshold t transforms the scoring from binary (guess always optimal) to ternary (guess optimal iff confidence exceeds t), breaking the game-theoretic incentive that drives hallucination. The requirement that t be stated in the prompt (not hidden in the grading rubric) is crucial because it makes the optimal strategy transparent to the model, enabling behavioral calibration that can be objectively audited.

This is a fundamental reframing of the hallucination evaluation problem. Prior work sought the "perfect hallucination eval" — a technical solution to a measurement problem. This paper argues the problem is not measurement but incentives, and the solution is not a better metric but a modification of the scoring rules in existing evaluations. This shifts the burden from evaluation designers (who have been unsuccessfully trying to gain adoption for hallucination benchmarks) to the maintainers of dominant leaderboards (who control the incentives that drive model development). Whether this is feasible as a socio-technical intervention is debatable — but the diagnosis itself is a significant intellectual contribution that changes the terms of the conversation.

Innovation 4: The Singleton Rate as a Computable, Interpretable Lower Bound on Hallucination

The connection between Good-Turing missing mass estimation and hallucination was introduced in Kalai and Vempala (2024). The current paper's innovation is to extend this connection to the prompted, IDK-aware setting and to show that the singleton rate sr — a quantity that can be estimated from training data without running the model — provides a tight lower bound on the hallucination rate for arbitrary factual queries.

What makes this distinctive is the operationalization of an impossibility result. Many theoretical papers prove that certain error rates are unavoidable, but the bounds involve unobservable quantities (e.g., the VC dimension of the true concept class, the total variation distance between train and test distributions). The singleton rate is different: it is defined purely in terms of training data counts (the fraction of answered prompts that appear exactly once), which any organization with access to their training corpus can compute. A practitioner can estimate sr for a category of factual queries (birthdays, dissertation titles, chemical symbols), plug it into Theorem 2, and obtain a prediction: "If we deploy this model, at least X% of answers to queries in this category will be hallucinations, no matter how we do post-training, unless we deliberately miscalibrate the model." This transforms hallucination from something discovered post-deployment into something that can be bounded pre-deployment from training data statistics alone.

The tightness of the bound (the matching upper bound in Theorem 2 shows the singleton rate is asymptotically achievable) means this is not a loose theoretical curiosity but a quantitatively meaningful engineering constraint. If sr = 0.20 for birthday facts in your training data, you cannot reduce the hallucination rate on birthday queries below ~20% without either (a) adding more training data to reduce sr (by repeating facts so they stop being singletons), (b) sacrificing calibration on birthday facts specifically, or (c) abstaining on birthday queries entirely. This gives concrete guidance for data curation: to reduce hallucinations on a class of facts, ensure each fact appears at least twice in training — a simple, actionable rule that follows directly from the theory.

The extension to include IDK in the singleton definition (Definition 2: a prompt is a singleton only if it appears exactly once without abstention) is a subtle but important refinement over Kalai and Vempala (2024). It accounts for the possibility that training data contain abstentions (e.g., "I don't know" responses in dialogue data) which don't help the model learn the correct answer. If a birthday query appears 10 times in training but 9 of those are answered with "I don't know" and only 1 with the correct date, the prompt is still a singleton for the purpose of the bound — the model has seen the correct answer only once. This makes the bound more realistic for training data that include uncertain or abstaining responses, which is common in dialogue and instruction-tuning datasets.

Follow-up empirical work by Miao and Kearns (2025) directly tested this prediction, studying the relationship between hallucination rates, singleton rates, and calibration in practice. Their validation of the theoretical framework demonstrates that the singleton rate is not just a theoretical construct but an empirically predictive quantity — exactly the kind of downstream impact that distinguishes an important theoretical contribution from a merely clever one.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses the MATH benchmark (Hendrycks et al., 2021), consisting of high-school competition-level math problems, specifically the 500-question test split from Lightman et al. (2022). The choice of MATH is deliberate (Section 4): test-time compute is expected to help most on tasks requiring complex logical deduction from knowledge the model already possesses.
Base model(s). All experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" (Section 4), sitting in a useful performance regime: non-trivial MATH accuracy (~10-19% pass@1 depending on prompting and sampling configuration) but far from saturation, leaving room for test-time compute to make a measurable difference. For the FLOPs-matched comparison, a second model with approximately 14× more parameters is used as a pretraining-scaled baseline.
Metrics. The primary metric is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground truth, graded using the grading function from Lightman et al. (2022) (Section 4, Appendix G). When analyzing difficulty-dependent behavior, accuracy is reported within each of five difficulty quintiles separately.
Baselines. The paper evaluates against several baselines: Majority voting (selecting the most common final answer among N sampled solutions, with no learned verifier); ORM best-of-N weighted (scoring N solutions with an outcome reward model and applying best-of-N weighted selection, following Li et al., 2023); PRM best-of-N weighted (scoring N solutions with the process reward model and applying best-of-N weighted selection); Parallel sampling for revisions (generating N independent solutions from the revision model and selecting via verifier or majority vote); and Greedy decoding from the ~14× larger model for the FLOPs-matched comparison.
Generation budget / compute accounting. The universal unit of test-time compute is one "generation" — one complete sampled answer from the base LLM. For best-of-N and beam search, the budget equals N (the number of beams or samples). For lookahead search with k lookahead steps, the cost is N × (k+1) to account for additional rollout computation (Section 5.3). Budgets are swept across powers of 2 from 2⁰ to 2⁹ (1 to 512 generations). FLOPs-matched comparisons (Section 7) use standard approximations: X = 6ND_pretrain for pretraining FLOPs and Y = 2ND_inference for inference FLOPs, where N is model parameters.
Cross-validation / statistical protocol. To prevent strategy selection from overfitting to the test set, the authors use two-fold cross-validation within each difficulty bin on the 500-question test set. The best-performing strategy is selected on one fold and evaluated on the other, with results averaged (Section 3.2). Difficulty bins are determined by sampling 2048 complete solutions per question from the base model and computing pass@1 rate (oracle difficulty) or average PRM final-answer score (predicted difficulty), then binning into five quintiles.

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

Aggregate performance of search algorithms (Figure 3, left). Across all 500 test questions with a maximum budget of 256 generations, the comparison reveals that beam search dominates at low budgets but loses its advantage at higher budgets:

At low budgets (2-8 generations), beam search with M = 4 significantly outperforms best-of-N weighted. At 4 generations, beam search (M = 4) achieves approximately 27% accuracy compared to roughly 16% for best-of-N weighted — a ~11 percentage point gap.
At high budgets (64-256 generations), beam search performance flattens and falls slightly below best-of-N weighted. Best-of-N weighted reaches approximately 38% at 512 generations; beam search (M = 4) plateaus around 34%.
Lookahead search (both k = 1 with M = √N, and k = 3 with both M = √N and M = 4) generally underperforms at the same generation budget due to its higher per-step cost. The 3-step lookahead variants converge to similar performance as other methods at very high budgets but never surpass them.
Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations.

Difficulty-dependent behavior of search (Figure 3, right). When results are broken out by difficulty quintile for beam search (M = 4) vs. best-of-N weighted at four budget levels (4, 16, 64, 256 generations), the performance patterns invert depending on difficulty:

Bin 1 (easiest, ~100 questions): Beam search accuracy decreases from roughly 78% to 77% as budget goes from 4 to 256 generations, while best-of-N weighted increases from 68% to 88%. Beam search's degradation at high budgets is the clearest evidence of PRM over-optimization — the model finds solutions that exploit verifier errors.
Bin 2: Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets.
Bin 3: Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations. The PRM provides genuine guidance toward correct solutions.
Bin 4: Beam search achieves roughly 17% vs. 10% for best-of-N at 256 generations — the strongest relative advantage.
Bin 5 (hardest): Both methods hover near 1-3% regardless of budget. No method makes meaningful progress.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level:

At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a 4× compute reduction.
At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (~37%).
Compute-optimal with predicted difficulty bins (using PRM's own score distribution, no ground-truth labels) tracks the oracle version closely, particularly at lower budgets. The two curves "largely overlap" (Figure 4), with the predicted version reaching approximately 37% at 256 generations.
Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (~29%).

PRM vs. ORM performance (Appendix F, Figure 14). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy vs. roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with sample count, confirming the PRM's superior scaling properties.

Revision Model Results (Section 6)

Revision model pass@1 trajectory (Figure 6, left). The revision model's per-step accuracy improves throughout a revision chain, starting from approximately 18.2% at step 1 and reaching roughly 24-25% by steps 15-20, remaining in the 23-25% range out to 64 steps. This generalization beyond the 4-step training horizon is evidence that the model has learned a generalizable revision skill.

Sequential vs. parallel performance (Figure 6, right). At 64 generations:

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35%

Sequential revisions outperform parallel sampling under both selection mechanisms.

Sequential-to-parallel ratio sweep (Figure 7). For a fixed generation budget, the optimal ratio between sequential revision length and parallel chain count varies with total budget and difficulty:

At 256 generations aggregated across all questions (Figure 7, left), the optimal ratio is around 2¹ to 2³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43-44% accuracy vs. ~40% for fully parallel (leftmost) and ~42% for fully sequential (rightmost). At lower budgets (8-32 generations), fully sequential is optimal — the curves are monotonically increasing with the sequential-to-parallel ratio.
When broken out by difficulty at a fixed budget of 128 generations (Figure 7, right): Bin 1 shows performance essentially flat across all ratios (~90-92%); Bin 2 shows a slight advantage for higher sequential ratios (~63% fully sequential vs. ~58% fully parallel); Bin 3 shows a clear optimal at moderate sequential-to-parallel values (~42% at the peak vs. ~35% at extremes); Bin 4 shows a similar peak at ~18% vs. ~14% at fully parallel; Bin 5 produces roughly 2-3% accuracy regardless of allocation.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin:

At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a 4× improvement.
At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
Compute-optimal predicted bins perform slightly below oracle bins at high budgets (approximately 41% at 256 generations vs. 44% oracle) but still substantially outperform the parallel baseline.
The parallel baseline appears to plateau around 36-37%, while compute-optimal scaling continues to improve, suggesting adaptive allocation gains compound at higher budgets.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

Revisions (Figure 9, left; Figure 1, top-right bar chart). Comparing PaLM 2-S* with compute-optimal revisions against the ~14× larger model (greedy decoding, no extra test-time compute), with results dependent on the inference-to-pretraining token ratio R = D_inference / D_pretrain:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy (bin 1)	+11.8% relative	+3.5% relative	−11.9% relative
Medium (bins 2-3)	+27.8% relative	+16.7% relative	+5.4% relative
Hard (bins 4-5)	+21.6% relative	(implied negative)	−37.2% relative

At R ≪ 1, test-time compute outperforms the larger model across all difficulty levels. At R ≫ 1, test-time compute remains preferable only on the easiest questions, with hard questions showing a −37.2% relative disadvantage.

PRM search (Figure 9, right; Figure 1, bottom-right bar chart). The same comparison using compute-optimal PRM search instead of revisions:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy	+19.1% relative	+2.2% relative	+2.0% relative
Medium	0.0% relative	−35.3% relative	−30.8% relative
Hard	−3.6% relative	−35.3% relative	−52.9% relative

PRM search shows weaker benefits than revisions for FLOPs-matched comparison, with substantial disadvantages on medium and hard questions even at moderate R. Easy questions remain the domain where test-time compute is preferable across all R regimes.

Figure 9 detail. The line plots show accuracy per difficulty bin as test-time compute scales for the smaller model. The ~14× larger model's greedy performance (shown as stars) is placed at three x-axis positions corresponding to the three R values. Where the compute-optimal scaling line exceeds the star, test-time compute wins. On bin 1 (purple line, highest accuracy), the scaling line is above all three stars for revisions. On bin 5 (blue line, lowest near zero), the line is below all three stars and essentially flat near 0-5%.

Ablation Studies and Robustness Checks

PRM step-wise aggregation strategy (Appendix E, Figure 13): The paper compares three methods for combining per-step PRM scores into a single solution score — taking the minimum across steps ("min"), taking the product ("prod"), and using only the final step's prediction ("last"). Contrary to prior work by Lightman et al. (2023) and Wang et al. (2023) which found "min" to be best, the current paper finds "last" performs best: at 256 samples, "last" achieves roughly 37%, "min" achieves roughly 35%, "prod" achieves roughly 27%, and ORM achieves roughly 34%. The authors hypothesize that soft Monte Carlo labels (rather than binary correctness labels) cause the different ranking. This matters because "last" effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM — suggesting step-level PRM training provides beneficial representation learning even when intermediate predictions aren't directly used.

PRM vs. ORM scaling (Appendix F, Figure 14): As noted above, the PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%.

Revision-specific verifier (Appendix J, Figure 15a): The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs: sequential + base-LM PRM achieves roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern — verifiers trained on base model outputs transfer imperfectly to revision model outputs.

Revision history in verifier context (Appendix J, Figure 15b): Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (approximately 1-2 percentage points at 64 generations), but both variants outperform the parallel baseline, confirming the sequential sampling benefit is not solely attributable to the verifier seeing more context.

Oracle vs. predicted difficulty bins (Appendix C, Figures 11-12): Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (~41% vs. ~44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4). This is the critical robustness check: the compute-optimal strategy works without ground-truth labels, though with some degradation at the highest budgets.

Majority voting for revisions (Appendix B, Figure 10): The sequential-to-parallel ratio trends observed with verifier-based selection are replicated using majority voting: easy questions are insensitive to ratio, harder questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This demonstrates the sequential sampling benefit is not an artifact of the verifier.

ReST^EM revision model (Appendix K, Figure 16): An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) backfires: additional sequential revisions substantially hurt performance. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly. This is a notable negative result highlighting the sensitivity of revision training to data generation methodology.

Critical Assessment

Claim: "Compute-optimal scaling improves efficiency by more than 4× over best-of-N." This is the paper's headline quantitative finding, and the evidence supports it but with important scope limitations. The 4× figure appears in two places: (a) compute-optimal search at 16 generations matching best-of-N at 64 generations (Figure 4), and (b) compute-optimal revisions at 64 generations matching parallel best-of-N at 256 generations (Figure 8). These comparisons are made at specific budget levels and the factor shrinks at higher budgets — for revisions, compute-optimal predicted bins achieve only ~41% at 256 vs. baseline ~38%, closer to a 1.3× effective gain. The 4× claim is thus most reliable in lower-to-moderate budget regimes and weakens as budgets grow. More critically, the difficulty estimation cost is not included: generating 2048 samples per question to bin difficulty consumes far more compute than the largest test-time budgets studied (256-512), which the paper acknowledges but does not account for ("our experiments do not account for this cost largely for simplicity," Section 3.2). The reported 4× figure is therefore an upper bound on strategy execution efficiency, not a realized deployment gain. Until cheaper difficulty estimation (e.g., from a small number of initial samples) is demonstrated, these gains are potential rather than actual.

Claim: "Test-time compute with a smaller model can outperform a ~14× larger model." The evidence supports this claim with sharp, well-characterized boundary conditions. The FLOPs-matched comparison (Figure 9) shows that for easy-to-medium problems when the inference-to-pretraining ratio R is small (R ≪ 1), test-time compute yields +27.8% relative improvement over the larger model for revisions and +19.1% for PRM search. At R ≈ 1, the advantage narrows but persists for revisions on easy-to-medium questions. At R ≫ 1, the advantage disappears or reverses on all but the easiest questions. The difficulty-dependence is also stark: on the hardest questions (bin 5), test-time compute is uniformly worse than pretraining at all R values, with relative disadvantages of −37.2% (revisions) and −52.9% (PRM search) at R ≫ 1. The paper is appropriately cautious about these boundaries.

However, a significant limitation weakens this comparison: the ~14× larger model uses greedy decoding with no test-time augmentation of its own. A fairer baseline would give the larger model some modest test-time budget (e.g., best-of-8 or best-of-16). Since test-time compute shows diminishing returns, even a small allocation to the larger model could shift some of the comparisons. Additionally, the larger model scales only parameters while holding data fixed (LLaMA-style), not the compute-optimal Chinchilla scaling that would scale both parameters and data equally. The paper acknowledges this explicitly (Section 7) and leaves Chinchilla-optimal comparison to future work. A compute-optimally trained larger model would likely be a stronger baseline, potentially reversing some of the close-call comparisons.

Claim: "Efficacy depends critically on prompt difficulty." This is the most robustly supported claim in the paper. The difficulty-bin analyses for both search (Figure 3, right) and revisions (Figure 7, right) show qualitatively different — sometimes opposite — effects at different difficulty levels. Beam search degrades easy-problem performance at high budgets due to PRM over-optimization while improving medium-problem performance substantially. Sequential revisions are optimal for easy problems while balanced sequential-parallel ratios are optimal for harder ones. The hardest problems (bin 5) show near-zero improvement from any strategy at any budget. These patterns are consistent across oracle and predicted difficulty bins, across both search and revision mechanisms, and under both verifier-based and majority-vote selection. The difficulty-dependence is the paper's most replicable finding.

That said, the difficulty bins are defined relative to the specific base model (PaLM 2-S*) on the specific task (MATH). Whether the same difficulty categories map to other models or tasks is open. A concept that is "hard" for PaLM 2-S* might be "medium" for a stronger model. The difficulty-dependent strategy selection is adaptive to this (model-specific bins), but the generalizability of the five-bin discrete partitioning to other model/task combinations is untested.

Missing experiments that would have strengthened the analysis:

Combined search and revisions. The paper studies PRM tree-search and iterative revisions as independent mechanisms but never combines them — e.g., using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision trajectories to pursue. The paper acknowledges this gap (Section 8), but it means the reported results are a lower bound on what a fully integrated system could achieve. The complementary difficulty-dependent strengths (revisions help easy problems, search helps medium problems) strongly suggest a combined system would outperform either alone.
Different model families and scales. All experiments use PaLM 2-S*. The claim that this model is "representative of the capabilities of many contemporary LLMs" (Section 4) is unverified. The over-optimization behavior of the PRM, the effectiveness of revision training, and the difficulty bin thresholds all depend on the base model's output distribution. Replication on models from different families (Llama, GPT, Gemini) and at different scales would be necessary to establish the universality of the difficulty-dependent scaling patterns.
Cost-included difficulty estimation. The entire compute-optimal framework depends on difficulty estimation, yet the paper never demonstrates a cost-effective method. A natural experiment would be to estimate difficulty from a small number of initial samples (say, 4-8) rather than 2048, and show that the resulting strategy allocation still outperforms baselines after including the estimation cost. Until this is demonstrated, practitioners reading the paper gain theoretical insight but not a deployable recipe.
Latency analysis. The paper measures compute in generations but ignores wall-clock time. Sequential revisions require serial execution (each revision depends on the previous one), while parallel best-of-N can leverage concurrent hardware. A strategy allocating 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than fully parallel generation. For latency-sensitive deployments, the sequential-heavy strategies the paper's compute-optimal policy recommends on easy problems may be impractical regardless of accuracy gains. No experiment addresses this tradeoff.
Larger test sets. The 500-question test set, divided into five difficulty quintiles of ~100 questions each, then further split by two-fold cross-validation (so ~50 per fold per bin), yields small sample sizes for strategy selection. The paper does not report confidence intervals on the compute-optimal scaling curves, making it impossible to assess whether the observed gains are statistically robust at this sample size.

Overall assessment: The experiments convincingly demonstrate the paper's core qualitative insight (difficulty-dependent strategy allocation is necessary because different methods shine on different problem types) and provide quantitative evidence for the 4× efficiency gain claim in specific budget regimes. The FLOPs-matched comparison properly characterizes the conditions under which test-time compute substitutes for pretraining. However, the experimental scope is limited to a single model family on a single benchmark, the difficulty estimation cost is unaccounted for, the pretraining baseline could be stronger (no test-time compute, non-Chinchilla-optimal training), and the natural combination of search with revisions is unexplored. The paper should be read as establishing the framework and proof of concept for compute-optimal test-time scaling, with the demonstration of practical deployment feasibility (cheap difficulty estimation, multi-model replication, latency-aware allocation) left to future work. The strongest finding — that difficulty-dependence is critical and that performance relationships are often non-monotonic — is well-supported and likely to generalize, given its consistency across methods and selection mechanisms.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Prohibitively Expensive

The entire compute-optimal framework depends on assigning each prompt to a difficulty bin before allocating the test-time compute budget. The paper's method for doing so — generating 2048 complete solutions per question and computing either the pass@1 rate (oracle) or the average PRM final-answer score (predicted) — consumes orders of magnitude more compute than the test-time budgets being optimized. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence is that the paper's headline efficiency gains — the 4× improvement over best-of-N (Figures 4 and 8) — are computed after difficulty is known, without amortizing the cost of learning it. In any realistic deployment, the total cost would be difficulty estimation (2048 generations per prompt, plus PRM scoring) plus strategy execution (say, 64–256 generations). The former dominates: even the smallest test-time budget studied (1 generation) would require a 2048× overhead for difficulty estimation. This means the 4× figure is an upper bound on achievable efficiency — an existence proof that better allocation is possible, but not a deployable system. A practitioner reading the paper gains no guidance on how to balance difficulty estimation cost against strategy execution cost. The paper frames this as an exploration-exploitation tradeoff (Section 3.2) and suggests future work on training models to predict difficulty directly from question text, but no such model is developed or evaluated.

What evidence exists: None. The paper does not include a single experiment that accounts for difficulty estimation cost in any budget calculation. No ablation varies the number of samples used for difficulty estimation, tests whether a smaller number (say, 8 or 32 samples) suffices, or compares the efficiency loss from coarse difficulty estimates against the savings from reduced estimation cost. Section 3.2 states the issue and defers it entirely to future work.

Mitigation status: Acknowledged but not addressed. The paper notes that "predicting difficulty in this way is an important avenue for future exploration" (Section 8), suggesting models that directly estimate difficulty from question text, or adaptive schemes that estimate difficulty from initial samples during the solution process. Neither is implemented.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Does Not Create New Capability

Across every method studied — best-of-N, beam search, lookahead search, sequential revisions, parallel revisions, and compute-optimal combinations thereof — the hardest difficulty quintile (bin 5) shows near-zero improvement at any budget. This is visible across multiple figures:

Figure 3 (right): bin 5 accuracy hovers at 1–3% for both beam search and best-of-N weighted across all budgets from 4 to 256 generations.
Figure 7 (right): bin 5 shows roughly 2–3% accuracy regardless of sequential-to-parallel ratio at a fixed budget of 128 generations.
Figure 9: the bin 5 scaling line is essentially flat near 0–5% for both revisions and PRM search, and at all three values of the inference-to-pretraining ratio R, the ~14× larger model (even with only greedy decoding) outperforms the smaller model with compute-optimal test-time scaling. The relative disadvantage reaches −37.2% (revisions) and −52.9% (PRM search) at R ≫ 1.

The consequence is a hard capability bound: test-time compute amplifies existing capability — finding and refining correct solutions the base model can already produce at some non-trivial rate — but cannot create capability that the base model lacks. If the base model's pass@1 on a problem class is near zero (as it is for bin 5 questions in MATH), then no amount of search, revision, or adaptive allocation will help, because there are simply no correct solutions in the proposal distribution to find or refine. This is not a limitation of specific methods but a fundamental constraint: test-time compute operates over the base model's output distribution, and if that distribution places negligible probability on correct answers, the expected return from any amount of search or revision is negligible.

The paper is transparent about this limitation. The takeaway box in Section 7 states: "On questions where the model cannot produce a correct solution (pass@1 ≈ 0%), essentially no method helps." But the implication for deployment is significant: there is no path forward for genuinely novel or out-of-distribution reasoning using test-time compute alone. For problem classes that exceed the base model's capability, the only viable strategy is scaling pretraining (more data, larger models, better training). The FLOPs-matched comparison quantifies this tradeoff: on hard problems, every additional FLOP spent on test-time compute would have been better spent on pretraining.

What evidence exists: The failure on bin 5 is consistent across all figures — search (Figure 3), revisions (Figure 7), compute-optimal scaling (Figures 4, 8), and FLOPs-matched comparisons (Figure 9). The paper's qualitative examples (Appendix M) are consistent with models producing degenerate or nonsensical outputs on the hardest questions regardless of budget.

Mitigation status: None. The paper does not propose any method for improving performance on problems where pass@1 is near zero. The finding is treated as a fundamental limitation rather than a solvable problem: "test-time compute is most effective on moderately difficult problems where the model is capable but not yet reliable" (Section 7). The implicit recommendation is that hard problems should be routed to larger pretrained models rather than given more test-time compute.

The `~14×` Larger Model Baseline Is Not Compute-Optimally Trained and Uses No Test-Time Compute

The FLOPs-matched comparison in Section 7 — which is central to the paper's claim that test-time compute can substitute for pretraining — compares PaLM 2-S* with compute-optimal test-time scaling against a model with ~14× more parameters. The larger model is trained by scaling parameters while holding data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining à la Chinchilla (Hoffmann et al., 2022), where both parameters and data are scaled equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Additionally, the larger model uses greedy decoding with no test-time augmentation — no majority voting, no best-of-N, no search.

The consequence is that the pretraining baseline is weaker than it should be on two fronts. First, a Chinchilla-optimal model trained with 14× more total FLOPs would distribute additional compute across both more parameters and more data, likely achieving better performance than a parameter-only-scaled model at the same total FLOP budget. Second, giving the larger model even a modest test-time compute budget (say, best-of-8 or majority voting at 16 samples) would create a stronger comparison point. Since the paper's own results show that test-time compute yields large initial gains (the steep part of the scaling curves in Figures 3 and 6 occur at low budgets), allocating even a small fraction of the total inference budget to the larger model's test-time compute could shift several of the head-to-head comparisons.

Quantitatively, the paper reports that on easy-to-medium questions at R ≪ 1, the smaller model with compute-optimal revisions outperforms the larger model by +27.8% relative (Figure 1, Figure 9). But if the larger model were Chinchilla-optimally trained and given best-of-8, this margin might narrow, disappear, or reverse. The paper's comparison answers the question "Is test-time compute with a small model better than naïve pretraining scaling with greedy decoding?" — a weaker question than "Is test-time compute with a small model better than optimal pretraining scaling with modest test-time augmentation?" The practical decision facing an organization allocating compute is closer to the latter: they would compare their best test-time strategy with a smaller model against their best strategy with a larger model, not against a deliberately weakened larger model.

What evidence exists: The paper explicitly notes the parameter-only scaling choice in Section 7. The sensitivity of the comparison to these choices is not explored — there is no ablation using a Chinchilla-optimal baseline, and no experiment giving the larger model any test-time compute.

Mitigation status: Acknowledged as a scope limitation. The authors state they leave Chinchilla-optimal scaling comparison to future work (Section 7). The bar charts in Figure 1 show three R values and three difficulty groupings, making the contingency of the comparison visible, but the specific numbers should be interpreted as an upper bound on the advantage of test-time compute over pretraining under generous assumptions about the pretraining baseline.

Revisions and Search Are Studied Independently — The Complementary Mechanisms Are Never Combined

The paper's unifying framework (Section 2) casts all test-time compute methods as modifications to either the proposal distribution (what the model generates — e.g., via iterative revisions) or the verifier (how outputs are selected — e.g., via PRM-guided search). The empirical results show that these two axes have complementary difficulty-dependent strengths: revisions work best on easy problems where the model needs local refinement (Figure 7, right, where purely sequential revisions dominate on bins 1–2), while PRM search works best on medium-hard problems where the model needs to explore different solution strategies (Figure 3, right, where beam search outperforms best-of-N on bins 3–4). The natural next step — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision trajectories to pursue — is never taken.

The paper states this gap explicitly in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence is that all reported results represent a lower bound on what a fully integrated system could achieve. A combined system could, for example, use beam search over revision model outputs — at each step of the search tree, the model conditions on previously rejected branches as context, producing higher-quality candidate steps than the base model alone. Or the PRM could score intermediate revision steps to decide when a revision trajectory is worth continuing versus when to restart from scratch. Since the two mechanisms show complementary strengths on different difficulty tiers, combining them could yield gains that exceed the sum of their individual benefits — particularly on medium-difficulty problems where both sequential refinement (revisions) and global search (beam search) independently help.

The paper's compute-optimal policy currently selects between search strategies (best-of-N vs. beam search) and between revision strategies (sequential vs. parallel ratios) independently, never considering allocations that mix the two axes. A joint optimization over both the proposal distribution modification method and the verifier/search method could discover regimes where, for example, a small amount of revision is combined with beam search on medium problems — an allocation pattern that the current experimental design cannot express.

What evidence exists: The difficulty-bin analyses (Figures 3 right and 7 right) show the complementarity — revisions peak on easy bins, search peaks on medium bins — but there is no experiment measuring the performance of a combined approach. The gap is theoretical as well as empirical: the paper's formalism for optimal strategy selection (Equation 1 in Section 3.1) operates over a strategy space θ that includes search algorithm choice and revision parameters but never defines strategies that combine modifications to the proposal distribution with verifier-guided search.

Mitigation status: Explicitly acknowledged as a direction for future work. Section 8 states that combining PRM tree-search with revisions "is a natural next step" and could "break through the performance ceiling that each method individually hits, particularly on medium-difficulty problems." No preliminary results or analysis are provided.

Single Benchmark, Single Model Family — Generality Is Unestablished

All experiments use exactly one benchmark (MATH, 500 test questions) and one base model family (PaLM 2-S*, with a ~14× larger PaLM 2 variant for the FLOPs-matched comparison). The authors state:

"we believe this model is representative of the capabilities of many contemporary LLMs" (Section 4)

but this claim is unverified. The paper's core findings — difficulty-dependent strategy optimality, beam search over-optimization on easy problems, revision model effectiveness varying with sequential-to-parallel ratio, the 4× efficiency gain from adaptive allocation — could depend critically on PaLM 2-S*'s specific properties: its calibration behavior, its PRM quality, its revision model's ability to learn from edit-distance-paired training data, and its output distribution on MATH problems.

The consequence is uncertain generalizability along several dimensions:

Model architecture and scale. PaLM 2-S* represents one point in a large space of architectures, scales, and training procedures. A model with different calibration properties (the δ term in Theorem 1) would exhibit different hallucination rates and different over-optimization behavior. A larger model might shift the difficulty bin boundaries substantially — problems that are "medium" (bin 3) for PaLM 2-S* might be "easy" (bin 1) for a stronger model, changing which strategies are compute-optimal.
Task domain. MATH consists of competition-level symbolic reasoning problems with unambiguous ground-truth answers. Whether the difficulty-dependent patterns extend to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual recall rather than inference is unknown. The PRM training procedure relies on Monte Carlo rollouts with automatic correctness checking (exact answer matching), which is feasible for MATH but may be expensive or impossible for tasks without clean verification signals.
Test set size and statistical reliability. The 500-question test set, divided into five difficulty quintiles of ~100 questions each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4 and 8), making it impossible to assess whether the observed efficiency gains are statistically robust at this sample size. A practitioner cannot determine whether the 4× improvement is a reliable expectation or a point estimate with wide variance.
Language. All experiments are on English-language math problems. The revision model's ability to learn from in-context incorrect examples may depend on English-specific properties of the training data.

What evidence exists: The paper presents all experiments on the single model-benchmark pair. No replication on other models, datasets, or tasks. No confidence intervals or statistical significance tests are reported for the main results. The paper's qualitative examples (Table 1, introduction) show hallucinations on factual queries from models other than PaLM 2, but these are illustrative anecdotes, not systematic evaluations.

Mitigation status: Not addressed. The paper does not claim generalizability beyond what was tested. The limitation is a scope constraint — the paper establishes a framework and provides proof-of-concept evidence on one model and one benchmark, leaving replication and extension to future work.

The Revision Model Has a High Correct-to-Incorrect Reversion Rate and Revision Training Is Fragile

The revision model is trained on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). At inference time, a revision chain may produce a correct answer at some step, and then — because the model has never been trained to recognize that the current answer is already correct — it may "revise" that correct answer into an incorrect one at the next step. The paper reports a specific failure rate:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

The paper mitigates this by selecting the best answer from any point in the revision chain (via majority voting or verifier-based selection) rather than always taking the final revision. However, this is a workaround, not a solution.

The consequence is that revision chains are inherently unstable. The model is statistically guaranteed to corrupt roughly 38% of the correct answers it produces during a chain. The selection mechanism (picking the best from the chain) partially compensates, but it introduces a new tradeoff: longer chains produce more opportunities to find correct answers but also more opportunities to corrupt them. If the selection mechanism is imperfect (majority voting can fail; verifiers have their own error rates), the net benefit of additional revision steps may vanish or reverse at some chain length. This is consistent with Figure 6 (left), where revision model pass@1 plateaus around 24–25% after roughly 15–20 steps and shows diminishing returns. The ~38% reversion rate means that in a chain of 64 steps, on average, a correct answer produced at some step has only ~62% probability of surviving to the next step without being corrupted — making very long chains increasingly wasteful.

Additionally, the ReST^EM experiment exposes fragility in the revision training methodology. Attempting to further optimize the revision model using ReST^EM (Singh et al., 2024) — an on-policy RL-style training procedure — caused performance to degrade with sequential revisions (Appendix K, Figure 16). At 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal ratio with the original revision model. The authors hypothesize that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This suggests the successful revision training procedure depends on specific, fragile choices (offline data construction, edit-distance-based incorrect-correct pairing) that do not transfer to more sophisticated self-improvement loops.

What evidence exists: The 38% figure is reported in Section 6.1 (attributed to the naive approach without within-chain selection). The plateau in revision model pass@1 (Figure 6, left) and the diminishing returns from longer chains are consistent with the reversion problem. The ReST^EM failure is documented in Appendix K (Figure 16) as a negative result.

Mitigation status: Partially addressed. The within-chain selection strategy (majority or verifier-based) reduces the impact of the 38% reversion rate but does not eliminate it. A more principled solution — training the model on trajectories that include "no revision needed" signals when the current answer is correct, or explicitly modeling the probability that a revision is needed — is not explored. The ReST^EM failure is presented as an empirical observation with hypothesized causes but no fix. The paper acknowledges this as an area needing further work (Section 8: "future work could explore combining revisions with better verifier guidance to decide when to continue vs. restart revision chains").

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes hallucination from an engineering bug to a statistical inevitability with a socio-technical persistence mechanism. The magnitude of this shift should not be understated: before this work, the dominant paradigm treated hallucination as a problem to be solved through better data, better architectures, or better post-training techniques — an essentially technical challenge. After this work, the framing becomes: hallucinations arise because the pretraining objective (cross-entropy minimization) forces calibrated models to express uncertainty through probability distributions over outputs, which on unlearnable facts translates directly into generating errors. Post-training fails to eliminate these errors not because the techniques are inadequate, but because the evaluation ecosystem systematically rewards the overconfident guessing that produces hallucinations.

This is not a paradigm shift in the Kuhnian sense — it doesn't overthrow a settled theory — but it is a diagnostic reframing with sharp practical consequences. The paper's core contributions change how specific sub-communities should think about their research programs:

For the pretraining community, the IIV reduction establishes that calibration and hallucination avoidance are in tension under the standard objective. This doesn't mean calibration is bad — it means that pretraining alone cannot produce models that are simultaneously well-calibrated and hallucination-free on arbitrary facts. The implication is that hallucination reduction must happen either by making the IIV problem learnable (adding more data so facts stop being singletons) or by deliberately sacrificing calibration during post-training (teaching models to abstain rather than match the data distribution). This reframes post-training not as a fix for pretraining's mistakes but as a necessary second stage that optimizes a different objective (abstention-aware) than the one pretraining optimized (density estimation). The paper doesn't resolve the tension but makes it explicit and quantified: Theorem 2 gives the quantitative relationship err ≥ sr - 2/min_c|E_c| - O(1/√N) - δ, meaning a practitioner can compute from training data statistics alone what fraction of queries will inevitably produce hallucinations regardless of post-training quality.

For the post-training and alignment community, the paper's most disruptive claim is that the evaluation ecosystem is an active cause of hallucination, not a passive measurement failure. Observation 1 (binary grading makes guessing strictly optimal) is mathematically trivial, but recognizing its implications for the competitive dynamics of model development is not. The paper documents that 9 of 10 major benchmarks use binary grading with no credit for uncertainty expressions (Table 2), and that the benchmarks driving model development were selected for criteria (quality, recency, saturation, reproducibility) that systematically excluded consideration of how they treat abstention (Appendix F). This creates an "epidemic" — a self-reinforcing condition where competitive pressure pushes all models toward guessing behavior because any model that abstains loses on the leaderboard. The implication is stark: no amount of post-training innovation can eliminate hallucinations if the evaluation metrics that drive model development continue to reward them. The solution is not better alignment techniques but modified scoring rules in the evaluations that developers actually optimize against.

For the evaluation community, the paper redirects effort from creating new hallucination-specific benchmarks toward modifying existing primary benchmarks. The paper's meta-argument is that hallucination benchmarks have "struggled to gain traction" (Maslej et al., 2025) not because they're poorly designed but because they're structurally outcompeted by binary-graded benchmarks that dominate leaderboards. Adding a 4th or 5th hallucination evaluation to an ecosystem with 10+ binary-graded evaluations doesn't change the incentive gradient. But modifying the scoring of existing benchmarks — by embedding explicit confidence targets into prompts — transforms the ecosystem without requiring adoption of new evaluations. This is a socio-technical intervention strategy: change the rules of the games people are already playing, rather than trying to convince them to play a different game.

The paper reconciles several prior tensions in the literature:

Why do hallucination mitigation techniques sometimes work and sometimes fail? The paper's framework explains this through difficulty-dependence and evaluation incentives. Kalai and Vempala (2024) showed calibrated models must hallucinate on rare facts; the current paper extends this to prompted settings with IDK and shows the bound tightens with the singleton rate. Hanneke et al. (2018) showed that interactive learning with a validity oracle could eliminate hallucinations in principle but was computationally intractable — the current paper's Observation 2 explains why: if the validity problem is computationally hard (encryption example), hallucination rates approach 1 regardless of interactivity. Gekhman et al. (2024) showed that fine-tuning on novel information can initially decrease then increase hallucination rates — the current paper's framework suggests the non-monotonicity arises because fine-tuning shifts the model's calibration, initially suppressing errors on in-distribution queries but amplifying overconfidence on out-of-distribution ones.
Why do post-training techniques reduce specific hallucinations (conspiracy theories, misconceptions) but not arbitrary-fact hallucinations? The paper's distinction between GIGO errors (Section 3.4) and arbitrary-fact errors (Section 3.3.1) provides the explanation. GIGO errors are learnable patterns in the training data — the model sees both the misconception and its correction sufficiently often to learn the distinction. RLHF can suppress the misconception. Arbitrary-fact errors arise from prompts that appear too rarely for any pattern to be learnable — no amount of RLHF on other facts teaches the model what it never saw. The IIV reduction shows these errors are fundamentally different: one is a learnable classification problem (GIGO), the other has high VC dimension and is provably hard (arbitrary facts). Post-training can address the first but not the second.

Research directions that become more attractive:

Training data curation informed by singleton counts. The paper makes a concrete, actionable prediction: reducing the singleton rate sr for a fact category reduces the minimum hallucination rate on that category. This suggests data augmentation strategies where facts that would otherwise appear only once are repeated (with paraphrasing) to push them above the singleton threshold. This is a directly measurable intervention: compute sr before and after augmentation, then measure hallucination rates.
Verifier and search mechanism development for test-time compute. The prior section's paper (on compute-optimal test-time scaling) showed that verifier quality is the primary bottleneck for test-time compute scaling. The current paper adds a new dimension: verifiers are essentially IIV classifiers operating at test time. Improving verifier robustness (to avoid over-optimization) and training them to recognize when a response is beyond the model's capability (rather than simply scoring it) connects directly to the hallucination framework — a verifier that can output "uncertain" rather than a forced binary score would enable test-time strategies that abstain appropriately.

Research directions that become less attractive:

Pure architectural solutions to hallucination. The IIV reduction applies to any model that learns a probability distribution over text, regardless of architecture. This means that switching from transformers to state-space models, or from autoregressive to diffusion-based generation, doesn't escape the fundamental statistical pressure — if the model is calibrated on the training distribution and the training distribution contains unlearnable distinctions, the model will hallucinate. Architecture-specific failure modes (exposure bias, attention glitches) are real but secondary; they modulate hallucination rates around the baseline set by the IIV error, they don't cause hallucination from nothing.
Hallucination-specific evaluations as a primary intervention. The paper's argument that hallucination evaluations are "drowned out" by binary-graded benchmarks implies that effort spent designing better hallucination metrics, while valuable for measurement, won't reduce hallucination rates unless it's accompanied by reform of the primary evaluations. A perfectly designed hallucination benchmark that captures 0.1% of the evaluation landscape won't shift developer behavior.
Post-training that optimizes for accuracy without addressing the evaluation incentive. If post-training optimizes against a binary-graded reward signal (as RLHF typically does, with human raters judging outputs as helpful/not-helpful on a binary or Likert scale), Observation 1 implies that the optimal policy will involve guessing when uncertain. Unless the reward model itself gives partial credit for appropriate abstention, post-training is optimizing toward hallucination on uncertain queries.

Follow-Up Research This Work Enables

1. Training a lightweight difficulty estimator to close the cost gap in compute-optimal test-time scaling. The prior paper's compute-optimal framework requires estimating prompt difficulty before allocating test-time compute, and the current method (2048 samples per prompt) is far too expensive for deployment. The current paper's IIV framework suggests a natural approach: train a classifier that takes only the prompt text as input and predicts the difficulty bin (or, more ambitiously, the expected singleton rate for facts of that type). This is essentially a meta-IIV problem — can a small model learn to predict which prompts will be hard for a larger model? A strong experiment would: (a) collect difficulty labels using the expensive 2048-sample method on a training set of prompts; (b) train a lightweight BERT-scale classifier to predict difficulty bins from prompt text alone; (c) measure whether compute-optimal test-time scaling using the classifier's difficulty estimates achieves efficiency gains close to those using oracle difficulty bins. The current paper's finding that predicted difficulty bins (using PRM scores) largely track oracle bins (Figure 4 of the prior paper) suggests difficulty is learnable from model behavior; the open question is whether it's learnable from prompt text alone, without running the model at all. Success would make compute-optimal test-time scaling deployable; failure would establish a fundamental cost floor for difficulty-aware allocation.

2. Measuring singleton rates in public training corpora and correlating with downstream hallucination rates. The paper's Theorem 2 makes a quantitative prediction: for arbitrary factual queries of a given type (birthdays, chemical symbols, capital cities, dissertation titles), the hallucination rate should be at least the singleton rate sr (minus small corrections). This is directly testable. A research team with access to a training corpus (or a good proxy like Common Crawl-derived datasets) could: (a) identify a class of factual queries with well-defined correct answers (e.g., "What is the chemical symbol for [element]?" for all 118 elements); (b) count occurrences of each (query, correct answer) pair in the training data to compute sr; (c) evaluate a language model trained on that corpus on the same queries, measuring hallucination rate (fraction of answers that are incorrect, excluding IDK); (d) test whether hallucination rate ≥ sr (the lower bound) and whether hallucination rate ≈ sr (the tightness prediction from the upper bound). Miao and Kearns (2025) have already begun this empirical validation, but a systematic study across multiple fact categories, multiple model scales, and multiple training data distributions would establish the predictive power of the singleton rate as a pre-deployment hallucination risk metric. A crucial stress test: do facts that appear exactly twice (not singletons) have substantially lower hallucination rates, as the theory predicts? This would directly inform data curation strategies.

3. Auditing the behavioral calibration of deployed models under explicit confidence targets. The paper proposes modifying existing benchmark prompts to include explicit confidence thresholds (e.g., "Answer only if you are >75% confident, since mistakes are penalized 3 points"). The key empirical question is: do current state-of-the-art models exhibit behavioral calibration — that is, can they adjust their abstention behavior appropriately when the threshold is stated in the prompt? A researcher could take an existing benchmark like MMLU-Pro or GPQA (which the paper identifies as binary-graded) and create a variant where each question is presented with multiple threshold conditions: t = 0 (standard binary: guess always), t = 0.5 (balanced penalty), t = 0.75 (3:1 penalty), t = 0.9 (9:1 penalty). For each threshold, measure: (a) the fraction of questions the model answers vs. abstains on; (b) the accuracy among answered questions; (c) whether the model's effective confidence (accuracy on answered questions) matches the stated threshold. A model with good behavioral calibration would answer fewer questions and achieve higher accuracy as t increases. Most deployed models likely have never been trained to modulate behavior based on explicit confidence thresholds, so the initial results may show poor calibration — but this establishes a baseline and a training objective. The experiment directly tests the paper's central proposal and would either validate it as immediately deployable or identify the gap between current model capabilities and the desired behavior.

4. Characterizing the computational hardness boundary for IIV problems in practice. The paper's Observation 2 shows that if the IIV problem is computationally hard (encryption example), the hallucination rate approaches 1. But this is a stylized cryptographic example. Many real-world reasoning problems are believed to be computationally intractable (NP-hard, PSPACE-hard) yet language models sometimes solve them — suggesting heuristic reasoning that partially circumvents worst-case hardness. A systematic study could: (a) select a range of computational problems of varying complexity (sorting, graph connectivity, SAT, TSP, theorem proving); (b) measure both the IIV classification accuracy (can the model distinguish correct from incorrect solutions?) and the generative error rate (how often does the model produce incorrect solutions?); (c) test Theorem 1's prediction that err ≥ 2·err_iiv - k/K - δ holds across complexity classes; (d) determine whether the gap between worst-case hardness predictions and observed model performance can be explained by the distribution of problem instances (average-case vs. worst-case) or by the model learning efficient heuristics that succeed on typical instances. This would connect the theoretical framework to the empirical literature on reasoning model capabilities, and potentially identify which complexity classes are genuinely hard for current architectures vs. which are hard only in synthetic worst-case constructions.

5. Combining PRM-guided search with iterative revisions in a unified test-time compute framework. The prior paper's most significant gap — acknowledged explicitly (Section 8) — is that PRM tree-search and iterative revisions are studied independently despite showing complementary difficulty-dependent strengths. The current paper's IIV framework provides the theoretical lens for understanding why this combination should help: revisions improve the proposal distribution (generating better candidates by conditioning on previous errors), while PRM search improves candidate selection (finding the best among generated candidates). A combined system could use the revision model as the proposal distribution within beam search — at each step of the search tree, the model conditions on previously rejected branches, potentially producing higher-quality candidate steps. Alternatively, the PRM could score intermediate revision steps to decide when a revision trajectory is worth continuing vs. restarting. The key experiment: implement combined search-and-revision on the MATH benchmark with the same PaLM 2-S* base model, using the compute-optimal policy framework to allocate budget between revision depth, beam width, and parallel chain count jointly. The hypothesis is that combined allocation achieves accuracy exceeding either method alone at the same total generation budget, particularly on medium-difficulty problems (bins 3–4) where both mechanisms individually show gains but neither saturates performance. A negative result (no improvement from combining) would suggest the mechanisms are redundant rather than complementary — an important finding that would refocus effort on improving each mechanism individually.

6. Training verifiers that output calibrated uncertainty rather than forced binary decisions. Both papers identify verifier quality as a bottleneck: the prior paper shows that verifier over-optimization limits test-time compute scaling (Figure 3, beam search degrading easy-problem performance at high budgets), and the current paper shows that binary-graded evaluations (which include verifier-based selection) reward guessing over abstention. A natural synthesis: train verifiers (PRMs or ORMs) that output a ternary signal — correct, incorrect, or uncertain — with a calibrated confidence estimate, rather than a forced binary score. The training procedure would use the same Monte Carlo rollout methodology but with a modified loss function that includes an abstention option: for steps where the rollout consensus is ambiguous (e.g., 40–60% of rollouts reach the correct answer), the verifier is trained to output "uncertain" and the downstream selection mechanism treats "uncertain" as a signal to either collect more samples or route to a different strategy. This connects directly to the current paper's confidence threshold proposal: verifiers with calibrated uncertainty would enable test-time compute strategies that adaptively decide whether to continue search or abstain, implementing behavioral calibration at the mechanism level rather than relying on the model to decide when to guess. The experiment would compare best-of-N weighted selection using a binary verifier vs. a ternary verifier on the MATH benchmark, measuring both accuracy and the fraction of problems where the system appropriately abstains (outputs IDK) rather than guessing incorrectly.

Practical Applications and Downstream Use Cases

1. Pre-deployment hallucination risk assessment for factual query systems. An organization building a question-answering system over a specific domain (e.g., medical advice, legal information, technical support) can use the singleton rate to bound the minimum hallucination rate before deployment, without running the model on test queries. The procedure: (a) identify the class of factual queries the system is expected to handle; (b) estimate the singleton rate sr for those queries in the training data (or a representative sample); (c) apply Theorem 2 to obtain a lower bound err ≥ sr - 2/min_c|E_c| - O(1/√N) - δ. For birthday-like facts where min_c|E_c| is large (hundreds or thousands), the bound simplifies to err ≥ sr - δ minus negligible corrections. If sr = 0.20 and the model is well-calibrated (small δ), the organization knows that at least 20% of answers to queries in this category will be hallucinations, regardless of post-training quality. This enables informed deployment decisions: the system can be configured to abstain on all queries in high-sr categories (routing them to human experts), or the training data can be augmented to reduce sr by ensuring each fact appears at least twice. The bound is computable from training data statistics alone — no model inference required — making it a cheap pre-deployment diagnostic. This is a direct operationalization of the paper's theoretical contribution: turn the impossibility result into an engineering constraint that guides data curation and deployment architecture.

2. Modifying existing evaluation leaderboards with explicit confidence targets to reduce hallucination incentives. The paper's proposed mitigation — embedding confidence thresholds into benchmark prompts — can be implemented by leaderboard maintainers without changing the underlying tasks or grading infrastructure. The modification: for each question in an existing benchmark (MMLU-Pro, GPQA, MATH, SWE-bench), create a prompt variant that appends a confidence target statement, e.g., "Answer only if you are at least 75% confident, since incorrect answers will be penalized 3 points while correct answers receive 1 point and abstentions ('I don't know') receive 0 points." The grading rubric is adjusted to deduct points for incorrect answers according to the stated penalty. A benchmark reported with multiple threshold variants (t = 0, t = 0.5, t = 0.75, t = 0.9) provides a richer picture of model capability: a model that achieves 85% accuracy at t = 0 but only 60% accuracy at t = 0.75 (with 25% abstention) is less trustworthy than a model that achieves 80% at t = 0 and 78% at t = 0.75 (with 2% abstention), even though the first model has the higher nominal accuracy. Leaderboard maintainers could report accuracy-at-threshold curves rather than single-point accuracy, creating competitive pressure for models that are both capable and honest about uncertainty. The paper's argument that this must happen on primary benchmarks, not new hallucination-specific ones, is the key strategic insight: modifying an existing leaderboard with thousands of users changes developer incentives immediately, while launching a new leaderboard requires building an audience from scratch.

3. Data curation for self-improving language models using singleton detection. Language models are increasingly used to generate training data for themselves (self-play, distillation, rejection sampling). The paper's singleton rate analysis provides a principled criterion for which generated examples are worth including in the next round of training. The insight: a generated (query, answer) pair where the query has a low singleton rate in the current training data is likely to be correct (the model has seen the fact multiple times and can reproduce it reliably). A pair where the query has a high singleton rate is likely to be a hallucination — the model is guessing on an unlearnable fact. A data curation pipeline could: (a) for each candidate generated example, look up the query's occurrence count in the current training corpus; (b) filter out examples where the query count is 1 (singleton) or 0 (unseen), since the answer is likely hallucinated; (c) keep examples where the query count is ≥ 2, since the answer is more likely grounded in training data. This is a simple, computationally cheap filter that could prevent self-improvement loops from amplifying hallucinations — directly addressing the failure mode documented in Gekhman et al. (2024) where fine-tuning on novel knowledge initially decreased then increased hallucination rates. The current paper's Theorem 2 upper bound shows that the hallucination rate achievable given singleton rate sr is err ≤ sr - sr/(max_c|E_c|+1) + 13/√N, meaning that pushing sr toward zero through data repetition can drive the achievable hallucination rate arbitrarily low — a theoretical guarantee that data repetition strategies work.

4. Designing verifier-guided generation systems that know when to abstain. Current systems using verifier-guided generation (best-of-N with PRM scoring, beam search with step-level verification) always produce an answer — the verifier selects the best candidate, and that candidate is returned to the user regardless of its score. The paper's Observation 1 explains why: under binary scoring of the system's outputs, always answering dominates abstention. But in applications where user trust matters more than coverage (medical, legal, financial advice), the system should abstain when the verifier's confidence is low. The paper's confidence threshold framework can be applied at the system level: after the verifier scores all candidates, if the best candidate's score is below a threshold corresponding to confidence t, the system outputs "I don't know" instead. The threshold can be set based on the application's tolerance for errors: t = 0.9 for high-stakes decisions, t = 0.5 for casual information retrieval. This is trivial to implement — it's a post-processing step on the verifier's output — but current systems don't do it because they're optimized for accuracy benchmarks that don't reward abstention. The paper's argument implies that deploying such a system requires changing the internal evaluation metric used by the development team, not just adding the abstention logic — the team must optimize for a thresholded accuracy metric rather than raw accuracy, or the abstention behavior will be trained away during post-training.

When to Prefer This Method

The paper does not propose a single "method" in the sense of an algorithm or system to be chosen over alternatives. It provides a diagnostic framework (the IIV reduction) for understanding where hallucinations come from and a strategic intervention (explicit confidence targets in evaluation prompts) for changing the incentive structure that perpetuates them. The question of "when to prefer this method" therefore decomposes into two separate decision contexts:

When to use the IIV framework to bound hallucination rates:

Prefer analyzing hallucination through the IIV lens when you can define a clean error set E of plausible falsehoods for your domain, the training data are largely noise-free on the relevant facts (or you can estimate the noise level), and you need a lower bound on hallucination rates that doesn't require running the model. This is most applicable to factual QA systems over well-defined knowledge domains (science, geography, mathematics, structured data) where correctness is unambiguous.
The framework is least informative when the error set is ambiguous or contested (subjective judgments, creative writing quality, nuanced policy analysis), when training data noise dominates the signal (GIGO is the primary error source rather than arbitrary facts), or when you need an upper bound on model capability rather than a lower bound on errors.

When to adopt explicit confidence targets in evaluations:

Prefer modifying existing benchmarks with confidence targets when you control a leaderboard or evaluation pipeline that drives model development, you can modify prompt instructions without changing the underlying task, and you want to create competitive pressure for models that express appropriate uncertainty. The paper's argument implies this is most urgent for the dominant benchmarks (GPQA, MMLU-Pro, SWE-bench, MATH, HLE) rather than niche hallucination evaluations.
This intervention is insufficient when the application requires nuanced uncertainty expression beyond the correct/incorrect/IDK trichotomy, the evaluation already uses non-binary scoring that appropriately rewards abstention (rare, per Table 2), or the goal is to measure calibration quality rather than to incentivize behavioral calibration. The paper explicitly notes that the "false trichotomy" is a pragmatic minimum viable change, not an ideal — richer uncertainty representations (hedging, probabilistic statements, clarification requests) require more sophisticated evaluation frameworks that the paper does not develop.