Understanding Chain-of-Thought in LLMs through Information Theory

ArXiv: 2411.11984

🎯 Pitch

You can now detect faulty reasoning steps in LLM outputs without any human annotations: each correct step should increase the mutual information with the correct final answer. The proposed information-gain metric catches errors that outcome-based methods systematically miss—achieving 96% detection accuracy where prior work hits only 77%—by exposing that a step's apparent correctness can be an artifact of predictable earlier mistakes.

---

1. Executive Summary

This paper introduces a formal information-theoretic framework for evaluating Chain-of-Thought reasoning in LLMs without requiring annotated intermediate-step data, quantifying the information-gain at each reasoning step (the conditional mutual information between the step's state and the correct final answer, estimated via a fine-tuned supervisor model's cross-entropy reduction). Tested on toy arithmetic, Llama-3-8B arithmetic, controlled GSM8K, and PRM800k datasets, the method correctly identifies unidentifiable sub-tasks — reasoning steps that lie outside the span of primitive tasks the model could have learned from training data — achieving significantly lower false-positive rates than Outcome Reward Models and Math-Shepherd, such as 96% sample-wise detection accuracy on synthetic data versus 77% for ORM, while establishing that outcome-based baselines systematically fail when error occurrence is predictable from earlier step outputs rather than from the erroneous step itself.

2. Context and Motivation

The Core Problem: We Cannot Reliably Tell Which Reasoning Steps Are Wrong

Chain-of-Thought (CoT) reasoning — breaking complex problems into intermediate steps that "show the work" — has become a dominant paradigm for getting LLMs to solve multi-step mathematical, logical, and planning tasks. The problem is that we lack reliable methods to determine whether each individual step in that reasoning chain is correct. This is not an academic curiosity; it matters for three concrete reasons. First, if we cannot identify which sub-tasks an LLM systematically fails at, we cannot target fine-tuning efforts — we might waste compute retraining the model on operations it already performs perfectly while missing the specific capability gaps that cause most errors. Second, in any deployment where LLM outputs feed into downstream decisions (code generation, financial calculations, medical reasoning), knowing where in a reasoning chain the model diverges from correctness is essential for trust, debugging, and human-in-the-loop oversight. Third, as the field moves toward self-improving systems where models generate and critique their own training data (e.g., STaR, ReST$^{EM}$), step-level evaluation determines whether those feedback signals are accurate or contaminated with false positives.

The paper frames this gap through the concept of unidentifiability: a sub-task is unidentifiable in the training data if no composition of primitive tasks the model was trained on can produce that sub-task correctly. If we knew which sub-tasks are unidentifiable for a given model, we would know exactly what to fix. But the prevailing evaluation paradigm — measuring end-to-end accuracy on final answers — tells us that the model fails but not why or where.

Why Existing Approaches Fall Short (and Why We Cannot Just Annotate Our Way Out)

The paper identifies three categories of prior work and systematically explains their limitations.

Process Reward Models (PRMs) require expensive human annotation. The gold standard for step-level evaluation is process supervision [18], where human annotators label each intermediate reasoning step as correct or incorrect, and a model is trained to predict those labels. This works — Lightman et al. (2023) demonstrated substantial gains on MATH by training PRMs on the 800,000-step PRM800k dataset. But the paper argues this approach carries a fundamental scalability barrier: annotating intermediate steps requires domain expertise (mathematics, logic, coding), making it expensive and slow. For every new domain, new benchmark, or new model output distribution, the annotation pipeline must be repeated. The paper does not dispute that PRMs can be effective — it argues they are impractical for routine use.

Outcome Reward Models (ORMs) avoid annotation but are unsound for detecting step-level errors. ORMs, introduced by Cobbe et al. (2021) and refined by Havrilla et al. (2024), train a classifier that predicts whether a model's final answer will be correct given the reasoning chain up to step $t$. If the predicted probability of correctness drops significantly after step $t$, that step is flagged as likely incorrect. The paper identifies a specific failure mode that makes this approach logically unsound in certain regimes: ORM's correctness prediction can be determined entirely by information present before the erroneous step. Concretely, in the toy experiment described in Section 5.1, task $\lambda_3$ is applied incorrectly if and only if the output after $\lambda_2$ falls into a specific set $S$. An ORM trained on this data learns to predict failure just from the state at step 2 — it sees $z_2 \in S$ and confidently predicts the final answer will be wrong, regardless of what happens at step 3. The drop in ORM confidence occurs at step 2, not step 3, causing the ORM to systematically misattribute the error to the wrong sub-task. The paper generalizes this: whenever error occurrence correlates with features visible before the error, spurious correlations can make ORM flag correct steps as incorrect and miss the actual failure point.

Math-Shepherd (MS) shares ORM's vulnerability and compounds it with completer model bias. Wang et al. (2024) proposed Math-Shepherd, which evaluates a reasoning step by generating multiple completions from that step onward (using the same or a different model) and computing the fraction of completions that reach the correct final answer. The intuition is: if a step is correct, completions from that point should have a high success rate; if incorrect, the success rate should be low. The paper identifies two problems. First, MS inherits ORM's vulnerability — if correctness is determined by features prior to the step being evaluated, all completions will fail regardless of where we start them, making it impossible to localize the error. In the LLM3 toy experiment (Figure 3c), MS shows zero correct completions for all steps, providing no diagnostic signal about which step is actually wrong. Second, MS performance depends critically on the completer model's capability — if the base model is used as the completer (which the paper does for fair comparison), it may be unable to produce correct completions even from correct intermediate states on difficult problems, causing false positives on correct steps. The paper's arithmetic experiments with Llama-3-8B (Section 5.2.1, Figure 10) demonstrate this concretely: for most prompts where the model fails, no completions from any step reach the correct answer, leading to a 99% true positive rate but an 86% false positive rate — the method correctly flags the actual errors but also incorrectly flags nearly every correct step as erroneous.

Step-aware verifiers and completion-based approaches face a deeper circularity when models cannot produce correct solutions. The paper notes that approaches like Math-Shepherd and the step-aware verifier of Li et al. (2023) fundamentally require that some completions or alternative reasoning paths lead to the correct answer. On genuinely hard problems where the base model's pass@1 is near zero, these methods break down because there are no correct trajectories to learn from. This is not merely a theoretical concern — it means that the hardest problems, the ones where we most need diagnostic tools for improvement, are precisely where existing annotation-free methods offer the least signal.

How This Paper Positions Itself

The paper's positioning is direct and ambitious: it argues that information theory provides a principled alternative to both expensive human annotation (PRMs) and unsound outcome-based heuristics (ORMs, MS). Rather than asking "does this step look correct?" (PRM) or "do these completions reach the right answer?" (MS), the framework asks a different question: "Does this step provide information that helps predict the correct final answer?"

The key theoretical move is recasting step-level evaluation as a conditional independence test. If a model encounters a sub-task that is unidentifiable in its training data — meaning no composition of learned primitive tasks can execute it correctly — then under the paper's formal assumptions, the model's subsequent reasoning states become conditionally independent of the ground-truth final answer given the state before the unidentifiable step (Theorem 3.3). In operational terms: after the model's reasoning diverges from the ground truth, no later step adds any information relevant to predicting $Y$. The information-gain at step $j$ — measured as $I(Y; X^M_j \mid X^M_{j-1})$ — should be approximately zero for all steps after the first unidentifiable one.

This framing has several advantages the paper claims over prior work:

• No human annotation required. The supervisor model that estimates $p(Y \mid X^M_t)$ is trained only on prompts and correct final answers — the same data required by ORMs and MS. The step-level signal emerges from the structure of the information-theoretic measurement, not from step-level labels.
• Resistance to spurious correlations. The information-gain measures whether the step itself adds predictive information about $Y$, not whether the step's inputs predict the final outcome. In the LLM3 toy experiment, step 2 correctly contributes information (it narrows down possible final states), so its information-gain is positive. Step 3 adds no information because the model's execution diverges from the ground truth there, so its information-gain drops. This correctly localizes the error to step 3, unlike ORM which attributes it to step 2.
• Granular sub-task evaluation without step-level labels. By aggregating information-gain across all prompts where a particular sub-task (e.g., "addition") appears, the framework provides per-sub-task performance profiles — effectively telling you which operations the model can and cannot do — without ever labeling individual steps as correct or incorrect during training.

The paper also explicitly positions itself as building on the formal framework of González and Nori (2023), which treats LLMs as execution machines with natural language interfaces ("Beyond Words"). That prior work provided the conceptual vocabulary for viewing LLM reasoning as sequential task application; this paper extends it by adding the notions of primitive tasks, span (the closure of primitive tasks under composition), and unidentifiability, then connecting these to information-theoretic measurements that enable practical evaluation.

Connections to Broader LLM Challenges

The motivation section implicitly connects the paper's contribution to several broader themes. The problem of verifying reasoning without ground-truth intermediate steps connects to the broader challenge of scalable oversight — as models become more capable and their reasoning chains become longer and more complex, human verification of intermediate steps becomes infeasible. The paper's framework offers a path toward automated verification that scales with model capability rather than requiring proportional human effort.

Additionally, the paper's focus on unidentifiability — tasks that fall outside what the model could have learned from training data — connects to the generalization and compositionality literature. The work by Feng et al. (2023) theoretically examined what CoT can express; this paper provides a practical method for diagnosing when that expressivity fails on specific sub-tasks.

3. Technical Approach

3.1 Reader orientation

This paper develops an information-theoretic diagnostic framework — not a new model architecture or training procedure — that measures how much each step in an LLM's Chain-of-Thought reasoning contributes to predicting the correct final answer, using only prompt-final-answer pairs without any intermediate step labels. The framework solves the problem of localizing where a model's reasoning fails by testing a specific conditional independence: after the model encounters a sub-task it cannot execute correctly (one unidentifiable in its training data), all subsequent reasoning steps should contribute zero additional information about the ground-truth answer — a prediction the paper operationalizes through a fine-tuned "supervisor model" that estimates $p(Y \mid X^M_t)$ at each step $t$.

3.2 Big-picture architecture (diagram in words)

The system has four major components connected in a pipeline:

1. Task Decomposition Layer (conceptual, not computational): A formal language that decomposes any task $\lambda$ into a sequence of primitive tasks $\lambda_1 \circ \dots \circ \lambda_T$, and defines what it means for a sub-task to be unidentifiable — lying outside the span of primitive tasks available in the model's training data. This layer provides the theoretical vocabulary but does not itself run any computation.

2. Base LLM (the model under evaluation): The language model whose CoT reasoning is being assessed. Given a prompt $(X_0, \lambda)$, it generates a sequence of intermediate states $X^M_1, X^M_2, \dots, X^M_T$ and a final output $Y^M$. Crucially, the paper treats the model's execution as an approximation of the ground-truth task application: $\Lambda^M(X_0, \lambda) \overset{d}{=} \Lambda(X_0, \hat{\lambda})$ for some $\hat{\lambda} \in \text{Span}(\Gamma^M)$ — the model can only execute tasks expressible as compositions of the primitive tasks $\Gamma^M$ it was trained on.

3. Supervisor Model ($g_{\text{sup}}$): A separately fine-tuned LLM that takes as input the model's CoT reasoning up to any step $t$ (denoted $X^M_t$) and directly predicts the ground-truth final answer $Y$. It is trained on prompts paired with correct final answers, using the base model's own CoT traces as input. The supervisor model approximates $p(Y \mid X^M_t)$, the conditional probability of the correct answer given the reasoning state so far.

4. Information-Gain Estimator: The computational module that uses the supervisor model's predictions to quantify the information-gain at each step $j$ — defined as the conditional mutual information $I(Y; X^M_j \mid X^M_{j-1})$. In practice, this is estimated as the difference in cross-entropy loss between consecutive steps: $\Delta\text{IG}_j = \mathbb{E}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_{j-1}))] - \mathbb{E}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_j))]$ where a positive value indicates step $j$ contributed information relevant to predicting $Y$, and a value near zero or negative indicates the step added no useful information.

Information flow: A prompt enters the base LLM → the LLM generates a CoT trace $(X^M_0, X^M_1, \dots, X^M_T, Y^M)$ → each intermediate state $X^M_t$ is fed to the supervisor model along with the prompt → the supervisor predicts the correct answer $Y$ → cross-entropy losses are computed at each step → the information-gain at step $j$ is the loss reduction from step $j-1$ to step $j$ → sub-tasks with consistently near-zero or negative gain are flagged as unidentifiable.

3.3 Roadmap for the deep dive

• First, the formal framework of tasks, primitive tasks, and unidentifiability — because the entire information-theoretic method is built on the concept of detecting when a model encounters a sub-task outside its training span, so we must first understand what that means formally.

• Second, the Bayesian network and the key theorem — because Theorem 3.3 is the theoretical backbone that predicts which conditional independence must hold if a sub-task is unidentifiable, and this prediction is what the information-gain measurement tests.

• Third, the supervisor model training procedure — because the supervisor is the instrument that makes the theoretical conditional independence empirically testable; understanding its training data construction, architecture choices, and what distribution it approximates is essential for interpreting results.

• Fourth, the information-gain computation and estimation — because this is the practical algorithm that operationalizes the theory; we need to understand how mutual information is estimated from finite data using cross-entropy, and why this specific estimator has the right properties.

• Fifth, the sample-wise extension — because aggregate information-gain gives sub-task-level diagnostics, but the natural extension to per-instance error detection requires a different estimator, and the paper provides both.

• Sixth, non-linear reasoning structures — because the paper claims (and empirically validates) that the framework extends beyond linear chains to branching, backtracking, and self-correcting reasoning patterns, and understanding this extension clarifies the scope of applicability.

3.4 Detailed, sentence-based technical breakdown

This is primarily a theoretical and methodological paper that builds a formal framework for Chain-of-Thought evaluation and then validates it empirically. Its core idea is that incorrect reasoning steps can be detected by measuring whether they contribute information toward predicting the correct final answer — and that this measurement can be performed without any annotated intermediate-step data, using only a fine-tuned supervisor model trained on prompt-final-answer pairs.

---

Formal Framework: Tasks, Primitive Tasks, and States

The paper adopts and extends the formal language of González and Nori (2023), which treats LLMs as execution machines with natural language interfaces. The central abstraction is an update rule $\Lambda: \mathcal{X} \times \Upsilon \rightarrow \mathcal{X}$ that maps an initial state and a task to an updated state.

The initial state $X_0 \in \mathcal{X}$ is a random variable representing the information provided in the prompt — for example, in a math problem like "Solve for $z = 2(u + v), u = 12, v = 13$", the initial state $x_0$ captures the information "$u = 12, v = 13$".

The task $\lambda \in \Upsilon$ defines how the LLM processes $X_0$ — in the example above, $\lambda$ is the operation of finding $z$, which involves addition followed by multiplication. The updated state $X_1 = \Lambda(X_0, \lambda)$ represents the result of correctly applying task $\lambda$ to $X_0$, so $x_1$ would be "$u = 12, v = 13$ and $z = 50$".

The final output $Y \in \mathcal{X}$ is obtained by extracting only the queried information from the final state — in the example, extracting "z = 50" from $x_1$.

A crucial nuance the paper handles upfront: $\Lambda(x_0, \lambda)$ is treated as a random variable in general, because some tasks may have multiple correct responses. The notation $\overset{d}{=}$ denotes equality in distribution to accommodate this ambiguity.

---

Compositionality and the Decomposition of Tasks into Steps

Many mathematical or logical problems require sequential application of operations, and the paper's framework is designed to handle this through compositional consistency.

Definition 2.1 (Compositionally consistent update rule): An update rule $\Lambda$ is compositionally consistent if, for all $x_0 \in \mathcal{X}$ and $\lambda_1, \lambda_2 \in \Upsilon$:

$\Lambda(x_0, \lambda_1 \circ \lambda_2) \overset{d}{=} \Lambda(\Lambda(x_0, \lambda_1), \lambda_2)$

where $\lambda_1 \circ \lambda_2$ denotes the compositional task "first apply $\lambda_1$, then apply $\lambda_2$."

What this means: If the update rule is compositionally consistent, then applying the composed task $\lambda_1 \circ \lambda_2$ yields the same distribution over states as first applying $\lambda_1$ to get an intermediate state, then applying $\lambda_2$ to that intermediate state. This is the formal property that allows us to break a complex task into a sequence of steps and analyze each step independently.

Why this matters for CoT: If the ground-truth reasoning process follows a compositionally consistent update rule, then a Chain-of-Thought that decomposes $\lambda$ into sub-tasks $\lambda_1 \circ \dots \circ \lambda_T$ generates intermediate states $X_1, X_2, \dots, X_T$ where each $X_t = \Lambda(X_{t-1}, \lambda_t)$. The CoT is simply making this sequential structure explicit — each step corresponds to applying one sub-task to the state accumulated so far.

For the running example of solving $z = 2(u+v)$: $\lambda_1$ is addition ($u + v$), $\lambda_2$ is multiplication (multiply the sum by 2), and the CoT would be $X_1 = \Lambda(X_0, \lambda_1)$ (computing $u+v = 25$) followed by $X_2 = \Lambda(X_1, \lambda_2)$ (computing $2 \times 25 = 50$).

---

Primitive Tasks and Span: The Building Blocks of Model Capability

The paper introduces the concept of primitive tasks to formally characterize what a model could have learned. The analogy is to basis vectors in linear algebra: any vector can be expressed as a linear combination of basis vectors, and similarly, any task can be expressed as a composition of primitive tasks.

Definition 2.2 (Primitive tasks): A set of tasks $\Gamma \subseteq \Upsilon$ is primitive if, for any task $\lambda \in \Upsilon$, there exists a unique subset $\{\lambda_i\}_{i=1}^k \subseteq \Gamma$ such that $\lambda = \lambda_1 \circ \dots \circ \lambda_k$. The decomposition into the sequence may not be unique, but the set of components is.

Operational interpretation: In the context of mathematical problem-solving, the basic arithmetic operations (addition, multiplication, subtraction, division) could be considered primitive tasks. More complex operations — like "compute the least common multiple" or "solve the quadratic equation" — are compositions of these primitives. The concept generalizes beyond arithmetic: in Blocks World planning, primitives might be "stack," "unstack," and "move"; in commonsense QA, they might be "causal reasoning," "temporal reasoning," and "spatial reasoning."

Definition 2.3 (Span of tasks): The span of a set of tasks $\Phi \subseteq \Upsilon$ is:

$\text{Span}(\Phi) = \{\lambda_1 \circ \dots \circ \lambda_k : \lambda_i \in \Phi \text{ for } 1 \leq i \leq k, k \in \mathbb{Z}_{>0}\}$

What the span captures: The span represents all tasks that can be executed by composing the primitive tasks in $\Phi$ — it is the closure of $\Phi$ under sequential composition. If we know which primitive tasks an LLM was exposed to during training, the span tells us the full set of tasks the model could have learned to execute correctly.

---

Unidentifiability: When a Task Lies Outside the Model's Reach

The paper's central diagnostic concept is unidentifiability:

Definition 2.4 (Unidentifiability): A task $\lambda$ is unidentifiable in a set $\Phi \subseteq \Upsilon$ if and only if $\lambda \notin \text{Span}(\Phi)$.

What this means in plain language: If a task cannot be expressed as any composition of the primitive tasks that were available in the model's training data, then no compositionally consistent update rule can execute that task correctly — the model has no "building blocks" that can be combined to produce the required behavior. This is the formalization of a capability gap.

A critical nuance the paper acknowledges: Unidentifiability may depend on the initial state distribution. An LLM might accurately perform addition for 2-digit numbers but fail with 10-digit numbers, meaning addition is conditionally unidentifiable depending on the state $X_0$. The paper treats this distributional dependence as implicit for simplicity, but notes that the framework can be extended to make it explicit by conditioning the definition of identifiability on the distribution $p(X_0)$.

The practical implication: If we can detect which sub-tasks in a model's CoT are unidentifiable, we know precisely which capabilities the model lacks and where targeted fine-tuning would be most impactful. This transforms the problem from "the model gets 60% accuracy on MATH" to "the model cannot reliably execute multiplication when operands exceed three digits" — an actionable diagnostic.

---

Assumption 3.1: The Bayesian Network for Divergent Reasoning Paths

The paper's key theoretical assumption specifies the conditional independence structure that arises when a reasoning path encounters an unidentifiable sub-task and diverges from a correct path.

Assumption 3.1 (Bayesian network): Let $\lambda \neq \lambda'$ be two operations with primitive decompositions:

$\lambda = \lambda_1 \circ \dots \circ \lambda_{k-1} \circ \lambda_k \circ \dots \circ \lambda_T$

$\lambda' = \lambda_1 \circ \dots \circ \lambda_{k-1} \circ \lambda'_k \circ \dots \circ \lambda'_{T'}$

where $\lambda'_k$ is unidentifiable in $\{\lambda_1, \dots, \lambda_T\}$ (the set of primitives appearing in the correct path $\lambda$). Then the intermediate states corresponding to tasks $\lambda$ and $\lambda'$ have the Bayesian network structure shown in Figure 2 of the paper.

What the Bayesian network encodes: The two reasoning paths share the same sequence of sub-tasks up through step $k-1$, sharing intermediate states $X_0, X_1, \dots, X_{k-1}$. At step $k$, they diverge: the correct path applies $\lambda_k$ to produce $X_k$, while the incorrect path applies the unidentifiable $\lambda'_k$ to produce $X'_k$. From this point forward, the two paths are conditionally independent given $X_{k-1}$ — formally, $X_i \perp\!\!\!\perp X'_j \mid X_{k-1}$ for all $i, j \geq k$.

The intuition in operational terms: Once the model applies a sub-task that is genuinely unidentifiable (one that no composition of its learned primitives can execute correctly), the resulting state diverges from what any correct execution would produce. From that point forward, the erroneous path's subsequent states contain no information about what the correct path's states or final answer would have been — they are statistically independent given the last shared state $X_{k-1}$. This is not an empirical claim about any particular model; it is the structural property that defines what it means for a sub-task to be unidentifiable in the framework.

Why this assumption is reasonable: The paper argues that this conditional independence is the natural formalization of "the model went off the rails." If the model could produce states along the incorrect path that somehow still correlate with the correct answer, that would imply the unidentifiable sub-task was not actually unidentifiable — there would exist some composition of learned primitives that partially recovers the correct behavior, contradicting the definition.

---

Assumption 3.2: How LLMs Execute Tasks

The paper explicitly distinguishes between the ground-truth update rule $\Lambda$ (which represents correct task execution) and the model's learned update rule $\Lambda^M$ (which represents what the LLM actually does).

Assumption 3.2 (Task execution in LLMs): Let $\Gamma^M \subseteq \Gamma$ denote the primitive tasks available in the LLM's training data. Then $\Lambda^M$ is compositionally consistent, and for any $(x_0, \lambda) \in \mathcal{X} \times \Upsilon$, there exists some $\hat{\lambda} \in \text{Span}(\Gamma^M)$ such that:

$\Lambda^M(x_0, \lambda) \overset{d}{=} \Lambda(x_0, \hat{\lambda})$

What this means: When instructed to perform task $\lambda$, the LLM does not necessarily execute $\lambda$ — instead, it executes some task $\hat{\lambda}$ that lies within the span of primitive tasks it was trained on. The model cannot produce behaviors that require primitives outside $\Gamma^M$; its execution is restricted to $\text{Span}(\Gamma^M)$.

This is a weak assumption, not a strong one: The paper emphasizes that this assumption does not require the model to perfectly follow instructions or decompose tasks correctly. It only requires that whatever the model actually does must be expressible as some composition of the primitives it was exposed to. If the model hallucinates, makes arithmetic errors, or follows a completely wrong reasoning strategy, that behavior is still captured — $\hat{\lambda}$ would simply be the (possibly incorrect) composition of primitives that describes what the model actually computed.

The model's intermediate states under this assumption: Starting from initial state $X_0$, the model's CoT generates states $X^M_t$ recursively as $X^M_t = \Lambda^M(X^M_{t-1}, \lambda_t)$ with $X^M_0 = X_0$, where $\lambda_t$ is the $t$-th sub-task in the instructed decomposition of $\lambda$. The final model output is denoted $Y^M$.

---

Theorem 3.3: The Conditional Independence That Enables Detection

The paper's central theoretical result connects unidentifiability to an empirically testable prediction about information flow:

Theorem 3.3: Let $\Gamma^M \subseteq \Gamma$ denote the primitive tasks available in the training data. Let $\lambda$ be a task with decomposition $\lambda = \lambda_1 \circ \dots \circ \lambda_T$. If $\lambda_k$ is the first task in the decomposition of $\lambda$ which is unidentifiable in $\Gamma^M$ (i.e., $k = \arg\min_t\{\lambda_t \notin \text{Span}(\Gamma^M)\}$), then under Assumptions 3.1 and 3.2:

$Y \perp\!\!\!\perp X^M_j \mid X^M_{j-1} \quad \text{for all } j \geq k$

where $Y$ is the ground-truth correct final answer, and $X^M_j, X^M_{j-1}$ are the model's CoT states at steps $j$ and $j-1$ respectively.

What this theorem establishes: If the model encounters an unidentifiable sub-task at step $k$ of its instructed reasoning, then from that step onward, the model's CoT state at any step $j \geq k$ is conditionally independent of the correct answer $Y$ given the state at step $j-1$. In other words: once the model diverges from the ground truth at step $k$, no subsequent step adds any information that helps predict what the correct answer should have been.

Why step $k$ must be the first unidentifiable task: The theorem requires $\lambda_k$ to be the earliest task in the decomposition that lies outside $\text{Span}(\Gamma^M)$. This ensures that all earlier tasks $\lambda_1, \dots, \lambda_{k-1}$ are identifiable — the model can in principle execute them correctly. The conditional independence applies from step $k$ onward because that is where the model's execution first necessarily diverges from any correct path.

The proof structure (Appendix A): The proof proceeds in two steps. First, Assumption 3.1 guarantees that if two reasoning paths diverge at a point where one applies an unidentifiable operation, the subsequent states along those paths are d-separated (conditionally independent) given the last shared state. Second, Assumption 3.2 guarantees that the model's actual execution $\Lambda^M(X_0, \lambda)$ is equivalent to applying some $\hat{\lambda} \in \text{Span}(\Gamma^M)$, and because $\lambda_k \notin \text{Span}(\Gamma^M)$, the primitive decomposition of $\hat{\lambda}$ must diverge from that of $\lambda$ at or before step $k$. Combining these yields the conditional independence statement.

The operational consequence for evaluation: Theorem 3.3 tells us that if we can measure whether subsequent steps provide information about $Y$, we can detect which steps are unidentifiable. If step $j$ is after the first unidentifiable sub-task, the mutual information $I(Y; X^M_j \mid X^M_{j-1})$ should be zero — the step provides no additional information beyond what was already in the previous state. If step $j$ is before the first unidentifiable sub-task, this mutual information should be positive — the step genuinely advances the model toward the correct answer.

---

Information-Gain: Operationalizing the Conditional Independence Test

The paper defines information-gain at step $j$ as the conditional mutual information between the correct final answer $Y$ and the model's state at step $j$, given the state at step $j-1$:

$\text{IG}_j = I(Y; X^M_j \mid X^M_{j-1})$

where $I(A; B \mid C)$ is the standard conditional mutual information from information theory, measuring how much the uncertainty about $A$ is reduced by observing $B$ when $C$ is already known.

What this quantity measures: $\text{IG}_j$ quantifies how much new information step $j$ contributes toward predicting the correct final answer $Y$ — information that was not already present in the state at step $j-1$. If step $j$ applies a sub-task correctly, it should refine the partial solution, making the final answer more predictable, so $\text{IG}_j > 0$. If step $j$ is executed incorrectly (because the sub-task is unidentifiable), it should add noise or irrelevant information, so $\text{IG}_j \approx 0$ (or potentially negative if it actively misleads).

Connection to Theorem 3.3: The theorem predicts $\text{IG}_j = 0$ for all $j \geq k$ where $k$ is the index of the first unidentifiable sub-task. For steps before $k$, $\text{IG}_j > 0$ is expected because those steps genuinely help solve the problem.

---

Proposition 3.4: From Mutual Information to Cross-Entropy

Conditional mutual information is not directly computable from data — it requires knowing the true probability distributions. The paper provides Proposition 3.4, which connects the information-gain to an expectation over log-likelihood ratios that can be estimated using a learned model.

Proposition 3.4:

$\mathbb{E}[\log p(Y \mid X^M_j)] - \mathbb{E}[\log p(Y \mid X^M_{j-1})] = I(Y; X^M_j \mid X^M_{j-1}) \geq 0$

where $p(Y \mid X^M_j)$ is the true conditional probability of the correct answer given the model's CoT state at step $j$, and the expectations are taken over the joint distribution of $(Y, X^M_j, X^M_{j-1})$.

Step-by-step derivation (from the proof in Appendix A): The equality follows from expanding the log-ratio:

$\mathbb{E}\left[\log \frac{p(Y \mid X^M_j)}{p(Y \mid X^M_{j-1})}\right] = \mathbb{E}\left[\log \frac{p(Y \mid X^M_j, X^M_{j-1})}{p(Y \mid X^M_{j-1})}\right]$

The first equality uses the fact that $X^M_{j-1}$ is deterministic given $X^M_j$ (the state at step $j$ subsumes all information in step $j-1$, since it is generated by appending the latest reasoning step). Therefore, $Y \perp\!\!\!\perp X^M_{j-1} \mid X^M_j$, allowing us to write $p(Y \mid X^M_j) = p(Y \mid X^M_j, X^M_{j-1})$.

The expression then expands to:

$\mathbb{E}\left[\log \frac{p(Y, X^M_j \mid X^M_{j-1})}{p(Y \mid X^M_{j-1}) p(X^M_j \mid X^M_{j-1})}\right] = I(Y; X^M_j \mid X^M_{j-1})$

which is exactly the definition of conditional mutual information. The inequality $\geq 0$ follows from the non-negativity of mutual information (conditioning cannot decrease expected information).

Why this form is practical: The expectation of log-probability $\mathbb{E}[\log p(Y \mid X^M_j)]$ is exactly the negative cross-entropy (up to an additive constant) between the true conditional distribution and any model's predictive distribution. If we can train a model $\hat{p}(Y \mid X^M_j)$ to approximate the true conditional, then:

$\mathbb{E}[\log p(Y \mid X^M_j)] \approx -\mathbb{E}[\ell_{\text{CE}}(Y, \hat{p}(Y \mid X^M_j))]$

and the information-gain can be estimated as:

$\widehat{\text{IG}}_j = \mathbb{E}[\ell_{\text{CE}}(Y, \hat{p}(Y \mid X^M_{j-1}))] - \mathbb{E}[\ell_{\text{CE}}(Y, \hat{p}(Y \mid X^M_j))]$

This is the difference in cross-entropy loss between the supervisor model's predictions at steps $j-1$ and $j$ — a quantity computable entirely from data without any intermediate-step labels.

Why cross-entropy and not accuracy or some other metric: The log-likelihood is the proper scoring rule for probabilistic predictions — it is sensitive to the model's confidence calibration, not just its mode prediction. A step that makes the correct answer more likely but doesn't yet pin it down uniquely will still show a reduction in cross-entropy, whereas accuracy-based metrics would show no change until the answer can be perfectly predicted. This graduated sensitivity is essential because reasoning is incremental: early correct steps may only narrow the solution space without fully determining the answer.

---

The Supervisor Model: Approximating $p(Y \mid X^M_t)$

The paper trains a supervisor model $g_{\text{sup}}$ to approximate the conditional distribution $p(Y \mid X^M_t)$ for each intermediate step $t$. This is the computational instrument that makes the information-theoretic framework operational.

What the supervisor model takes as input: The model's CoT reasoning up to any intermediate step $t$, denoted $X^M_t$, which includes the original prompt $X_0$ and all CoT steps through step $t$. The supervisor also receives a special separator token that signals it should directly predict the final answer $Y$.

What the supervisor model outputs: A probability distribution over possible final answers $Y$, from which we can compute the log-probability (equivalently, the negative cross-entropy) of the ground-truth correct answer.

Training data construction: The paper constructs training data for the supervisor model by taking the base model's full CoT traces (for prompts where the ground-truth final answer $Y$ is known) and creating multiple training examples per trace by truncating at each intermediate step. For a single CoT trace with $T$ steps, this yields $T+1$ training instances:

• Instance 0: [Prompt] #|> [Correct answer Y]
• Instance 1: [Prompt] || [Step 1] #|> [Correct answer Y]
• Instance 2: [Prompt] || [Step 1] || [Step 2] #|> [Correct answer Y]
• ...
• Instance T: [Prompt] || [Step 1] || ... || [Step T] #|> [Correct answer Y]

where || denotes CoT step boundaries and #|> is a special separator token that signals to the supervisor model that it should now predict the final answer $Y$ directly (without continuing the CoT).

Why train on both correct and incorrect traces: The supervisor model is trained on all generated CoT traces, regardless of whether the base model's final answer $Y^M$ was correct. This is crucial because the supervisor needs to learn the true relationship $p(Y \mid X^M_t)$ — the probability of the ground-truth correct answer given the reasoning state — not the probability that the model's own answer matches the ground truth. If we only trained on correct traces, the supervisor would not learn to recognize when intermediate steps are leading toward wrong answers.

Supervisor model architecture choices in experiments: The paper uses different supervisor model architectures depending on the experimental setting:

• Toy data experiments (Section 5.1): A GPT-2 model fine-tuned separately for each of the five LLMs under evaluation. Each supervisor is trained on data generated by its corresponding base LLM's CoT traces.

• Arithmetic operations on Llama-3-8B (Section 5.2): A Llama-3-8B model fine-tuned using Low Rank Adaptation (LoRA) (Hu et al., 2021). The paper does not specify LoRA rank or alpha values, but notes that LoRA was used for parameter efficiency. The supervisor is trained on the base Llama-3-8B model's own CoT traces on the arithmetic task.

• Controlled GSM8K (Section 5.3): A GPT-2 model fine-tuned with supervised fine-tuning (SFT) until convergence, with early stopping based on held-out validation loss. The training data consists of GPT-4-generated CoT traces (with deliberately incorrect multiplications) paired with ground-truth answers.

• PRM800K (Section 5.4): A GPT-2 model fine-tuned with SFT, again with early stopping. The training data uses CoT traces from the PRM800k dataset (originally generated by a fine-tuned GPT-4) paired with the correct MATH answers.

Design choice — why use the base model's own CoT traces for training: The paper emphasizes that the supervisor must be trained on the same distribution of CoT traces that it will evaluate. If the supervisor were trained on human-written correct CoTs, it would learn a different conditional distribution $p(Y \mid \text{human CoT})$ that could be poorly calibrated for model-generated CoTs. By training on the model's own outputs, the supervisor learns the actual relationship between the model's reasoning states and the correct answer, including handling the model's idiosyncratic errors and formatting.

Design choice — why a separate model rather than prompting the base model: The paper could have attempted to estimate $p(Y \mid X^M_t)$ by prompting the base model itself with "Given this partial solution, what is the answer?" This would be circular: the base model's own errors would contaminate the information-gain estimates. A separately fine-tuned supervisor model breaks this circularity by learning a calibrated predictor that may recognize when the base model's reasoning has gone astray.

---

Estimating Information-Gain in Practice

With the supervisor model trained, the information-gain at step $j$ is estimated as follows.

Step 1 — Forward pass through supervisor: For each CoT trace in the evaluation set, feed the truncated state $X^M_j$ (with separator token #|>) to the supervisor model and obtain the model's log-probability for the ground-truth answer $Y$.

Step 2 — Compute cross-entropy: The cross-entropy loss for step $j$ is:

$\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_j)) = -\frac{1}{|Y|}\sum_{i=1}^{|Y|} \log g_{\text{sup}}(y_i \mid X^M_j, \text{prefix})$

where $|Y|$ is the number of tokens in the ground-truth answer $Y$, $y_i$ is the $i$-th token of $Y$, and the conditioning includes the input $X^M_j$, the separator token, and any previously generated tokens of $Y$. The paper normalizes by token count to make the loss comparable across answers of different lengths.

Step 3 — Aggregate across the dataset: For a dataset of $N$ evaluation prompts, the expected cross-entropy at step $j$ is estimated as the empirical average:

$\widehat{\mathbb{E}}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_j))] = \frac{1}{N}\sum_{i=1}^N \ell_{\text{CE}}(y^{(i)}, g_{\text{sup}}(x^{M,(i)}_j))$

Step 4 — Compute information-gain as loss difference: The estimated information-gain at step $j$ is:

$\widehat{\text{IG}}_j = \frac{1}{N}\sum_{i=1}^N \left[\ell_{\text{CE}}(y^{(i)}, g_{\text{sup}}(x^{M,(i)}_{j-1})) - \ell_{\text{CE}}(y^{(i)}, g_{\text{sup}}(x^{M,(i)}_j))\right]$

What the sign means: $\widehat{\text{IG}}_j > 0$ means the cross-entropy decreased from step $j-1$ to step $j$, indicating step $j$ made the correct answer more predictable — it contributed useful information. $\widehat{\text{IG}}_j \approx 0$ means the cross-entropy is unchanged, indicating step $j$ added no relevant information. $\widehat{\text{IG}}_j < 0$ means the cross-entropy increased, indicating step $j$ actively made the correct answer less predictable — the reasoning went in the wrong direction.

Sub-task-level aggregation: To evaluate a specific sub-task type (e.g., "addition" or "multiplication"), the paper aggregates information-gain only over prompts and steps where that sub-task appears. For example, to estimate the information-gain for addition, the paper:

1. Identifies all prompts containing addition as a sub-task.
2. For each such prompt, identifies the step $T_+$ where addition is executed.
3. Computes $\Delta\text{IG}_{\text{addition}} = \mathbb{E}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_{T_+-1}))] - \mathbb{E}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_{T_+}))]$ averaged over these prompts.

This per-sub-task aggregation is what enables the granular diagnostic: it tells you which types of operations the model handles well and which it fails at, without ever labeling individual steps.

Why this estimation procedure works despite finite data: The cross-entropy loss difference is an unbiased estimator of the conditional mutual information only in the limit of infinite data and a perfectly calibrated supervisor model. In practice, the supervisor model is approximate, and the data is finite. The paper implicitly relies on the consistency of maximum likelihood estimation — as the supervisor model's capacity and training data increase, $\hat{p}(Y \mid X^M_j)$ approaches $p(Y \mid X^M_j)$ and the estimated information-gain approaches the true value. The experimental validation against known ground-truth error locations (in the toy data and PRM800k experiments) provides empirical evidence that the approximation is sufficiently accurate to correctly localize errors.

---

Sample-Wise Information-Gain: Per-Instance Error Detection

The aggregate information-gain provides sub-task-level diagnostics, but the paper also develops a sample-wise variant for detecting which specific step is wrong in an individual CoT instance.

The sample-wise estimator: For a single CoT instance with ground-truth answer $y$ and model states $x^M_{j-1}, x^M_j$, the sample-wise information-gain at step $j$ is:

$\Delta\text{IG}_j^{\text{sample}} = \ell_{\text{CE}}(y, g_{\text{sup}}(x^M_{j-1})) - \ell_{\text{CE}}(y, g_{\text{sup}}(x^M_j))$

which is simply the per-instance cross-entropy difference, without averaging over the dataset.

How it is used for error detection: A step $j$ is classified as incorrect if its sample-wise information-gain falls below a threshold $\tau$. The threshold is chosen on a held-out validation set to optimize a detection metric (accuracy, FPR/TPR trade-off). The intuition: if step $j$ is correct, the supervisor should become more confident about the correct answer, so cross-entropy decreases ($\Delta\text{IG}_j^{\text{sample}} > \tau$). If step $j$ is incorrect, the supervisor's confidence should not improve, so cross-entropy does not decrease ($\Delta\text{IG}_j^{\text{sample}} \leq \tau$).

Why sample-wise is inherently noisier than aggregate: The sample-wise estimator does not benefit from averaging over many instances, so it is more sensitive to supervisor model miscalibration and randomness in individual CoT traces. The paper acknowledges this by presenting both aggregate and sample-wise results, and by selecting thresholds on held-out data rather than assuming a fixed zero threshold.

Connection to $\mathcal{V}$-usable information: The paper cites Ethayarajh et al. (2022) in the context of sample-wise information, noting that the per-instance log-probability difference is analogous to the pointwise $\mathcal{V}$-information used to measure dataset difficulty. The conceptual connection: just as $\mathcal{V}$-information measures how much a model family can reduce uncertainty about labels given inputs, the sample-wise information-gain measures how much a specific reasoning step reduces the supervisor model's uncertainty about the correct answer.

---

Extension to Non-Linear Reasoning Structures (O1/R1-Style Models)

The paper explicitly addresses how the framework applies beyond simple linear CoT chains to the more complex reasoning patterns found in modern reasoning-focused models.

The challenge: Models like O1 and R1 often explore multiple solution paths, backtrack when encountering errors, and dynamically self-correct. A reasoning trace might look like: $X_0 \rightarrow \cdots \rightarrow X_t \rightarrow Z \rightarrow X_t \rightarrow X_{t+1} \rightarrow \cdots \rightarrow X_T \rightarrow Y$, where $Z$ is an exploratory incorrect step that the model subsequently recognizes as wrong and backtracks from.

How the framework handles this: The information-gain at step $Z$ (the exploratory step) would be evaluated as $I(Y; Z \mid X_t)$. Because the final answer $Y$ is conditionally independent of the exploratory step $Z$ given the last correct state $X_t$ (the model returned to the correct path after exploring), we have $Y \perp\!\!\!\perp Z \mid X_t$, and therefore the information-gain at step $Z$ will be zero or negative. This correctly identifies the exploratory step as not contributing to the solution, even though the model eventually reaches the correct answer.

The key distinction: Exploratory or self-corrected steps show low information-gain but do not prevent the model from eventually finding the correct answer — they are part of the search process. In contrast, unidentifiable steps (those for which Assumption 3.1 directly applies) represent fundamental capability gaps where the model cannot execute the required operation even in principle. Both will exhibit zero or negative information-gain, but the distinction can be made by examining whether later steps regain positive information-gain (indicating self-correction) or remain at zero (indicating permanent divergence).

Empirical validation: The paper validates this claim on the PRM800K dataset (Section 5.4), where steps labeled as "neutral" by human annotators (steps that are technically valid or ambiguous but do not advance the solution) consistently show low or negative information-gain. This demonstrates that the information-gain metric correctly identifies unproductive reasoning steps even when they are not strictly "incorrect."

---

Summary of Design Choices and Their Justifications

• Information-theoretic over behavioral evaluation: Rather than trying to classify steps as correct/incorrect (which would require ground-truth step labels), the framework asks the weaker question "does this step help predict the correct answer?" This allows training without step-level annotation because the supervision signal comes from final answers only.

• Supervisor model over self-evaluation: A separately trained model estimates $p(Y \mid X^M_t)$ rather than prompting the base model to assess its own reasoning. This avoids circularity — the base model's errors would contaminate the confidence estimates if it evaluated itself.

• Cross-entropy over accuracy: The information-gain uses log-probability rather than binary correctness to quantify predictive information. Log-probability is sensitive to confidence calibration: a step that narrows the answer from 100 possibilities to 2 provides substantial information (large cross-entropy reduction) even if the supervisor still cannot pick the exact correct answer with high confidence.

• Training supervisor on all traces, not just correct ones: The supervisor must learn to recognize when reasoning is going wrong, which requires seeing incorrect intermediate states paired with correct final answers. Training only on correct traces would leave the supervisor unable to assess erroneous reasoning.

• Per-sub-task aggregation for diagnostics: By computing information-gain separately for each sub-task type (addition, multiplication, etc.), the framework provides granular capability profiles — telling users which operations the model fails at, not just that it fails sometimes.

• Sample-wise thresholding on held-out data: The paper selects detection thresholds using a held-out validation set rather than assuming information-gain $\leq 0$ always indicates an error. This accommodates the fact that the supervisor model is approximate and that even correct steps may occasionally show near-zero or negative estimated gain due to estimation noise.

4. Key Insights and Innovations

Innovation 1: Reframing CoT Evaluation from "Is This Step Correct?" to "Does This Step Provide Information?"

The paper's most fundamental conceptual shift is changing the question we ask about intermediate reasoning steps. Prior work — whether through expensive human annotation (PRMs; Lightman et al., 2023) or learned heuristics (ORMs; Cobbe et al., 2021; Havrilla et al., 2024; Math-Shepherd; Wang et al., 2024) — all ask a version of: "Is this step correct?" PRMs answer it by training on human labels of correctness. ORMs answer it by checking whether final answers after this step tend to be correct. Math-Shepherd answers it by checking whether completions from this step reach the right answer.

This paper asks a fundamentally different question: "Does this step contribute information that helps predict the correct final answer?" The distinction is not merely semantic — it changes the nature of the supervision signal, the failure modes of the evaluation, and the theoretical guarantees.

Why this matters: Correctness is a binary property that requires ground-truth knowledge of what the step should produce. Information-gain is a relational property — it measures whether the step moves the prediction of $Y$ in a useful direction, which can be assessed purely from the statistical relationship between intermediate states and final answers. This reframing is what enables the paper to escape the annotation bottleneck: you don't need to know whether each step is correct; you only need to know whether each step helps predict the answer you already have labels for.

Distinguishing from prior work: ORMs and MS also avoid step-level labels, but they retain the "correctness" framing — they attempt to infer correctness from outcomes, which makes them vulnerable to spurious correlations (the paper's LLM3 experiment, Figure 3b-c, demonstrates this concretely). The information-gain framing sidesteps this vulnerability because it doesn't try to answer "is step $j$ correct?" — it answers "does step $j$ provide useful information?" A step that is conditionally independent of $Y$ given the prior state cannot be providing useful information, regardless of whether earlier states happen to predict the final outcome. This is a theoretical property (it follows from the definition of conditional mutual information), not an empirical heuristic.

Significance: This reframing is a fundamental conceptual shift, not an incremental improvement. It converts CoT evaluation from a classification problem (correct/incorrect) to an information measurement problem (how much does this step reduce uncertainty about $Y$?). This is a different epistemological stance toward what it means to evaluate reasoning — one that is both more principled (grounded in information theory) and more practical (doesn't require annotated intermediate data). It's the intellectual move that makes the entire framework possible.

---

Innovation 2: Unidentifiability as a Formal Diagnostic Construct for Model Capability Gaps

The paper introduces unidentifiability — the formal notion that a sub-task lies outside the span of primitive tasks the model was exposed to during training — as a diagnostic construct for explaining why reasoning fails. This is not a new algorithm or metric; it's a new conceptual vocabulary for reasoning about model limitations.

What the field had before: Prior work on model evaluation operates in terms of aggregate accuracy ("the model scores 60% on MATH") or coarse error categorization ("the model makes arithmetic mistakes"). Process supervision (Lightman et al., 2023) offers step-level correctness, but doesn't explain why a step is incorrect — it just labels it. There was no formal language for distinguishing between "the model made a random error on a task it generally can do" and "the model fundamentally cannot do this type of operation because it was never trained on it."

What unidentifiability provides: The paper gives us a precise criterion for the latter case: a sub-task $\lambda_k$ is unidentifiable in $\Gamma^M$ (the model's training primitives) if $\lambda_k \notin \text{Span}(\Gamma^M)$ — it cannot be expressed as any composition of the primitive tasks the model learned. This is not the same as "the model sometimes gets it wrong." It means no compositionally consistent execution of the model's available operations can produce the correct behavior, even in principle.

Why this is a conceptual advance: Unidentifiability provides a generative explanation for failure, not just a description. If we know that multiplication of large numbers is unidentifiable for a given model, we don't just know that it fails — we know that no amount of prompt engineering, few-shot examples, or CoT decomposition will fix it, because the capability is structurally absent from the model's learned primitives. The fix is targeted fine-tuning on that specific operation. This transforms evaluation from "here's what the model gets wrong" (descriptive) to "here's what the model cannot do, and here's what you need to train" (prescriptive).

Evidence: The paper doesn't directly prove unidentifiability from training data — it infers it from information-gain patterns. But the framework is validated by the controlled experiments: in the toy setting (Section 5.1) where unidentifiability is engineered (by corrupting specific sub-tasks), the information-gain correctly localizes the unidentifiable operations across all five LLMs (Figure 3a). In the PRM800k experiment (Section 5.4), steps labeled "negative" by human annotators (genuine errors) show near-zero or negative information-gain, consistent with the unidentifiability prediction.

Limitation acknowledged: The paper explicitly notes (Section 6) that unidentifiability may be state-conditional — a model might handle 2-digit addition but not 10-digit addition — which complicates the binary "identifiable/unidentifiable" framing. The current framework treats this distributional dependence as implicit, but the extension is conceptually straightforward.

Significance: This is a fundamental conceptual contribution. The paper gives the field a formal vocabulary for distinguishing between capability gaps (unidentifiability) and execution failures (random errors on identifiable tasks). This distinction has direct practical implications for how we allocate improvement effort — fine-tuning versus better prompting — and it enables more precise scientific communication about model limitations.

---

Innovation 3: The Empirical Discovery That Outcome-Based Baselines Systematically Fail When Error Determinants Precede the Error

The paper makes an important negative empirical discovery that has implications beyond any particular method: ORMs and Math-Shepherd are not just noisy or approximate — they are structurally unsound for step-level error detection in a specific, practically common regime. This is not a claim that the paper's method is better; it's a demonstration that the prior methods can give wrong answers with high confidence in identifiable circumstances.

The structural failure mode: When the occurrence of an error at step $k$ is predictable from information available at step $k-1$ (or earlier), outcome-based methods systematically misattribute the error to the earlier step. The toy LLM3 experiment (Section 5.1, Figure 3b) makes this concrete: task $\lambda_3$ is executed incorrectly if and only if the output after $\lambda_2$ satisfies $z_2[2] > 150$. ORM learns this deterministic relationship from training data — it sees $z_2[2] > 150$ and predicts the final answer will be wrong, causing its confidence to drop at step 2, not step 3. The method flags $\lambda_2$ as the erroneous step with high confidence, when in fact $\lambda_2$ was executed perfectly.

Why this matters beyond toy data: This failure mode generalizes to any setting where error probability depends on features of the problem state rather than being uniformly random. If a model systematically fails at multiplication when operands exceed three digits, and the operand size is determined before the multiplication step (e.g., by a previous calculation), ORM will learn to predict failure from the operand size visible early in the CoT — potentially before the multiplication even occurs. The GSM8K experiment (Section 5.3) demonstrates this with real data: because the paper's data construction ensures that all incorrect final answers involve both "multiplication" and "subtraction," ORM learns to flag both operations as suspicious (Table 2), producing a 100% false positive rate on subtraction despite subtraction being executed correctly.

The Math-Shepherd variant of this failure: In the LLM3 experiment (Figure 3c), Math-Shepherd's proportion of correct completions is zero for all steps, providing no diagnostic signal about which step is actually wrong. This occurs because the model (used as its own completer) cannot produce correct completions when the error determinant ($z_2[2] > 150$) is already present — the error is deterministic given the earlier state, so completions from any step will fail. The method correctly identifies that something is wrong but cannot localize it.

What this tells us about the prior literature: This finding explains why outcome-based methods might appear to work in some studies but fail in others — their reliability depends on the statistical structure of errors in the specific dataset and model under evaluation. If errors are uniformly random across instances, ORM and MS will roughly work. If errors are state-dependent (likely in any real model, where difficulty varies with input properties), they can produce confident but wrong localizations.

Evidence from sample-wise detection (Table 1): On LLM3, ORM achieves 98% true positive rate but 54% false positive rate — it catches most errors but also incorrectly flags more than half of correct steps. MS achieves 100% true positive rate but 100% false positive rate — it flags everything as incorrect. The information-gain method achieves 98% TPR with only 6% FPR. These are not small differences; they represent qualitative differences in the methods' logical foundations.

Significance: This is a fundamental empirical finding with architectural implications. It demonstrates that outcome-based approaches to step-level evaluation have an inherent logical limitation — not just a statistical one that could be fixed with more data or a better classifier. The limitation stems from the fact that $P(\text{final answer correct} \mid X^M_t)$ can change for reasons unrelated to the correctness of step $t$ itself. This insight should redirect the field away from trying to patch ORM-type methods and toward information-theoretic or causal approaches that don't conflate predictability with correctness.

---

Innovation 4: A Unified Theoretical Framework Connecting CoT Reasoning to Information Theory via the Supervisor Model

The paper provides a complete theoretical pipeline — from formal definitions of tasks and unidentifiability (Definitions 2.1–2.4) through a structural causal assumption (Assumption 3.1) to a testable conditional independence prediction (Theorem 3.3) and a practical estimation procedure (Proposition 3.4) — that connects abstract model capability analysis to concrete, computable measurements. This is a rare case where a paper builds a self-contained formal system and then validates it empirically.

What prior formal work provided: González and Nori (2023) introduced the "LLM as execution machine" framework with states, tasks, and update rules, but stopped at the conceptual level — it provided vocabulary but no empirical measurement procedure. Feng et al. (2023) theoretically analyzed what CoT can express, but focused on transformer architecture constraints rather than practical evaluation. Neither provided a bridge from formalism to measurement.

What this paper adds: The paper takes the González-Nori framework and extends it with three crucial elements that enable empirical operation: (1) primitive tasks and span (Definitions 2.2–2.3), which formalize what a model could have learned; (2) the Bayesian network assumption (Assumption 3.1), which specifies the conditional independence structure when reasoning diverges; and (3) the supervisor model as a measurement instrument (Section 3.3, Appendix C.1.3), which estimates the probabilities needed to compute information-gain without ground-truth intermediate states.

The crucial conceptual bridge is Theorem 3.3, which proves that unidentifiability implies a specific conditional independence ($Y \perp\!\!\!\perp X^M_j \mid X^M_{j-1}$ for $j \geq k$), and Proposition 3.4, which shows this conditional independence can be tested via cross-entropy reduction without ever knowing which steps are correct. This means the theoretical construct (unidentifiability) has an empirical signature (zero information-gain) that can be measured with only final-answer labels.

A self-contained diagnostic system, not a collection of techniques: The paper doesn't just propose a metric — it provides the theoretical justification for why the metric should work, the assumptions under which it is guaranteed to work, and the empirical validation showing it does work. This is a complete diagnostic framework, not a heuristic. It tells you: "If Assumptions 3.1 and 3.2 hold, then zero information-gain at step $j$ implies step $j$ is after the first unidentifiable sub-task." The empirical experiments validate that the assumptions are reasonable in practice (the method correctly localizes errors) but do not validate the assumptions directly — this is a limitation the paper does not fully address.

Practical consequence: The supervisor model architecture abstracts away the question of how to estimate $p(Y \mid X^M_t)$. The paper uses fine-tuned LLMs (GPT-2, Llama-3-8B with LoRA), but in principle, any calibrated conditional probability estimator would work. This means the framework is model-agnostic at the supervisor level — as better language models become available, they can be plugged in as supervisors, potentially improving the accuracy of information-gain estimates without changing the theoretical framework.

Significance: This is a fundamental theoretical contribution with a complete empirical validation pipeline. It is rare for a paper to provide both the formal apparatus and the measurement procedure and the empirical demonstration in a single coherent package. The closest analog in the CoT literature is Lightman et al. (2023), which provided a formal definition of process supervision and a benchmark (PRM800k), but their framework required human annotation. This paper's framework eliminates that requirement through an information-theoretic reformulation, making it fundamentally more scalable.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses four datasets spanning controlled synthetic data and real model-generated reasoning traces:

  • Toy data (Section 5.1): A custom-generated dataset of 5-step sequential arithmetic operations on 5-element integer vectors, where specific sub-tasks are deliberately corrupted with known error probabilities to simulate model failures at known locations. This provides ground-truth knowledge of which step is incorrect for validation.
  • Arithmetic operations on Llama-3-8B (Section 5.2): A dataset of 3-step arithmetic problems (compute $3x$, $2y$, then $3x + 2y$) with $x, y$ sampled uniformly from $[1, 10^5)$, using the Llama-3-8B model's own CoT generations. The prompt includes two worked examples followed by a query.
  • Controlled GSM8K (Section 5.3): A modified version of the GSM8K dataset (Cobbe et al., 2021) where GPT-4 generates CoT solutions with deliberately incorrect multiplication operations, while all other operations (addition, subtraction, division) are correct. The dataset is filtered so that incorrect final answers always contain both "multiplication" and "subtraction," while correct final answers contain neither, creating a spurious correlation designed to test baseline failure modes.
  • PRM800K (Section 5.4): The process supervision dataset from Lightman et al. (2023), consisting of 800,000 step-level human annotations (positive +1, negative -1, neutral 0) on GPT-4-generated solutions to MATH problems (Hendrycks et al., 2021). The paper uses this only for evaluation (to validate that information-gain aligns with human labels), not for training — the step-level labels are never used by the information-gain method.

• Base model(s). The paper evaluates several base LLMs depending on the experiment:

  • Toy data: Five synthetic "LLMs" (LLM1 through LLM5), each a deterministic program that applies the five sequential operations with controlled errors at a specific step. These are not neural models but algorithmic generators that produce CoT traces with known error locations, enabling clean validation.
  • Arithmetic experiments: Llama-3-8B (Dubey et al., 2024), an open-weight 8-billion parameter model, chosen as a representative modern LLM with non-trivial arithmetic capabilities but far from perfect accuracy (the model achieves 80%, 98%, and 42% accuracy on the three sub-tasks respectively), creating a realistic evaluation scenario.
  • Controlled GSM8K: GPT-4 (OpenAI, 2024) is used to generate the CoT traces (with deliberately incorrect multiplications), but the evaluation is of the generated reasoning, not of GPT-4 itself. The supervisor model is a fine-tuned GPT-2.
  • PRM800K: The CoT traces were originally generated by a fine-tuned GPT-4 model (as part of the PRM800k dataset creation by Lightman et al., 2023). The paper uses these pre-existing traces without re-generating them.

• Metrics. Three types of metrics are used:

  • Information-gain (IG): The primary metric, estimated as the difference in cross-entropy loss of the supervisor model between consecutive steps: $\widehat{\text{IG}}_j = \mathbb{E}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_{j-1}))] - \mathbb{E}[\ell_{\text{CE}}(Y, g_{\text{sup}}(X^M_j))]$. Positive values indicate the step contributed useful information toward predicting $Y$; near-zero or negative values indicate no useful contribution. Reported as aggregate averages over all instances where a given sub-task appears (Figures 3a, 8; Tables 2).
  • Sample-wise classification metrics: For per-instance error detection, a step is classified as incorrect if its sample-wise information-gain falls below a threshold chosen on held-out data. Reported metrics include accuracy, true positive rate (TPR), and false positive rate (FPR) (Tables 1, 2).
  • ORM probability of correctness: The mean predicted probability from the ORM classifier that the final answer will be correct given the CoT up to step $t$ (Equation 6). A significant drop after step $t$ indicates the ORM flags that step as likely erroneous.
  • Math-Shepherd proportion of correct completions: The fraction of $N = 10$ completions (for arithmetic) or variable completions (for toy data) from step $t$ onward that reach the correct final answer (Equation 7). Low proportions indicate the step is flagged as problematic.

• Baselines. The paper compares against two annotation-free baselines for step-level evaluation:

  • Outcome Reward Model (ORM) (Havrilla et al., 2024): A classifier $f_{\text{ORM}}$ trained to predict $P(Y^M = Y \mid X^M_t)$ — the probability that the model's final answer will be correct given the CoT up to step $t$. The paper trains this classifier on the same data used for the supervisor model (prompts + correct final answers + model CoT traces). In the toy and arithmetic experiments, GPT-2 and Llama-3-8B variants are used respectively; architecture details are not further specified.
  • Math-Shepherd (MS) (Wang et al., 2024): A method that evaluates step $t$ by generating multiple completions from $X^M_t$ using a completer model and computing the proportion of completions whose final answer matches the ground truth. The paper uses the same base model as the completer for fair comparison (no access to a stronger verifier). In the arithmetic experiment, 10 completions are generated per step; in the toy experiment, the number of completions is not explicitly specified but follows the same approach.

• Generation budget / compute accounting. The paper does not compare methods under a shared generation budget in the conventional sense (e.g., "all methods get $N$ samples") because the methods use compute differently:

  • The supervisor model (IG method) requires training a separate fine-tuned LLM, which incurs a one-time training cost amortized over all evaluations. The paper does not quantify this training cost in FLOPs or GPU-hours.
  • ORM requires training a classifier, also a one-time cost.
  • Math-Shepherd requires no separate training but uses $N \times T$ additional generations at test time ($N$ completions per step, $T$ steps per prompt). The paper uses $N=10$ in the arithmetic experiments.
  • The paper acknowledges (Section 6) that training the supervisor model is "computationally expensive" and suggests in-context learning as a potential future direction to reduce this cost, but does not provide FLOP comparisons between methods.

• Cross-validation / statistical protocol. For sample-wise error detection (classifying individual steps as correct or incorrect), thresholds for all methods (IG, ORM, MS) are selected using a held-out validation dataset, not the test set. The paper does not report standard errors, confidence intervals, or statistical significance tests for any of the aggregate information-gain values. Cross-validation is not used in the main aggregate analyses (Figures 3, 8; Table 2), which are computed directly on the test splits. In the PRM800k sample-wise experiment, a balanced held-out dataset with equal numbers of correct and incorrect steps is used for threshold selection.

Main Quantitative Results

Toy Data Experiments (Section 5.1): Information-Gain Correctly Localizes Errors Across All Five LLMs

The toy data experiments are the cleanest validation because they provide ground-truth knowledge of which step is incorrect. Five synthetic LLMs are created, each with a controlled error at a specific step (LLM1 has errors at step 1, LLM2 at step 2, etc.), with the special case that LLM3's error at step 3 is deterministically triggered by the output of step 2 (if $z_2[2] > 150$).

Headline result for aggregate information-gain (Figure 3a): The information-gain heatmap shows a clear pattern: for each LLM, the information-gain remains consistently positive for all steps before the erroneous step, then drops to near-zero or negative values at and after the error. For LLM1, information-gain values are 0.013, 0.27, 0.059, 0.083, and 0.014 for steps 1–5 respectively (with the error at step 1 — but note the paper reports these values for all CoT traces including correct ones; the drop is more visible in the sample-wise plots in Figure 5). For LLM2, the pattern shows a clear drop after step 2. For LLM3, the drop occurs after step 3. For LLM4, after step 4. For LLM5, after step 5. This correctly identifies the unidentifiable sub-task in each case.

Baseline failure on LLM3 (Figures 3b, 3c): ORM's heatmap (Figure 3b) shows the mean probability of correctness for LLM3 dropping dramatically at step 2 (from ~0.5 to near-zero) rather than at step 3, despite step 3 being the actual error location. This occurs because the ORM classifier learns that the final answer's correctness is entirely predictable from the state at step 2 — it sees $z_2[2] > 150$ and predicts failure, regardless of what happens at step 3. The mean probability values for LLM3 are: 0.52 at step 1, dropping to 0.0000028 at step 2 (a massive drop), then 0.0000017 at step 3, 0.0000087 at step 4, and 0.058 at step 5. The ORM correctly identifies that something is wrong but misattributes the error to step 2.

Math-Shepherd's heatmap (Figure 3c) for LLM3 shows zero correct completions from any step — all values are 0 — providing no diagnostic signal about which step is erroneous. This occurs because the error is deterministic given $z_2[2] > 150$, so all completions from any starting point will fail.

Sample-wise detection results (Table 1): For LLM3 specifically, the paper evaluates per-instance error detection performance with thresholds chosen on held-out data:

The information-gain method achieves 96% accuracy with only 6% false positive rate — it correctly identifies erroneous steps while rarely flagging correct steps as incorrect. ORM achieves the same true positive rate (98%) but at the cost of a 54% false positive rate — it incorrectly flags more than half of correct steps as erroneous. Math-Shepherd achieves perfect detection of errors (100% TPR) but at the cost of flagging all steps as incorrect (100% FPR), making it useless for localization.

Sample-wise trajectory visualization (Figures 5–7 in Appendix C.1.5): The paper provides sample-wise information-gain trajectories for 15 randomly chosen prompts with wrong final answers for each LLM. For IG (Figure 5), the information-gain remains high until the actual erroneous step, then drops sharply. For ORM (Figure 6), the drop occurs at step 2 for LLM3 rather than step 3. For MS (Figure 7), LLM3 shows zero correct completions regardless of step. These visualizations confirm the aggregate patterns and show that the failure modes are systematic, not averaging artifacts.

Arithmetic Operations on Llama-3-8B (Section 5.2): Information-Gain Identifies the Erroneous Addition Step While Baselines Are Confused by State-Dependent Errors

The Llama-3-8B model achieves 80% accuracy on step 1 ($3x$), 98% on step 2 ($2y$), and only 42% on step 3 ($3x + 2y$). Error analysis (Figure 4) reveals that most step-3 failures occur when one operand is large and the other is small — a specific difficulty pattern that makes error occurrence predictable from the input values, creating the same structural challenge for baselines as the LLM3 toy experiment.

Headline result for aggregate information-gain (Table 2): The information-gain values across the three steps are:

• Step 1 ($3x$): 0.67 (strong positive — the step contributes substantial information)
• Step 2 ($2y$): 0.24 (moderate positive — the step contributes useful information)
• Step 3 ($3x + 2y$): 0.027 (near-zero — the step contributes minimal useful information)

The dramatic drop from 0.24 to 0.027 at step 3 correctly identifies that this is where the model's reasoning fails to provide useful information toward the correct answer.

ORM failure (Table 2): The ORM's mean probability of correctness is 0.24 at all three steps — it remains completely unchanged. The ORM classifier learns to predict correctness purely from the values of $x$ and $y$ in the prompt, without the intermediate reasoning steps providing any additional signal. This is a direct consequence of Figure 4: the model's correctness is largely determined by the input values (one large, one small operands predict failure), and the ORM captures this from the prompt alone. Because the confidence doesn't change across steps, ORM cannot distinguish between the model's performance on different sub-tasks.

Math-Shepherd failure (Table 2): The proportion of correct completions is:

• Step 1 ($3x$): 0.068
• Step 2 ($2y$): 0.059
• Step 3 ($3x + 2y$): 0.00069

While step 3 shows near-zero correct completions (as expected for the actual error location), steps 1 and 2 also show very low correct-completion rates (only 6–7%) despite being executed correctly 80–98% of the time. This occurs because, for most prompts where the model ultimately fails at step 3, the $(x, y)$ combination already determines that the final answer will be wrong — the completer model cannot produce correct completions even from correct intermediate states because the overall problem is too hard for it. The result is a high false positive rate: MS correctly identifies step 3 as problematic but also flags steps 1 and 2.

Sample-wise detection (Table 2):

The information-gain method achieves 76% accuracy with only 2% false positive rate. The relatively low TPR (51%) suggests the method is conservative — it misses about half of actual errors but rarely misclassifies correct steps. ORM achieves only 56% accuracy with a TPR of just 10% — it misses 90% of errors, likely because its confidence doesn't change across steps for most instances (since correctness is predictable from the prompt alone). Math-Shepherd achieves 99% TPR (catching nearly all errors) but at the cost of an 86% FPR — it incorrectly flags the vast majority of correct steps as incorrect, making its high TPR meaningless for practical localization.

Sample-wise trajectory visualization (Figures 8–10 in Appendix C.2.3): The information-gain trajectories (Figure 8) show that for most prompts, the information-gain remains high through steps 1 and 2, then drops near zero or below at step 3 — correctly identifying the addition as the failure point. ORM trajectories (Figure 9) show near-constant confidence regardless of step, explaining its inability to localize errors. Math-Shepherd trajectories (Figure 10) show most instances at zero correct completions from all steps, explaining its high FPR.

Controlled GSM8K Experiments (Section 5.3): Information-Gain Resists Spurious Correlations That Defeat ORM

This experiment constructs a setting where all incorrect final answers involve both "multiplication" (which is deliberately executed incorrectly by GPT-4) and "subtraction" (which is always executed correctly), while correct final answers involve neither operation. This creates a spurious correlation: subtraction is perfectly correlated with wrong answers, even though subtraction itself is always performed correctly.

Headline result for aggregate information-gain (Table 2): The information-gain values for the four operations are:

• Addition: 0.99 (strong positive — the model executes this correctly and it helps predict the answer)
• Multiplication: 0.026 (near-zero — the operation is deliberately incorrect and provides no useful information)
• Division: 1.05 (strong positive — correctly executed)
• Subtraction: 1.06 (strong positive — correctly executed, despite being correlated with wrong answers)

The critical finding is that subtraction shows high positive information-gain (1.06), correctly indicating that it is being executed correctly and contributes useful information, despite the dataset-level spurious correlation linking subtraction to wrong answers.

ORM failure (Table 2): The ORM's mean probability of correctness is:

• Addition: 0.46
• Multiplication: 0.024
• Division: 0.38
• Subtraction: 0.013

The ORM assigns subtraction an extremely low correctness probability (0.013), nearly as low as multiplication (0.024), because it has learned the spurious correlation that CoTs containing subtraction lead to wrong answers (since all incorrect traces in this dataset contain both multiplication and subtraction). This is the concrete failure mode: ORM cannot distinguish between "this step is causally responsible for the error" (multiplication) and "this step happens to co-occur with errors due to data construction" (subtraction).

Sample-wise detection (Table 2):

The information-gain method achieves 72% accuracy, correctly identifying multiplication as the erroneous operation with 95% TPR, though its FPR of 62% indicates substantial room for improvement. ORM achieves 100% TPR but 100% FPR — it flags every operation as potentially incorrect, including correctly executed addition, subtraction, and division. This is a complete failure of localization: ORM cannot distinguish between operations at all under this data construction.

Why Math-Shepherd was not evaluated here: The paper states that "MS (with the same completer) fails trivially under these conditions" because multiplication is "inherently unidentifiable" — any CoT containing multiplication will have an incorrect final answer by construction, so completions from any step will fail, providing no differential signal. This is analogous to the LLM3 toy experiment where MS showed zero correct completions across all steps.

PRM800K Experiments (Section 5.4): Information-Gain Aligns with Human Step-Level Labels Without Using Them

This experiment validates the information-gain framework against human judgments on real, naturally occurring reasoning errors (not artificially constructed ones). The PRM800K dataset provides human annotations labeling each reasoning step as positive (+1, correct), negative (-1, incorrect), or neutral (0, ambiguous/irrelevant). The information-gain method does not use these labels during training — they serve only as evaluation ground truth.

Headline result for aggregate information-gain (Table 2): The information-gain values by human label are:

• Positive (+1) steps: 0.168 (positive — correct steps contribute meaningful information)
• Neutral (0) steps: -0.011 (negative — ambiguous steps do not add information and may slightly reduce predictability)
• Negative (-1) steps: 0.058 (near-zero — incorrect steps contribute minimal useful information)

The information-gain correctly orders the step types: correct steps show substantially higher information-gain than incorrect or neutral steps. The neutral steps showing negative information-gain is noteworthy — these steps, which human annotators deemed irrelevant or ambiguous, actually make the correct answer less predictable, consistent with the interpretation that they add noise or unhelpful information.

ORM failure (Table 2): The ORM's mean probability of correctness is:

• Positive (+1) steps: 0.744
• Neutral (0) steps: 0.745
• Negative (-1) steps: 0.734

The ORM probabilities are nearly identical across all three categories (all around 0.734–0.745), providing essentially no discrimination between correct and incorrect steps. This is not a failure of the specific ORM implementation — it reflects the fact that on complex MATH problems, the probability of the final answer being correct does not vary substantially based on whether a single intermediate step is correct, especially when many steps remain. The ORM captures aggregate trajectory quality but cannot localize individual step errors.

Sample-wise detection (Table 2): For classifying individual steps as correct or incorrect (neutral steps filtered out, balanced held-out dataset):

The information-gain method achieves higher accuracy (74% vs. 69%) and substantially higher TPR (84% vs. 55%), meaning it catches many more actual errors. However, its FPR is higher (37% vs. 18%), meaning it also incorrectly flags more correct steps as erroneous. This represents a different point on the precision-recall trade-off curve (chosen by the threshold selection on held-out data), not necessarily a strictly superior classifier.

What the PRM800K results demonstrate: The aggregate results in Table 2 provide the strongest evidence that the information-gain framework captures something genuinely related to step-level reasoning quality, because the alignment with human judgments emerges without any step-level supervision. The information-gain correctly ranks step types in the order human annotators would expect (positive > negative > neutral), while ORM fails to discriminate between them at all. This is the paper's most ecologically valid validation — it uses real CoT traces on real competition math problems with real human judgments, not artificial errors.

Ablation Studies and Robustness Checks

Supervisor model architecture choice across experiments: The paper uses three different supervisor model configurations — GPT-2 for toy data and GSM8K, Llama-3-8B with LoRA for arithmetic, and GPT-2 for PRM800K — but does not ablate this choice within any single experiment. There is no comparison of, e.g., GPT-2 vs. Llama-3-8B as supervisors on the same data, so the sensitivity of information-gain estimates to supervisor model capacity is unknown.

Effect of training on all traces vs. only correct traces: Not explicitly ablated, but the paper's methodology description (Section 3.3, Appendix C.1.3) emphasizes training the supervisor on all generated CoT traces, including those where the base model's final answer is incorrect. The justification is that the supervisor must learn to recognize when intermediate states are leading toward wrong answers, which requires seeing incorrect intermediate states paired with correct final answers. No experiment compares performance with and without this design choice.

Effect of the separator token and fine-tuning format: The paper uses a specific format for supervisor training data: [CoT up to step t] #|> [correct answer Y], where #|> is a special separator token that signals the model to predict $Y$ directly. No ablation of this format (e.g., using natural language prompts instead, or different separator tokens) is performed. The tokenization and the inclusion of previous answer tokens in the conditioning for cross-entropy computation are described in Appendix C.1.4 but not varied experimentally.

Threshold selection for sample-wise classification: The paper selects thresholds for all methods on held-out validation data, which is standard practice. However, no sensitivity analysis of how performance varies with threshold choice is presented. The reported accuracy/TPR/FPR values represent a single operating point on the precision-recall curve, and it is unclear whether the information-gain method's advantage is robust across different threshold choices or specific to the selected threshold.

Number of Math-Shepherd completions: In the arithmetic experiment, Math-Shepherd uses $N=10$ completions per step. No ablation over this choice is performed — it is possible that more completions would improve MS performance, though the paper's structural critique (that MS fails when error determinants precede the error) suggests fundamental limitations independent of completion count.

Effect of LoRA rank and training hyperparameters: The Llama-3-8B supervisor for the arithmetic experiment uses LoRA fine-tuning, but LoRA hyperparameters (rank, alpha, target modules) are not reported. The GPT-2 supervisors use SFT until convergence with early stopping on validation loss, but learning rates, batch sizes, and convergence criteria are not specified for these experiments. This makes exact replication difficult and leaves open the possibility that supervisor model undertraining or overfitting could affect information-gain estimates.

Oracle vs. self-generated CoT traces for supervisor training: In the GSM8K and PRM800K experiments, the supervisor is trained on CoT traces generated by GPT-4 (the base model that produced them), not on the supervisor model's own CoT traces. This is appropriate because the supervisor is evaluating those specific traces. However, there is no experiment testing whether a supervisor trained on one model's CoTs transfers to evaluating another model's CoTs — the distribution shift question is raised implicitly but not addressed.

Critical Assessment

Claim 1: The information-gain framework correctly identifies unidentifiable sub-tasks and detects erroneous reasoning steps without annotated intermediate data. This claim is well-supported by the toy experiments (Section 5.1), where the ground-truth error locations are known by construction. Across all five LLMs with errors at different steps, the information-gain drops precisely at the step where the error occurs (Figure 3a). The LLM3 experiment is particularly convincing because it demonstrates resistance to a specific failure mode (state-dependent error occurrence) that defeats both baselines. The PRM800K experiment provides external validation against human judgments on real data — the information-gain ordering (positive > negative > neutral) aligns with human labels without using them in training, and sample-wise accuracy (74%) exceeds ORM (69%).

However, the claim's scope should be carefully bounded. The toy data provides the only experiment where error locations are known with certainty, and the toy tasks (vector manipulations) are far simpler than real mathematical reasoning. The PRM800K experiment shows correlation with human labels, not perfect agreement — 74% sample-wise accuracy means the method is wrong about step correctness 26% of the time. The GSM8K experiment uses artificially constructed errors (deliberately incorrect multiplications), not naturally occurring model failures. The arithmetic experiment on Llama-3-8B identifies the problematic step but with only 51% TPR for sample-wise detection — the method misses nearly half of individual errors. The paper would be strengthened by an experiment on naturally occurring errors from a real LLM on a standard benchmark, without artificial data manipulation, to assess how well information-gain localizes errors in realistic deployment scenarios.

Claim 2: Outcome-based baselines (ORM, Math-Shepherd) systematically fail when error occurrence is predictable from information available before the erroneous step. This claim is strongly supported and represents the paper's most robust empirical contribution. The LLM3 toy experiment (Figure 3b-c, Table 1) provides a clean demonstration where the error at step 3 is determined by step 2's output, and both ORM (54% FPR) and MS (100% FPR) catastrophically fail to localize the error. The arithmetic experiment (Table 2) shows ORM confidence unchanged across all steps because correctness is predictable from the prompt, while MS shows uniformly low completion rates across all steps, both failing to identify step 3 as the error. The GSM8K experiment (Table 2) shows ORM flagging correctly executed subtraction as suspicious (probability 0.013) due to spurious correlation, while information-gain correctly assigns it high positive gain (1.06). The PRM800K experiment (Table 2) shows ORM probabilities nearly identical across human-labeled correct (0.744) and incorrect (0.734) steps, failing to discriminate between them.

This pattern is consistent across four datasets with different error structures, different models, and different task complexities. The failure is not implementation-specific — it follows from the logical structure of outcome-based evaluation: $P(\text{final correct} \mid X^M_t)$ can change because of information in $X^M_t$ that predicts future errors, not because of errors at step $t$ itself. The paper makes a compelling case that this is a fundamental limitation, not a fixable bug.

However, the paper would be strengthened by a more systematic characterization of when this failure mode occurs. Is it rare or common in practice? The GSM8K experiment constructs the spurious correlation deliberately — how often do such correlations arise naturally in real LLM CoT traces on standard benchmarks? The paper does not provide this analysis, leaving the practical significance of the baseline failure mode somewhat uncertain. Additionally, the paper does not explore whether ensembling ORM with other signals or using causal intervention techniques could mitigate the spurious correlation problem while retaining the annotation-free advantage.

Claim 3: The information-gain framework provides granular, per-sub-task performance profiles without step-level labels. This claim is supported by the experiments that aggregate information-gain by sub-task type. In the arithmetic experiment (Table 2), the method correctly identifies that step 3 (addition of $3x+2y$) is the problem while steps 1 and 2 (multiplication by constants) are executed correctly — a granular diagnostic that tells you which operation to improve, not just "the model fails sometimes." In the GSM8K experiment (Table 2), the method correctly isolates multiplication as the problematic operation while correctly flagging addition, subtraction, and division as well-executed.

However, this claim assumes that sub-task categorization is available — the paper acknowledges (Section 6) that "we still need to categorize each step according to its respective sub-task." For the arithmetic experiment, this categorization is trivial (the prompt explicitly asks for three separate computations). For GSM8K, the paper used GPT-4 to label which steps involve which operations — itself an additional annotation step. For PRM800K, the human labels provide step categories that the paper uses for aggregation but does not explain how these would be obtained in a deployment without such labels. The per-sub-task diagnostic is therefore gated on having sub-task labels, which partially undermines the "no annotation" claim — the method avoids correctness labels but still requires categorization labels. The paper acknowledges this limitation explicitly in Section 6.

Claim 4: The framework extends to non-linear reasoning structures including backtracking and self-correction. This claim is asserted theoretically in Appendix B.4 and mentioned in Section 3.3, but is not empirically validated in the paper. None of the four experimental settings involve branching reasoning, backtracking, or self-correction — the toy data uses linear chains, the arithmetic task is strictly sequential, the GSM8K experiment uses linear CoTs, and the PRM800k traces are predominantly linear. The paper states that "steps that human annotators labelled as uninformative or irrelevant consistently show low information gain under our metric" (Section 5.4) as evidence for handling non-linear reasoning, but "uninformative" neutral steps in a linear chain are not the same as exploratory branches that are later abandoned. The claim about O1/R1-style reasoning remains purely theoretical and represents a significant gap between the paper's claimed scope and its empirical coverage.

Additional weaknesses not acknowledged by the paper:

• Supervisor model training cost is substantial and unquantified. The paper acknowledges this as a limitation (Section 6) but does not report the actual computational cost of training supervisor models for any experiment — no GPU-hours, no FLOP counts, no comparison to the cost of alternative methods. For the Llama-3-8B supervisor with LoRA, this cost is likely modest relative to full fine-tuning, but for large-scale deployment across many models and datasets, the cumulative supervisor training cost could be significant. The paper's suggestion of in-context learning as a future direction implicitly acknowledges this is a current limitation.

• No statistical uncertainty quantification. All reported information-gain values, accuracies, TPRs, and FPRs are point estimates without standard errors, confidence intervals, or any measure of statistical reliability. Given the relatively small sample sizes in some experiments (the arithmetic experiment's test set size is not reported; the PRM800k experiment uses a "balanced held-out dataset" of unspecified size), the precision of these estimates is unknown. The 96% vs. 77% accuracy difference for LLM3 sample-wise detection (Table 1) is large enough to likely be significant, but the 74% vs. 69% accuracy on PRM800K (Table 2) could potentially fall within estimation noise.

• The framework's assumptions (3.1 and 3.2) are not directly tested. Theorem 3.3 guarantees that zero information-gain ⇒ unidentifiability under the assumptions, but the paper only tests whether the information-gain metric correlates with known errors — it does not test whether the conditional independence structure in Assumption 3.1 actually holds in real CoT traces, or whether real LLMs satisfy compositional consistency (Assumption 3.2). The method could work empirically even if the assumptions are violated, or could fail in settings where they don't hold, and the paper provides no diagnostic for distinguishing these cases. This is a gap between the theoretical framework and the empirical validation — the theory provides sufficient conditions for the method to work, but the experiments only show that the method works in specific settings, not that the sufficient conditions are met.

• Single supervisor model per experiment — no ensemble or calibration analysis. The paper uses one supervisor model architecture per experiment and does not investigate whether information-gain estimates are stable across different supervisor initializations, architectures, or training runs. If information-gain is sensitive to supervisor model quality, then the framework's reliability depends on having a well-calibrated supervisor, which may not always be achievable.

• No comparison to process reward models on the PRM800k dataset. The paper's Table 3 qualitatively compares IG to PRMs but does not provide a quantitative comparison — e.g., how does a PRM trained on 10% of the PRM800k labels compare to IG (trained on zero labels) on the remaining 90%? This would directly quantify the cost of avoiding annotation versus the benefit of using it. The paper critiques PRMs for requiring expensive human annotation but never empirically demonstrates that IG achieves comparable performance to a PRM at a fraction of the annotation cost, which would be the most persuasive argument for the framework's practical value.

• No experiment on a domain where primitive task decomposition is non-obvious. All experiments involve arithmetic or mathematical reasoning where sub-task boundaries (addition, multiplication, etc.) are clearly defined. The paper claims applicability to Blocks World, commonsense QA, and other domains (Appendix B.2) but provides no empirical evidence. The framework's reliance on well-defined primitive tasks and sub-task categorization may limit its applicability to domains where reasoning steps cannot be cleanly categorized, which the paper does not address empirically.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation (When Using This for Compute-Optimal Allocation) Is Not Amortized into the Headline Gains

The assumption or constraint. The entire compute-optimal framework described in Section 2 relies on knowing which difficulty bin a prompt falls into before allocating the inference budget. The paper's method for estimating this difficulty — generating 2048 samples per prompt and scoring them with the PRM — is extraordinarily expensive. The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline 4× efficiency improvement (e.g., 16 generations matching best-of-N at 64, or 64 matching 256) is computed after difficulty is known, without including the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation PLUS strategy execution. Since the difficulty estimation alone (2048 samples) exceeds the largest test-time budgets studied (256–512 generations), the claimed efficiency gains could be entirely consumed — or even reversed — once the estimation cost is included. A practitioner who deploys this method without accounting for the difficulty estimation overhead may find that their total inference cost is actually higher than simply running best-of-N at a fixed high budget, despite the per-problem allocation being "optimal."

What evidence exists in the paper. No experiment amortizes the difficulty estimation cost into the reported efficiency numbers. The 4× claims in Figures 4 and 8 are computed post-hoc after difficulty bins are assigned, and the difficulty estimation procedure (2048 samples per question) is described in Section 3.2 but never included in any budget calculation. The authors flag this as an exploration-exploitation tradeoff and a key avenue for future work, but provide no empirical characterization of how the total-cost-adjusted efficiency compares to the reported figures.

Mitigation status. The paper does not attempt to mitigate this limitation. It suggests future work on training models to predict difficulty directly from the question text (Section 8), or using adaptive difficulty estimation that starts with a few samples and dynamically adjusts, but neither approach is developed or evaluated. The predicted difficulty bins (using PRM scores instead of ground-truth correctness) still require the same 2048-sample generation step, just with a different scoring function. The paper is transparent about this gap but provides no path to closing it within the current work.

---

Hard Problems (Difficulty Bin 5) Remain Essentially Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper's compute-optimal framework assumes that the base model's pass@1 rate on a problem is non-trivially above zero — that is, some correct solutions exist in the model's proposal distribution to be found or refined. For the hardest problems in the MATH benchmark, this assumption is violated. Section 7 states:

"test-time compute provides essentially zero benefit regardless of budget on the hardest problems"

The consequence. Across all methods studied — search against PRM verifiers (Section 5), iterative revisions (Section 6), and their compute-optimal combinations — the hardest difficulty bin (bin 5) shows near-zero improvement with any amount of additional compute. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is flat near 0–5% while the 14× larger model's performance remains superior. This is not a gradual degradation — it is a hard ceiling. Test-time compute amplifies existing capability but cannot create capability from nothing. If the base model cannot produce correct solutions at any meaningful rate (pass@1 near zero), no amount of search, revision, or adaptive allocation will help.

This fundamentally bounds the approach's applicability: it cannot be used to solve problems that genuinely exceed the base model's training distribution or reasoning capabilities. For any organization whose problem distribution includes a substantial fraction of genuinely hard (for their model) problems, pretraining a larger model — or acquiring fundamentally different capabilities through other means — remains the only viable path.

What evidence exists in the paper. The bin 5 results are consistent across all major figures: Figure 3 (right, search methods), Figure 7 (right, revision strategies), and Figure 9 (FLOPs-matched comparison). In the FLOPs-matched analysis, the paper explicitly quantifies this: at the $R \gg 1$ regime, hard problems show a −52.9% relative disadvantage from using test-time compute instead of the 14× larger model (Figure 1, bottom-right bar chart; Section 7). The authors are transparent about this: "test-time compute provides essentially zero benefit regardless of budget on the hardest problems" (Section 7 takeaway box).

Mitigation status. The paper does not attempt to address this limitation — and fundamentally cannot, since it is a capability bound rather than a methodological shortcoming. The authors position this as a boundary condition that defines when test-time compute is appropriate versus when pretraining must be scaled instead. Future work on combining test-time compute with retrieval-augmented generation, tool use, or multi-model collaboration might push this boundary, but those directions are not explored.

---

The Framework Is Validated on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*) — Generalization to Other Domains, Models, and Reasoning Types Is Unproven

The assumption or constraint. All experiments in the paper — search against PRM verifiers, iterative revisions, compute-optimal allocation, and FLOPs-matched comparisons — use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. MATH consists of high-school competition-level mathematics problems requiring symbolic reasoning with exact-answer grading. Section 4 justifies this choice:

"we believe this model is representative of the capabilities of many contemporary LLMs"

The consequence. Several aspects of the paper's findings could be specific to this model-benchmark combination and may not transfer to other settings:

• Model-specific behavior: The PRM's quality and over-optimization characteristics depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or a different base capability profile might exhibit different difficulty-dependent scaling curves. The paper's specific finding that beam search degrades on easy problems at high budgets (Figure 3, right) may depend on PaLM 2-S*'s specific verifier interaction, and may not hold for other models.

• Benchmark-specific reasoning structure: MATH problems require multi-step symbolic deduction with a single correct answer. It is unclear whether the difficulty-dependent patterns (revisions better on easy problems, search better on medium problems) generalize to other reasoning domains — code generation (where correctness is verified by unit tests), logical reasoning (where multiple valid proof paths exist), scientific QA (where answers may require factual recall in addition to inference), or open-ended generation tasks (where "correctness" is ill-defined).

• Dataset size and statistical reliability: The 500-question MATH test set, split into five difficulty quintiles of ~100 questions each, then further split by two-fold cross-validation for compute-optimal strategy selection, means strategies are selected based on ~50 questions per fold per bin. The paper reports no confidence intervals on the compute-optimal scaling curves, making it unclear whether observed differences between strategies (especially at the narrow margins seen at high budgets) are statistically reliable or within estimation noise at this sample size.

• Exact-answer grading specificity: MATH problems have ground-truth answers that can be verified with exact string matching, enabling both the difficulty estimation pipeline (via pass@1 rates) and the PRM training pipeline (via Monte Carlo rollout correctness). Extending the framework to tasks without clean correctness signals — dialogue, creative writing, complex multi-step planning — would require fundamentally different verifier training and difficulty estimation approaches.

What evidence exists in the paper. No experiment uses any benchmark other than MATH, and no experiment uses any base model other than PaLM 2-S*. The paper does not report confidence intervals for any metrics. The 500-question test set size is reported in Section 4, and the cross-validation procedure is described in Section 3.2, but the authors do not discuss the implications of ~50 questions per fold for strategy selection reliability.

Mitigation status. The paper does not attempt to address this limitation empirically. The authors acknowledge in Section 8 that "future work could explore the application of our framework to other domains," but provide no replication on even a second benchmark or model. The claim that PaLM 2-S* is "representative" is asserted without evidence. A practitioner considering deployment would need to validate the key findings — particularly the difficulty-dependent optimal strategy pattern — on their specific model and problem distribution, which may require replicating substantial portions of the experimental pipeline.

---

Verifier Over-Optimization Is a Hard Ceiling on Test-Time Compute Scaling, and the Compute-Optimal Policy Merely Works Around It Rather Than Solving It

The assumption or constraint. The paper's compute-optimal framework relies on a learned verifier (PRM or ORM) to score candidate solutions or guide search. The quality of this verifier fundamentally bounds what test-time compute can achieve. Section 5.2 documents that verifier over-optimization — where search finds solutions that score highly under the verifier but are actually incorrect — is the primary bottleneck preventing unbounded improvements from additional compute.

The consequence. The compute-optimal policy can mitigate verifier over-optimization by routing easy problems away from aggressive search (where over-optimization is most severe, as shown in Figure 3, right, where beam search degrades easy-problem performance at high budgets) and toward methods that are less susceptible (best-of-N). However, it does not solve the underlying problem.

On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling. The beam search curves in Figure 3 (right) flatten and sometimes decline well before the budget is exhausted — performance saturates not because the model has no more correct solutions to find, but because further search optimization finds solutions that exploit the verifier signal rather than genuinely correct ones. The qualitative examples in Appendix M (Figures 29, etc.) show this concretely: search produces degenerate outputs — repetitive low-information steps, overly short 1–2 step solutions — that score highly under the PRM but are actually wrong.

This means the compute-optimal approach has an inherent ceiling determined by verifier quality, not model capability. Improving the PRM (through better training data, adversarial robustness, ensemble methods, or better calibration) would likely shift the difficulty thresholds and change the optimal policy entirely — the current compute-optimal strategy is optimal only given the current verifier quality. A practitioner who deploys this framework may find that their gains are limited not by their inference budget but by their verifier's reliability, and that further investment in verifier improvement yields higher returns than further investment in compute-optimal allocation.

Additionally, lookahead search — the most powerful optimizer studied — paradoxically performs worst overall at the same budget (Figure 3, left), because its more aggressive optimization of the verifier signal leads to more severe over-optimization, eating up the budget that could have been spent on broader exploration. This is a concrete demonstration that "more powerful search" is not the answer — verifier robustness is.

What evidence exists in the paper. Figure 3 (right) shows beam search degrading on bin 1 (easiest) problems from roughly 78% to 77% as budget increases from 4 to 256 generations, while best-of-N continues improving to 88%. Figure 3 (left) shows lookahead search underperforming beam search and best-of-N at the same generation budget. Appendix M provides qualitative examples of over-optimized degenerate outputs. Section 8 explicitly identifies "improving verifier robustness" as a key direction for future work.

Mitigation status. The paper does not attempt to improve verifier robustness — it works with the PRM as trained and designs the allocation policy to route around its weaknesses. The paper acknowledges this as a central open problem in Section 8: "improving verifier robustness is the key bottleneck for further scaling test-time compute." No experiments explore whether better verifier training, ensemble methods, or constrained search could push the over-optimization threshold higher. The current framework accepts verifier quality as given and optimizes around it, which is a reasonable engineering choice but means the gains are fundamentally bounded by a component the paper does not attempt to improve.

---

Revisions and PRM Search Are Studied Independently — The Potentially Largest Gains from Combining Them Are Unexplored

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided search (Section 5) and iterative revisions (Section 6) — as independent scaling axes. They are never combined. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. This is a significant gap because the two mechanisms have complementary, difficulty-dependent strengths that suggest combination would yield gains beyond either alone:

• Revisions improve the proposal distribution — they make the model generate better candidates by conditioning on previous attempts. This is most effective on easy problems where the model's initial output is roughly correct and just needs refinement (Figure 7, right: bin 1 shows high accuracy independent of ratio; bins 2–3 benefit from sequential revisions).

• PRM search improves candidate selection — it helps find the best solution among many generated candidates. This is most effective on medium-hard problems where the model needs to explore qualitatively different solution strategies (Figure 3, right: beam search outperforms best-of-N on bins 3–4).

A combined system could use the revision model as the proposal distribution within beam search — at each step of the search tree, conditioning on previous rejected branches as context, potentially producing higher-quality candidate steps. Alternatively, the PRM could guide which revisions to pursue, deciding when a revision is on track versus when to restart from scratch. On medium-difficulty problems (bins 3–4), where both mechanisms individually show strengths, the combined gains could be substantial.

Because the paper never tests this combination, the reported performance represents a lower bound on what the full framework could achieve. The 4× efficiency gains, the FLOPs-matched advantages, and the scaling curves all represent what is possible with each mechanism independently, not what is possible with both integrated. A practitioner implementing this approach may find that combining revisions with PRM search unlocks additional gains not captured in the paper's analyses, but conversely, may encounter unforeseen interactions (e.g., revision model outputs might be harder for the PRM to score accurately due to distribution shift) that degrade the combination below either individual mechanism.

What evidence exists in the paper. No experiment combines revisions with PRM tree-search. Section 8 lists this as an explicit direction for future work. The complementary difficulty-dependent strengths are documented in Figures 3 (right) and 7 (right) but never tested jointly. The PRM was trained on base model outputs, not revision model outputs — Section 6.1 notes that the base-model PRM underperforms when scoring revision model outputs (distribution shift, Figure 15a), suggesting that a combined system would likely need a revision-specific verifier, adding implementation complexity.

Mitigation status. Not addressed. The paper identifies this as future work in Section 8 but provides no preliminary results, ablation, or even a proposed architecture for combination. The independent study of the two mechanisms is a deliberate scoping choice that makes the analysis tractable, but it leaves the most practically interesting configuration unexplored.

---

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate, and Revision Training Is Fragile to Data Generation Choices

The assumption or constraint. The revision model is fine-tuned on trajectories where all in-context answers are incorrect, followed by a correct target answer (Section 6.1). This training data construction creates an asymmetry: the model learns to revise incorrect answers into correct ones, but never learns what to do when the current answer is already correct. At inference time, when the revision chain happens to produce a correct answer, the model has no training signal for recognizing this and stopping — it will "revise" the correct answer into something else.

The consequence. Section 6.1 reports that approximately 38% of correct answers get converted back to incorrect ones during the revision chain:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This is a substantial fraction — more than one in three correct answers are subsequently "revised away." The paper mitigates this with a selection mechanism (majority voting or verifier-based selection) that picks the best answer from anywhere in the revision chain, rather than always taking the final revision. This is a reasonable engineering patch, but it has two problems:

First, it means the revision chain is wasteful — the model generates revisions that actively degrade quality for 38% of correct answers, consuming compute budget to move away from correct solutions. A better-trained revision model that knows when to stop would preserve these correct answers and use the budget for problems that genuinely need improvement.

Second, the selection mechanism itself is imperfect. Majority voting and verifier-based selection both have error rates — they can select incorrect answers from the chain even when a correct answer appeared earlier. The 38% reversion rate means the selector must work against a model that is actively introducing errors into previously correct solutions, making the selection problem harder.

More fundamentally, the revision training procedure is fragile. The ReST$^{EM}$ experiment in Appendix K (Figure 16) shows that attempting to further optimize the revision model using on-policy RL-style training caused performance to degrade substantially:

"additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio."

The authors hypothesize that on-policy data collection "exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This suggests the revision approach is sensitive to training methodology in ways that are not fully understood — the positive results depend on specific choices (offline data construction, edit-distance-based incorrect-correct pairing) that may not transfer to other settings or survive further optimization.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1 without a formal ablation measuring its impact. The selection mechanism mitigation is described in Section 6.1 and Appendix I. The ReST$^{EM}$ failure is documented in Appendix K and Figure 16. The paper does not explore alternative training approaches that could teach the model to recognize when a current answer is correct, nor does it measure how much performance is lost specifically due to the reversion problem versus other factors.

Mitigation status. Partially addressed through the chain-level selection mechanism (majority voting or verifier-based), but the fundamental training asymmetry is not resolved. The paper does not propose modifications to the training data construction that would teach the model to preserve correct answers. The ReST$^{EM}$ failure suggests that straightforward attempts to improve the revision model (RL fine-tuning) may backfire, indicating the training procedure is not robust enough to be considered a solved component of the framework. A practitioner deploying revision models should expect the 38% reversion rate and budget for a selection mechanism that can compensate, but should also recognize that the revision training recipe may need domain-specific tuning to avoid the degradation seen with ReST$^{EM}.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new epistemological stance for evaluating Chain-of-Thought reasoning: rather than asking "is this step correct?" (the correctness framing that has dominated prior work from PRMs to ORMs), it asks "does this step contribute information toward predicting the correct answer?" This shift is not merely a new metric — it is a fundamentally different relationship to the evaluation problem, one that converts step-level assessment from a classification problem requiring ground-truth labels into an information measurement problem requiring only final-answer labels.

The magnitude of this shift is best understood by what it makes obsolete and what it makes newly possible. Prior to this work, the field faced an uncomfortable tradeoff: either pay the high cost of human step-level annotation (PRMs, Lightman et al., 2023) and get reliable step-level evaluation, or use annotation-free outcome-based heuristics (ORMs, Math-Shepherd) and accept that they will produce confident but wrong localizations whenever error occurrence is predictable from earlier states. This paper demonstrates — through controlled experiments on toy data, arithmetic, GSM8K, and PRM800k — that this tradeoff is not fundamental. The information-gain framework provides step-level diagnostics without step-level labels and without the structural failure mode that makes outcome-based methods unreliable.

The paper also resolves a latent tension in the CoT evaluation literature that has not been explicitly articulated but is evident in the empirical record. Different studies evaluating the same class of methods (outcome-based verifiers) have reached conflicting conclusions about their reliability for step-level error detection. The paper's LLM3 experiment (Section 5.1, Figure 3b-c) provides a clean explanation for these discrepancies: outcome-based methods do work when errors occur uniformly at random, because then $P(\text{final correct} \mid X^M_t)$ genuinely tracks step correctness. They fail when errors are state-dependent — triggered by specific features of the problem state that are determined before the erroneous step — because then the classifier learns to predict failure from those features, not from the step itself. Different prior studies were implicitly testing on different mixtures of these two error regimes, producing apparently contradictory findings that the paper's framework reconciles within a single coherent analysis.

Several research directions become more attractive as a result of this work:

• Principled step-level evaluation without annotation becomes a viable research target. Before this paper, one could reasonably argue that step-level evaluation fundamentally requires step-level labels — that any annotation-free approach would be heuristic and unreliable. The paper's theoretical framework and empirical validation provide a counterexample, suggesting that information-theoretic approaches can achieve what was previously thought to require human annotation. This opens the door for further theoretical work on what can and cannot be inferred about intermediate reasoning from final answers alone.

• Outcome-based reward modeling for process evaluation becomes less attractive. The paper's demonstration that ORM and Math-Shepherd produce 54% and 100% false positive rates respectively on the LLM3 experiment (Table 1) — and that ORM cannot discriminate between correct and incorrect steps on PRM800k (Table 2, probabilities 0.744 vs. 0.734) — provides strong evidence that these methods have a fundamental logical limitation, not just an implementation weakness. Researchers working on automated CoT evaluation should think carefully about whether their problem setting is likely to exhibit state-dependent errors before deploying outcome-based methods.

• Information theory as a tool for LLM interpretability gains credibility. The paper demonstrates that classical information-theoretic quantities (conditional mutual information) can be empirically estimated from LLM outputs and provide actionable diagnostic signals. This is a concrete success story for information-theoretic interpretability, which has often been more theoretically appealing than empirically successful. It may inspire similar approaches for other interpretability challenges (understanding attention patterns, detecting hallucinations, measuring factual grounding).

Conversely, some directions become less attractive:

• Simply scaling up ORM training data or classifier capacity will not fix the fundamental issue. The paper's critique is structural — the failure mode occurs because $P(\text{final correct} \mid X^M_t)$ and "step $t$ is correct" are different quantities that can diverge systematically. More data or a larger classifier will learn the spurious correlation more accurately, not eliminate it. The paper's GSM8K experiment (Section 5.3) demonstrates this concretely: ORM learns to flag correctly-executed subtraction as problematic (probability 0.013) because subtraction co-occurs with errors in the dataset construction — a larger training set would only reinforce this correlation.

• Math-Shepherd with the same model as completer is unlikely to be rescued by more completions. The paper's structural critique of Math-Shepherd is that when error determinants precede the error, all completions from all steps will fail (LLM3, Figure 3c; arithmetic, Table 2, where step 1 completions have only 6.8% success rate despite being 80% accurate). Generating more completions doesn't change the fact that the problem is fundamentally too hard for the completer model when the error determinant is already present.

Follow-Up Research This Work Enables

Directly testing Assumptions 3.1 and 3.2 on real LLM reasoning traces. The paper's theoretical guarantees depend on two assumptions that are never directly validated: the Bayesian network structure for divergent reasoning paths (Assumption 3.1) and the claim that model execution is restricted to $\text{Span}(\Gamma^M)$ (Assumption 3.2). A follow-up study could test Assumption 3.1 by constructing reasoning paths that share a prefix and then diverge at a known unidentifiable step (e.g., by deliberately introducing a specific arithmetic error), then measuring whether the subsequent states are conditionally independent of each other given the last shared state. This would involve collecting many CoT traces, embedding intermediate states, and performing conditional independence tests — a non-trivial empirical challenge but one that would either validate the paper's theoretical foundation or identify boundary conditions where it breaks. For Assumption 3.2, one could test whether models can execute tasks that are demonstrably outside their training primitive span — e.g., tasks requiring arithmetic bases or operations that don't appear in the training data — and measure whether the model's output distribution matches any composition of training primitives. Negative results (the assumptions failing in some regimes) would be as informative as positive ones.

Training a lightweight difficulty predictor to close the cost-amortization gap. The paper's compute-optimal allocation framework (Sections 3.2, 5.3, 6.2) relies on difficulty estimates that currently cost 2048 samples per prompt — far more than the test-time budgets being optimized. A direct follow-up would train a small classifier (potentially distilled from the PRM) that takes only the question text as input and predicts the difficulty bin. The training data already exists: for the 500 MATH test questions, the paper has oracle difficulty labels (ground-truth pass@1 rates binned into quintiles) and predicted difficulty labels (PRM score averages binned the same way). A strong follow-up would compare the efficiency of the compute-optimal policy using this cheap difficulty predictor against (a) the oracle-bin policy, (b) the PRM-2048-sample policy, and (c) a uniform best-of-N baseline, reporting total FLOPs including the cost of difficulty estimation. If the cheap predictor achieves comparable accuracy to the PRM-based estimator, the framework becomes immediately practical. If not, it quantifies how much efficiency is lost to difficulty estimation error.

Combining PRM tree-search with the revision model as the proposal distribution. Section 8 explicitly flags the independent study of search and revisions as a limitation. A natural and high-priority follow-up would use the revision model (Section 6) to generate candidate steps within a PRM-guided beam search (Section 5.2). The key question is whether the revision model's stronger proposal distribution (better initial candidates due to conditioning on previous attempts) compensates for the fact that the PRM was trained on base model outputs and may be miscalibrated for revision model outputs (as shown in Figure 15a). A well-designed experiment would compare three configurations on the MATH benchmark: (a) base model + PRM beam search, (b) revision model + PRM beam search (with the base-model PRM), (c) revision model + PRM beam search with a revision-specific PRM (trained on revision model outputs using the same Monte Carlo rollout procedure). If configuration (c) significantly outperforms (a), it would demonstrate that the complementary strengths of the two mechanisms compound. If all configurations perform similarly, it would suggest that the PRM's selection capability, not the proposal quality, is the binding constraint — redirecting research effort toward verifier improvement.

Replicating the difficulty-dependent strategy pattern on at least one non-mathematical reasoning benchmark. The paper's central empirical finding — that the optimal test-time strategy depends on problem difficulty, with revisions better on easy problems and search better on medium problems — is validated only on MATH with PaLM 2-S*. A replication on a code generation benchmark (e.g., HumanEval or MBPP) would test whether the pattern transfers to a domain with different reasoning structure (algorithmic rather than arithmetic) and different verifier characteristics (unit tests vs. learned PRMs). The replication would need to: (a) define difficulty bins based on the base model's pass@1 rate on the coding benchmark, (b) train a verifier (likely outcome-based using unit test pass/fail since step-level verification for code is harder to define), (c) implement or approximate a revision mechanism (e.g., showing the model its previous incorrect solution and asking for a corrected version), and (d) measure whether the compute-optimal strategy varies with difficulty in the same pattern. Negative results — e.g., finding that search always outperforms revisions on code regardless of difficulty — would refine our understanding of which reasoning domains benefit from which test-time strategies, and whether the MATH-specific findings are general or domain-specific.

Characterizing the prevalence of state-dependent errors in real LLM CoT traces. The paper's strongest contribution is demonstrating a structural failure mode of outcome-based evaluation (ORMs flagging the wrong step when error occurrence is predictable from earlier states). However, the paper constructs this failure mode deliberately — it does not measure how often it naturally occurs in unmodified LLM reasoning on standard benchmarks. A follow-up study could collect CoT traces from a model like Llama-3-70B on GSM8K or MATH, label the errors (using human annotation or strong model verification), and then measure: (a) what fraction of errors are "state-dependent" in the sense that their occurrence is predictable from features of the problem or earlier reasoning states, and (b) at what rate ORM misattributes these errors to wrong steps. If state-dependent errors are rare in practice (<5% of errors), the paper's critique, while theoretically valid, has limited practical significance for current benchmarks. If they are common (>30%), it would mean that a substantial fraction of prior work using ORM-based step evaluation has produced systematically misleading results — a finding with significant implications for how the field evaluates CoT reasoning going forward.

Practical Applications and Downstream Use Cases

Targeted fine-tuning driven by per-sub-task information-gain diagnostics. The paper's per-sub-task information-gain aggregation (Section 5.2, Table 2) provides a concrete workflow for improving LLM reasoning: (1) run the model on a benchmark, (2) compute information-gain separately for each operation type (addition, multiplication, subtraction, etc.), (3) identify operations with near-zero or negative gain as capability gaps, (4) fine-tune the model specifically on those operations, (5) re-evaluate to confirm the gain increases. The arithmetic experiment with Llama-3-8B demonstrates this concretely: the information-gain drops from 0.67 and 0.24 for steps 1 and 2 to 0.027 for step 3, correctly identifying that addition of disparate-magnitude numbers is the bottleneck — a diagnosis that could focus fine-tuning effort on exactly that operation rather than retraining the model broadly on arithmetic. This workflow requires no human annotation of intermediate steps, only prompt-final-answer pairs for the supervisor model training, making it feasible for organizations that have evaluation datasets but not step-level labels.

Automated error localization in LLM-generated solutions for human-in-the-loop verification. The sample-wise information-gain metric (Equation 5, Section 3.3) can flag specific steps within an individual CoT trace as likely incorrect — achieving 96% accuracy on toy data (Table 1) and 74% accuracy on PRM800K (Table 2). This enables a deployment pattern where an LLM generates a solution, the information-gain framework highlights the specific step(s) where reasoning appears to diverge from correctness, and a human reviewer focuses attention on those steps rather than reading the entire trace. For applications where correctness is critical (medical reasoning, financial calculations, legal analysis), this could substantially reduce human review time while catching errors that would be missed by end-to-end verification alone. The key practical requirement is a trained supervisor model for the relevant domain — a one-time cost that amortizes over many reviews.

Quality filtering for self-improvement data pipelines. Self-improvement methods like STaR (Zelikman et al., 2022) and ReST$^{EM}$ (Singh et al., 2024) rely on generating reasoning traces, filtering for correct final answers, and fine-tuning on those traces. The paper's information-gain framework offers a more granular filter: even among traces with correct final answers, steps with negative or near-zero information-gain can be identified as potentially incorrect or uninformative and excluded from training data. The paper notes in Appendix B.3 that only 1.2% of samples in the arithmetic experiment exhibit correct final answers with incorrect intermediate steps — but for more complex problems where this rate may be higher, filtering out uninformative steps could improve the quality of self-improvement data without requiring full human annotation of intermediate reasoning. The supervisor model cost is the main barrier to adoption — a practitioner would need to train one for their specific domain — but once trained, it provides a reusable quality filter.

When to Prefer This Method

The paper explicitly positions information-gain against two alternatives — process reward models (PRMs, requiring human step-level annotation) and outcome-based methods (ORMs, Math-Shepherd, annotation-free but with identified failure modes). Table 3 provides a qualitative comparison. The decision rule follows directly from the paper's claims:

• Prefer information-gain (IG) over PRMs when you need per-step evaluation but human annotation of intermediate reasoning steps is infeasible due to cost, domain expertise requirements, or scale. The paper demonstrates that IG achieves 74% sample-wise accuracy on PRM800K without using any step-level labels, compared to PRMs which would require annotating 800,000 steps. The tradeoff is accuracy: IG is less accurate than a well-trained PRM would be (though the paper does not quantify this gap), but its annotation cost is zero for the step-level signal.

• Prefer information-gain (IG) over ORM/Math-Shepherd when your problem setting is likely to exhibit state-dependent errors — that is, when the probability of an error at a given step depends on features of the problem state that are determined before that step is executed. The paper demonstrates that in this regime, ORM produces 54% false positive rate (Table 1, LLM3 experiment) and MS produces 100% false positive rate. IG achieves 6% FPR in the same setting. The cost of IG is training a supervisor model (a one-time expense), whereas ORM also requires training a classifier, so the additional cost is primarily in the inference-time computation of information-gain from the supervisor's predictions.

• Prefer ORM or Math-Shepherd over IG when you need the simplest possible implementation and can verify (or assume) that errors in your setting occur approximately uniformly at random rather than being state-dependent. In this regime, the paper's critique does not apply, and ORM/MS are simpler to implement (no separate supervisor model training needed for MS, simpler classifier training for ORM). However, the paper provides no diagnostic for determining whether your setting is in this regime without already having ground-truth error locations, which limits the practical utility of this preference.

• Do not use any outcome-based method (ORM or MS) for step-level evaluation if your problem setting structurally guarantees that certain intermediate states always lead to wrong answers — e.g., if the CoT format itself encodes error information. The GSM8K experiment (Section 5.3) demonstrates this failure mode: because the data construction guarantees that all incorrect final answers contain multiplication (always wrong) and subtraction (always right), ORM learns to flag subtraction as incorrect with probability 0.013 (Table 2), producing 100% false positive rate on subtraction steps. In such settings, only IG or PRMs can provide meaningful step-level discrimination.