RATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoning

ArXiv: 2410.01044

🎯 Pitch

LLMs skip logical steps because everyday text omits them—so RATIONALYST mines 79k unstated rationales from web data and uses them to police reasoning chains, without injecting the rationale directly. The result is a lightweight LLaMa-8B process supervisor that boosts accuracy 3.9% across seven reasoning benchmarks, outscoring even GPT-4’s verifier.

---

1. Executive Summary

This paper introduces RATIONALYST, a three-stage approach for improving LLM reasoning by mining implicit rationales—logical connections left unstated in everyday text—from web-scale unlabeled data and using them to provide process supervision during inference. The system extracts 79k rationales from The Pile and reasoning datasets, trains a specialized rationale generation model from LLaMa-3-8B-Instruct, and then uses that model to guide an agent LLM's step-by-step reasoning by scoring candidate next steps against generated rationales (implicit supervision, where the rationale estimates the probability of each candidate continuation rather than being directly inserted into the reasoning chain). RATIONALYST improves accuracy by an average of 3.9% across 7 reasoning benchmarks spanning mathematical, commonsense, scientific, and logical reasoning, and outperforms both GPT-4's process supervision and similarly sized verifiers fine-tuned on matching training data, establishing that web-scale rationale mining provides broadly transferable process supervision that generalizes beyond the specific domains used for extraction.

2. Context and Motivation

The Core Problem: LLMs Reason with Gaps Because They Mimic Their Training Data

The fundamental problem this paper addresses is that LLMs produce incomplete reasoning chains — they skip logical steps, make unstated leaps, and leave crucial rationales implicit. This is not a random failure mode but a systematic consequence of how LLMs are trained. Pre-training data — web text, dialogue, articles — is written for human readers who share common-sense knowledge and cultural context. Authors routinely omit intermediate logical connections because they assume readers will fill them in. Figure 1's Harry Potter example crystallizes this: the text states "Harry used magic outside... He is punished to attend a disciplinary hearing," but the connecting rationale — "When someone breaks the rule, he will be punished" — remains unstated. Human readers infer it effortlessly; LLMs trained to predict next tokens learn to replicate the surface pattern of omission without necessarily learning the underlying logical structure.

This matters because incomplete reasoning chains produce unreliable conclusions. When an LLM skips an implicit rationale during chain-of-thought generation, it can reach a conclusion that superficially resembles its training distribution but lacks the logical grounding that would have been provided by the missing step. The paper's opening example — a model concluding that stealing food for a hungry family will be "rewarded" because it helps the family — is a direct illustration of this failure mode. The model's pre-training data contains narratives where helping family leads to reward, but lacks the countervailing rationale about rule-breaking and punishment that a human would supply. The model hasn't learned to generate that rationale because such rationales are typically left implicit in the text it was trained on.

This gap is both a theoretical problem (it reveals a limitation in how next-token prediction transfers to structured reasoning) and a practical one (it causes errors on reasoning tasks where logical completeness matters: math word problems, scientific inference, legal reasoning, multi-step planning).

Why the Gap Matters: Process Supervision Requires Explicit Rationales

The problem is amplified by the growing importance of process supervision in LLM reasoning. Recent work (Lightman et al., 2023; Snell et al., 2024) has shown that evaluating and guiding individual reasoning steps — rather than just the final answer — produces more reliable reasoning. When a verifier can check whether each intermediate step follows logically from the previous ones, the system can prune incorrect trajectories early and steer toward correct solutions. This approach has demonstrated parameter efficiency advantages: a smaller model with good process supervision can outperform a much larger model with only outcome-based feedback.

But process supervision has a critical dependency: it requires access to good rationales — either as objects to verify (is this step correct?) or as heuristics to guide search (what should the next step be?). If the rationales embedded in or generated from the reasoning model are themselves incomplete — if they contain the same unstated logical leaps as the pre-training data — then the process supervisor is working with degraded information. It cannot verify what isn't stated, and it cannot guide toward connections that haven't been made explicit. The result is that process supervision's potential is capped by the quality of the rationale surface form it operates on.

Real-world impact: As LLMs are deployed for high-stakes reasoning tasks — medical diagnosis support, legal analysis, scientific research assistance, educational tutoring — the completeness and correctness of their reasoning chains directly affects reliability and trust. A system that routinely skips logical steps is harder to audit, harder to debug, and more likely to produce plausible-sounding but incorrect conclusions. Making implicit rationales explicit therefore has downstream consequences for safety, interpretability, and accountability.

Prior Approaches and Where They Fall Short

The paper identifies four broad categories of prior work, each with specific limitations that RATIONALYST is designed to address:

1. Outcome-based verifiers (Cobbe et al., 2021; Snell et al., 2024). These approaches train models to judge the correctness of a complete solution — the final answer and optionally the full reasoning chain — rather than evaluating individual steps. The advantage is simplicity: training data requires only question-answer pairs with correctness labels, which are easier to obtain than step-level annotations. The paper acknowledges this as a strong baseline (Section 5.4 shows RATIONALYST outperforms outcome-supervised verifiers trained on matching data). But outcome-based approaches have a fundamental limitation: they provide no intermediate guidance. The system must generate an entire solution before receiving feedback, which wastes computation on obviously flawed trajectories and provides no mechanism for steering the generation process mid-stream. More importantly, outcome verifiers can be gamed — a model might learn to produce reasoning patterns that correlate with correct final answers without actually encoding sound step-by-step logic.

2. Process-based supervision with human annotation (Lightman et al., 2023; Luo et al., 2024; Wang et al., 2024a). These methods train verifiers that score each reasoning step individually, enabling early pruning and fine-grained guidance. They have demonstrated strong results — Lightman et al.'s PRM achieves substantially better scaling than outcome-based verifiers on MATH. However, the human annotation bottleneck is severe: annotating step-level correctness for thousands or millions of reasoning trajectories requires significant expert labor and cost. Automated approaches like Math-Shepherd (Wang et al., 2024a) reduce but don't eliminate this dependency — they still require ground-truth answers or automatic step verification via execution, which only works for domains with executable formal semantics (code, formal math). For natural language reasoning, commonsense inference, and other "informal" domains, ground-truth step labels depend on human judgment. This makes process supervision expensive to scale across diverse reasoning tasks, and the paper argues this limitation has restricted its adoption to a narrow set of domains.

3. General-purpose verifiers using compositional reasoning (Dhuliawala et al., 2023; Weir et al., 2024, 2023; Vacareanu et al., 2024). Rather than training domain-specific verifiers, these approaches use LLMs themselves (via prompting or light fine-tuning) to verify reasoning through principles like consistency checking, decomposition, or chain-of-verification. The appeal is broad applicability without per-task training. However, the paper argues these methods don't fully leverage available unlabeled data — they treat the reasoning problem as one of clever prompt engineering or decomposition strategy rather than learning from the latent reasoning signals already present in web-scale text. They also inherit the limitations of the underlying LLM's reasoning capabilities: if the verifier model has the same tendency to skip implicit rationales as the generator model, the verification itself may be unreliable. The paper provides evidence for this in Section 5.4, showing that LLaMa-3-8B-Instruct used as a process verifier actually degrades performance slightly compared to the no-verifier baseline (77.6% → 77.4% on GSM8K), which the authors attribute to the model's difficulty in "differentiating among its own generated outputs" — a well-documented self-evaluation problem (Huang et al., 2023; Jiang et al., 2024b).

4. Knowledge extraction from unlabeled data (Schick et al., 2023; Cornille et al., 2024; Zelikman et al., 2024). This line of work addresses the bottleneck of human annotation by mining training signals from the pre-training data itself. Toolformer (Schick et al., 2023) demonstrated that future token prediction can serve as a self-supervised signal for learning when and how to use tools — the model extracts tool-use annotations by checking whether inserting a tool call improves the likelihood of subsequent text. Cornille et al. (2024) applied similar reasoning to learn article planning. Most directly relevant to RATIONALYST is Quiet-Star (Zelikman et al., 2024), which uses a comparable future-text-perplexity signal to extract underlying rationales from everyday text and trains the same model to generate them internally during reasoning.

RATIONALYST fundamentally differs from Quiet-Star in its architectural separation: rather than training the reasoning model to generate rationales for itself (an integrated approach), RATIONALYST trains a separate "supervisor" model that provides rationales as heuristics to any general-purpose agent model. The paper argues this offers "a simpler and more reliable method" because rationale extraction and reasoner training are decoupled. This matters operationally: an integrated approach like Quiet-Star must balance rationale quality against final answer accuracy during training (the same model does both, potentially optimizing shortcuts that improve accuracy but produce opaque rationales — a point the paper makes in Section 5.5 about Quiet-Star's rationales being less human-interpretable). A separated approach lets each component specialize: the supervisor focuses entirely on generating accurate, helpful rationales, while the agent focuses on solving the task using those rationales as guidance.

How RATIONALYST Positions Itself

The paper positions RATIONALYST at the intersection of process supervision and knowledge extraction from unlabeled data, combining the strengths of both while addressing their individual weaknesses:

From process supervision, RATIONALYST takes the goal of providing step-level guidance during reasoning, enabling fine-grained trajectory evaluation and steering. From knowledge extraction, it takes the methodology of mining training signals from unlabeled text using future-token-prediction as a self-supervised objective.

The key contribution framing is data-centric: rather than seeking better verifier architectures, better prompting strategies, or more efficient human annotation pipelines, RATIONALYST asks: can we obtain process supervision signals from the vast amount of unstructured text that already exists, without any human labels? This reframes process supervision from an annotation problem to a data mining problem, which is fundamentally more scalable.

The paper also positions itself as complementary to outcome-based verifiers, prompting techniques, and search strategies. Section 5.4 notes RATIONALYST can be combined with self-consistency, and Section 6 discusses compatibility with test-time compute scaling frameworks like beam search and lookahead search. The claim is not that RATIONALYST replaces these approaches but that it provides a missing component — process-level heuristics derived from web-scale data — that can enhance any architecture that benefits from step-by-step guidance.

A subtle but important positioning choice: the paper explicitly avoids relying on stronger models for rationale extraction (unlike many distillation-based approaches that use GPT-4 as a teacher). The same model family (LLaMa-3-8B-Instruct) is used for rationale extraction, filtering, RATIONALYST training, and agent inference. This means the results reflect what can be achieved through self-supervision alone, without importing capabilities from larger or proprietary models. The authors note (Section 5.1) that using stronger models for extraction would likely improve results — this is framed as an opportunity for future scaling rather than a limitation of the current approach.

Reconciling with Conflicting Evidence

The paper implicitly addresses a tension in the verifier literature: some work finds process supervision highly effective (Lightman et al., 2023), while other work finds LLMs struggle to evaluate their own outputs (Huang et al., 2023; Jiang et al., 2024b). RATIONALYST's design suggests a resolution: the problem isn't process supervision per se, but where the supervision signal comes from. An LLM prompted to "check your work" is being asked to both generate and evaluate simultaneously, which is demonstrably difficult. A model specifically trained on extracted rationales from diverse web data has learned a different skill — identifying logical connections that are typically left implicit — and brings this external knowledge to bear on evaluating steps it didn't generate. The separation of generator and verifier roles, combined with the rationale-specific training, may explain why RATIONALYST succeeds where vanilla LLM self-verification fails.

3. Technical Approach

3.1 Reader Orientation

RATIONALYST is a system that trains a specialized model to generate the hidden logical steps that LLMs typically skip during reasoning, then uses those generated steps as heuristics to guide any general-purpose LLM toward more complete and accurate reasoning chains. The problem it solves is that LLMs trained on web text learn to replicate the omissions in their training data — they skip logical connections that humans leave implicit in everyday communication — which leads to incomplete reasoning and unreliable conclusions; RATIONALYST fixes this by mining those implicit connections from unlabeled text itself, training a dedicated "supervisor" model to surface them at inference time, and using the supervisor's output to score which candidate next steps are most likely to be correct, all without requiring any human annotations of intermediate reasoning steps.

3.2 Big-Picture Architecture (Diagram in Words)

The RATIONALYST system has three major components that operate in sequence (training phase) and then two that interact at inference time:

1. Rationale Extraction Pipeline (§3.1): Takes unlabeled text from both web-scale corpora (The Pile) and existing reasoning datasets (GSM8K, ECQA) as input. Uses an LLM (LLaMa-3-8B-Instruct) to generate candidate implicit rationales — logical connections that bridge what is stated to what follows — and then filters them by checking whether each rationale makes future text more predictable (lower perplexity). The output is a curated dataset of 79k filtered rationales paired with their preceding context.

2. RATIONALYST Training (§3.2): Fine-tunes LLaMa-3-8B-Instruct on the filtered rationale dataset. The model learns to predict, given a context (a document segment or a partial reasoning trajectory), what implicit rationale should connect the context to what comes next. The output is the trained model M_Ra, which specializes in generating logical bridging statements.

3. Inference-Time Supervision (§3.3): During reasoning, a separate agent LLM M_Agent (any general-purpose model) generates candidate next reasoning steps. M_Ra reads the current reasoning trajectory (the question plus all steps generated so far) and produces an implicit rationale. In the implicit supervision mode (the primary approach), a probability estimation model M (the same LLaMa-3-8B-Instruct used during extraction) scores each candidate next step conditioned on both the trajectory and the rationale — essentially asking "how likely is this next step, given that we know this bridging rationale is true?" The highest-scoring candidate is appended to the trajectory, and the process repeats until an answer is reached.

The key architectural insight is the separation of roles: M_Ra (the supervisor) only generates rationales; M_Agent (the reasoner) only generates candidate steps; M (the scoring model) only estimates conditional probabilities. No single model has to both generate and evaluate its own outputs, which circumvents the well-documented self-evaluation problem (Huang et al., 2023; Jiang et al., 2024b). Information flows in a loop: trajectory → rationale → scored candidates → selected step → extended trajectory, with each component specializing in one function.

3.3 Roadmap for the Deep Dive

The technical approach has four sequential stages, and I'll explain them in the order they occur because each stage produces data that feeds the next:

• First, the rationale extraction process from both web-scale data (The Pile) and reasoning datasets (GSM8K, ECQA) — how the model is prompted, what controls ensure quality, and how answer leakage is prevented.
• Second, the filtration mechanism using future-text perplexity — the core self-supervised signal that determines whether a rationale is "good" without human labels, including the mathematical definition of the filtering criterion and the threshold-setting procedure.
• Third, the training procedure for RATIONALYST — what inputs and targets look like, why the model sees different context types from different data sources, and the training objective.
• Fourth, the inference algorithm in detail — the two supervision modes (implicit and explicit), the probability estimation mechanics, the stopping condition, and the computational cost analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data-centric process supervision paper whose core idea is that implicit rationales — logical connections left unstated in everyday text — can be mined from unlabeled data at scale using future-text predictability as a self-supervised training signal, and that a model trained specifically to generate these rationales can provide step-by-step guidance to any general-purpose LLM during reasoning, improving accuracy without requiring human annotations of intermediate steps.

---

Rationale Extraction from Unlabeled Data (§3.1)

What a rationale is in this framework. The paper defines an implicit rationale as a natural language statement that captures the logical connection between what has been stated and what follows. Unlike chain-of-thought steps that explicitly solve the problem (e.g., "Natalia sold 48 / 2 = 24 clips in May"), a rationale articulates the reasoning about those steps — why the next computational step is the right one, what principle justifies the transition, or what intermediate goal is being pursued. In the paper's example from GSM8K (Figure 3, bottom), after computing the number of clips sold in May, the implicit rationale is: "Now we should calculate the sum of chips in April and May." This doesn't solve anything new; it names the next logical operation and justifies why it's needed. Similarly, in the Harry Potter example (Figure 1), the implicit rationale "When someone breaks the rule, he will be punished!" provides the causal principle that connects the action (using magic outside school) to the consequence (disciplinary hearing), something the original text leaves unstated.

Why extraction from unlabeled data is possible. The key enabling observation is that while implicit rationales are not explicitly stated in text, they are predictable from context — and, crucially, having them makes future text more predictable. This creates a self-supervised signal: an LLM can generate candidate rationales at any point in a document, and the quality of each candidate can be assessed by checking whether inserting it before the subsequent text reduces the model's perplexity on that subsequent text. A rationale that genuinely captures the logical connection should make the following words more predictable (lower loss); a spurious or irrelevant rationale should have little effect or make prediction harder. This is the same principle that Toolformer (Schick et al., 2023) used for tool-use annotations and Quiet-Star (Zelikman et al., 2024) used for thinking tokens — RATIONALYST applies it specifically to process supervision rationales.

Pre-filtering the Pile for reasoning-rich documents. The Pile is an 800GB dataset containing diverse web text: Wikipedia, GitHub, StackExchange, PubMed, HackerNews, and CommonCrawl-derived content (Pile-CC). Only a fraction of this text contains the kind of multi-step reasoning that would yield useful rationales. To avoid wasting computation on, say, boilerplate legal text or repetitive social media posts, the paper implements a two-stage pre-filter:

1. Compute an average semantic embedding of representative reasoning training sets using MPNet-base (Song et al., 2020), a paragraph embedding model. The reasoning training sets used for this embedding are not explicitly named but are presumably the training portions of the evaluation benchmarks (GSM8K, ECQA, and possibly others) — the text that exemplifies the kind of reasoning RATIONALYST should learn to supervise.

2. Select documents from The Pile whose cosine similarity to this average reasoning embedding exceeds a threshold α = 0.3. Documents longer than 2000 tokens are excluded to fit within LLaMa-3's context window. The resulting subset contains approximately 266.6k documents from Pile-CC, 21.8k from StackExchange, 19.9k from GitHub, 5.8k from HackerNews, 4.9k from PubMed Central, and 4.2k from Wikipedia (Table 1).

This pre-filtering is a practical necessity driven by compute constraints — processing all 800GB of The Pile would be infeasible — rather than a principled component of the method. The threshold α = 0.3 is not ablated and likely represents a point on the precision-recall curve chosen empirically to yield a manageable number of documents while retaining those with recognizable reasoning patterns.

Rationale generation from pre-training data (The Pile). After pre-filtering, the selected documents are segmented into 2000-token chunks. For each chunk, the model M (LLaMa-3-8B-Instruct) is prompted to insert implicit rationales after each sentence, wrapped in special tokens <BOT> (beginning of thought) and <EOT> (end of thought). The prompt (Appendix B, Figure 6) provides few-shot demonstrations showing the model how to generate rationales that capture the logical flow between sentences. For example, given a mathematical text passage, the model learns to insert rationales like "First solve the equations to find j, then substitute j into g(b) to calculate g(j)" between computational steps.

The generation is done with standard LLM sampling. Because The Pile documents are long (average length exceeding the 2000-token chunk size even after filtering), the model generates multiple rationales per document — an average of approximately 3.2 rationales per document across all Pile subdomains (computed from the "# Rationales" and "# Docs" columns in Table 1: for Pile-CC, 853.2k rationales / 266.6k docs ≈ 3.2).

Rationale generation from reasoning datasets (GSM8K and ECQA). In parallel with the Pile extraction, the paper also extracts rationales from the training sets of two reasoning benchmarks: GSM8K (mathematical word problems, 7,473 training examples) and ECQA (commonsense question-answering with explanations, 7,600 training examples). These datasets were chosen for "complementary coverage of mathematical and commonsense reasoning, respectively" (Section 4.1).

The extraction procedure differs from The Pile case because these datasets have structured question-answer pairs rather than free-form text. For each example (q_i, a_i) in the training set, the model M receives a prompt (Appendix B, Figures 4 and 5) that includes the full question and its ground-truth answer with chain-of-thought reasoning. The model is instructed to generate implicit rationales that connect each reasoning step to the next, wrapped in the same <BOT>...<EOT> delimiters. Unlike The Pile's open-ended generation, the model here is guided by the already-structured reasoning chain — it's identifying the logical bridge between steps that the original answer may have left implicit.

This yields an average of 2.34 rationales per document on GSM8K and 2.58 on ECQA (Appendix G). The key reason for including these reasoning-dataset rationales, despite the paper's emphasis on unlabeled data, is domain adaptation and training stability: the Pile rationales cover diverse reasoning patterns but may not closely match the mathematical and commonsense formats the system will be evaluated on. The 14k reasoning-dataset rationales provide a "seed" of in-domain examples that stabilize training and ensure the model learns patterns directly relevant to the evaluation tasks.

Crucial anti-leakage measure. The paper explicitly states: "we explicitly instruct M to exclude answers from the extracted rationales. This precaution prevents answer leakage in our prompts." This means the rationales extracted from GSM8K and ECQA training sets contain bridging logic ("Now we should calculate the sum...") but do not contain the final numerical answers or the specific computation results. This prevents RATIONALYST from inadvertently memorizing answer patterns that would artificially inflate evaluation performance. The extraction prompt's in-context examples (Figures 4 and 5) demonstrate this: rationales discuss what operation should happen next without revealing what the operation's output will be.

---

Rationale Filtration via Future-Text Perplexity (§3.1, Section C)

The core insight. Not every rationale generated by M will be accurate or helpful. Some may misunderstand the logic, some may be vague, and some may simply restate what's already obvious. The paper needs a filtering mechanism to retain only rationales that genuinely aid in predicting what comes next — since that's exactly what makes a rationale useful for process supervision: it should help anticipate the next correct reasoning step.

The filtration leverages a self-supervised signal: does inserting the rationale before the subsequent text make that subsequent text more predictable? A good rationale should reduce the model's uncertainty about what follows; a bad one should have little effect or increase uncertainty. This is operationalized through a difference in cross-entropy losses.

The filtration loss function. Let a text sequence be x = x_1, ..., x_n, where i is the position after which the rationale r is inserted. The cross-entropy loss for predicting future tokens given the rationale is:

$L_i(r) = -\sum_{j=i}^{n} w_{j-i} \cdot \log p_M(x_j \mid r, x_{1:j-1})$

where M is the same LLaMa-3-8B-Instruct model used for extraction (now in a different role — probability estimation rather than generation), w_{j-i} is a weight that depends on how far token j is from the rationale position, and p_M(x_j | r, x_{1:j-1}) is the model's predicted probability of token x_j conditioned on the rationale and all preceding tokens.

The weight sequence w_k for k = j - i (the distance from the rationale) is defined as decreasing by an exponential factor: each step further away from the rationale position reduces the weight by a factor of 0.9. This means: w_0 = 1, w_1 = 0.9, w_2 = 0.81, w_3 = 0.729, and so on. The intuition is that a rationale should be most predictive of immediately following text, and its predictive value should decay for tokens further away (where other rationales or context may become more relevant).

What it computes: This is the weighted sum of negative log-probabilities over all future tokens after position i, with each token's contribution downweighted by its distance from the rationale. A lower L_i(r) means the model assigns higher probability to the actual future text when the rationale is present in the context.

Why exponential decay weights. The exponential decay encodes a locality assumption: rationales primarily explain the immediately following reasoning or text, and their relevance fades for content that appears much later. Without decay, the filtering would be dominated by distant tokens that the rationale may have little connection to, diluting the signal. An alternative would be uniform weights or a hard cutoff window; exponential decay is a smooth compromise that the paper inherits from prior work on future-token-prediction-based filtering (Schick et al., 2023; Zelikman et al., 2024).

The filtering criterion. The paper doesn't use the absolute loss L_i(r) directly. Instead, it computes the reduction in loss compared to having no rationale:

$L_i = L_i(\varepsilon) - L_i(r)$

where ε represents an empty rationale — i.e., predicting future tokens based only on the preceding context x_{1:j-1} without any inserted rationale. L_i(ε) is the baseline cross-entropy when no rationale is provided.

A rationale is retained if:

$L_i \geq \tau_f$

where τ_f is a filtering threshold. In other words, the rationale must reduce the future-text cross-entropy by at least τ_f compared to having no rationale. If inserting the rationale makes prediction easier (loss decreases), the difference is positive and the rationale passes if it exceeds the threshold. If the rationale makes prediction harder (loss increases), the difference is negative and the rationale is discarded regardless of the threshold.

What this computes operationally: For each candidate rationale at each position, the system runs the language model twice on the subsequent text — once with the rationale prepended and once without — computes the weighted cross-entropy difference, and compares it to the threshold. This is computationally intensive (requiring two forward passes per candidate) but provides an automatic quality assessment with no human involvement.

Why this form rather than alternatives. An alternative would be to train a classifier on human-labeled rationale quality judgments — but that reintroduces the annotation bottleneck the paper is trying to avoid. Another alternative would be to use the rationale's own generation probability under M as a quality signal (keep high-probability rationales). The paper's approach is more principled because it directly measures the property of interest: does this rationale help predict the future? A rationale could be high-probability under the model (a fluent, natural-sounding statement) but irrelevant to what follows; conversely, a rationale could be somewhat awkwardly phrased but capture the exact logical connection needed. The future-text-perplexity criterion optimizes for utility, not just fluency.

Threshold setting procedure. The filtering threshold τ_f is set differently for different data sources (Table 1):

• For GSM8K: τ_f = 1.2, leaving 19.5% of generated rationales (approximately 3,425 retained from 17,566 generated). The threshold was calibrated using 100 manually annotated examples (50 positive, 50 negative) to achieve 95% precision — meaning 95% of filtered rationales are judged as accurate by human annotators. This prioritizes precision over recall, which the paper justifies on the grounds that "it's straightforward to scale up the extraction of rationales from unlabelled data for filtration" — they can afford to throw away many generated rationales to keep only high-quality ones, and they can always generate more.

• For ECQA: τ_f = 0.5, leaving 57.6% of generated rationales (approximately 11,329 retained from 19,669 generated). The lower threshold reflects that ECQA rationales are generally more helpful; the 95% precision criterion is less stringent because more generated rationales are naturally good.

• For The Pile subdomains: τ_f = 0 across all subdomains. This means no filtering beyond the sign check: any rationale that reduces future-text perplexity at all (positive L_i) is retained. The paper cites the diverse composition of The Pile making annotation-based threshold calibration impractical. This is a pragmatic concession — human annotation of 100 examples per Pile subdomain would be expensive and might not generalize across the heterogeneous content. Setting τ_f = 0 keeps more rationales at the cost of potentially including lower-quality ones, which is acceptable given the volume (65k retained from approximately 1.08M generated, a 6% overall retention rate).

What the filtering yields. After filtration, the paper obtains approximately 79k rationales total: about 14k from GSM8K and ECQA combined (3,425 + 11,329 ≈ 14,754), and about 65k from The Pile. The Pile rationales are dominated by StackExchange (29.8% retention, 33.9k rationales) and Pile-CC (2.9% retention, 24.7k rationales), with smaller contributions from HackerNews, Wikipedia, GitHub, and PubMed Central. The high retention rate for StackExchange is attributed (Appendix G) to its question-answering format aligning with reasoning tasks — StackExchange posts already contain explicit reasoning steps, making implicit rationales easier to identify and more predictive of subsequent text.

Why manual annotation was done only for GSM8K and ECQA. The 100-pair annotation sets (50 positive, 50 negative) are only created for the reasoning datasets, not The Pile. This is because the reasoning datasets have a consistent format and reasoning type where "good rationale" has a clear meaning that annotators can agree on. The Pile's heterogeneous content (code, academic papers, forum discussions, encyclopedic articles) makes it much harder to define and annotate rationale quality consistently. The τ_f = 0 approach for The Pile sidesteps this by keeping all loss-reducing rationales, effectively treating "any predictive value" as the quality criterion rather than "human-judged accuracy."

---

RATIONALYST Training (§3.2)

Training objective and formulation. RATIONALYST is trained as a standard supervised fine-tuning (SFT) task. The base model is LLaMa-3-8B-Instruct, and the training data consists of input-target pairs derived from the filtered rationales.

For each filtered rationale, the input is the context that preceded the rationale position: for The Pile data, this is the text segment up to the insertion point; for GSM8K/ECQA data, this is the question and any preceding reasoning steps in the chain. The target is the rationale text itself (without the <BOT> and <EOT> delimiters — these special tokens are part of the formatting that the model learns to generate).

The training objective is the standard per-token cross-entropy loss:

$\mathcal{L}_{\text{train}} = -\frac{1}{|y|} \sum_{t=1}^{|y|} \log p_{M_{\text{Ra}}}(y_t \mid x, y_{<t})$

where x is the input context, y is the target rationale token sequence, y_t is the token at position t in the rationale, and p_{M_{\text{Ra}}}(y_t | x, y_{<t}) is RATIONALYST's predicted probability of token y_t given the context and previously generated rationale tokens.

What it computes: This is the standard autoregressive language modeling loss — exactly the same objective used for pre-training and fine-tuning large language models. The model learns to maximize the probability of the rationale tokens conditioned on the preceding context. By training on filtered, high-quality rationales that demonstrably help predict future text, the model learns to generate rationales that serve the same predictive function.

Why standard SFT rather than preference optimization. The paper notes in its limitations (Section 8) that preference tuning methods like DPO could potentially improve performance by teaching the model to distinguish good and bad rationales, but this is listed as future work. The choice of SFT reflects the paper's emphasis on simplicity and data-centricity: the rationale quality comes from the extraction and filtering process, not from a sophisticated training objective. The filtered rationales are already selected to be helpful (via the perplexity criterion), so the model simply needs to imitate them — a straightforward supervised learning problem.

What the model learns that differs from normal fine-tuning. A normal fine-tuning run on GSM8K would train the model to produce complete solutions — the reasoning steps and the final answer. RATIONALYST's training is fundamentally different because the targets are not the reasoning steps themselves but the meta-commentary about those steps: the implicit rationale that bridges one step to the next. This means RATIONALYST learns to recognize and articulate the logical structure of reasoning rather than to execute the reasoning operations. For example, rather than learning to compute "48 / 2 = 24," it learns to generate "Now we should calculate the number of chips sold in May" — it learns what needs to happen at a planning level rather than how to execute the computation. This is a crucial architectural choice: by separating rationale generation from step execution, the system can apply the same supervision logic across different reasoning domains and different agent models, since the rationales operate at the level of reasoning structure rather than domain-specific computation.

Training hyperparameters and configuration. The paper states explicitly (Section 4.1): "We use the default hyperparameters as specified in the LLaMa-3 technical report (MetaAI, 2024) for fine-tuning." This means the training uses the same optimizer, learning rate schedule, batch size, and other settings as Meta's official LLaMa-3 fine-tuning recipe. The paper does not reproduce these hyperparameters in detail, which is a notable omission for reproducibility, but the reference to the official report provides a pointer. The key architectural detail is that the base model is the Instruct variant of LLaMa-3-8B, which has already been fine-tuned for instruction following — this likely helps RATIONALYST learn to generate rationales in response to the kind of structured contexts it will encounter during inference.

Data mixture and its rationale. The training data combines rationales from all sources: GSM8K, ECQA, and all Pile subdomains. The exact mixture proportions are not specified (which is a limitation for reproducibility), but the total is approximately 79k rationales with roughly 18% from reasoning datasets (14k) and 82% from The Pile (65k). The inclusion of both sources serves different purposes:

• Reasoning dataset rationales provide in-domain examples that closely match the evaluation tasks. They teach RATIONALYST the specific format and reasoning patterns of mathematical word problems and commonsense QA — what kinds of bridging statements are useful for these domains.

• The Pile rationales provide diversity and scale. They expose RATIONALYST to reasoning patterns from code (GitHub), academic discourse (PubMed), technical Q&A (StackExchange), general knowledge (Wikipedia), and informal discussion (HackerNews). This diversity is hypothesized to improve generalization — the model learns that rationales can take many forms and apply across many domains, making it more robust when applied to unseen reasoning tasks like ARC (science) or ProofWriter (logic) that were not in its training data.

The ablation in Section 5.2 (Table 3) confirms both sources matter: removing The Pile rationales causes consistent accuracy drops across all seven benchmarks, with particularly large drops on MMLU-Pro (−4.1%) and ProofWriter (−2.5%), supporting the claim that web-scale data improves generalization.

Post-training validation. The paper reports (Section 4.1) that after training, "we conducted manual annotation of the model's output and found that the accuracy of rationales generated on unseen test data closely matches the filtration accuracy we specified for training data through our filtration parameters." This means that RATIONALYST generates rationales at inference time with roughly the same quality (95% judged as accurate for GSM8K-type patterns) as the filtered training data. This is an important sanity check: it confirms the model has successfully learned the rationale generation task rather than just memorizing training examples, and that the filtering thresholds calibrated on training data transfer to the model's generalization behavior.

---

Inference with RATIONALYST (§3.3, Algorithm 1)

The agent model and its role. During inference, any general-purpose LLM can serve as the agent model M_Agent. In the paper's experiments, this is LLaMa-3-8B-Instruct (the same base model used for extraction and RATIONALYST, but now in a reasoning role with no special fine-tuning for rationales). M_Agent is responsible for the actual problem-solving: it generates candidate next steps in the reasoning chain, conditioned on the current trajectory (the question plus all previously selected steps). It uses temperature 0.7 and top-k sampling with k = 3, meaning for each reasoning step it generates 3 candidate continuations from which the system will select the best one.

The reasoning trajectory. The trajectory T is the accumulating sequence of text that represents the problem-solving process so far. It begins as just the question q (line 1 of Algorithm 1). At each iteration, a new reasoning step is appended to T (line 20), so T grows incrementally: question → step_1 → step_2 → ... → step_n → final answer. This trajectory serves as the input to both M_Ra (which reads it to generate a rationale) and M_Agent (which reads it to generate candidate next steps).

Rationale generation at each step (line 3). At each iteration of the reasoning loop, RATIONALYST M_Ra reads the current trajectory T and generates a single implicit rationale r. This rationale captures the logical connection between where the reasoning is now and what should happen next. For example, if the trajectory currently contains "Michael started with 58 golf balls. After losing 23 on Tuesday, he had 58 - 23 = 35 golf balls," RATIONALYST might generate: "Since Michael only has 35 balls, the next calculation should start from 35, not 58" (as illustrated in Figure 2). This rationale doesn't compute the next step — it identifies the potential pitfall (using the original 58 instead of the updated 35) and thereby constrains what a correct next step looks like.

The two supervision modes. The paper describes two ways to use the generated rationale to guide candidate selection. The key difference is whether the rationale is used as an external scoring criterion (implicit supervision) or as additional context for generation (explicit supervision).

Implicit Supervision (Lines 6, 13)

In implicit supervision, M_Agent generates candidate next steps conditioned only on the trajectory T (line 6) — the rationale is NOT shown to the agent. The candidates are generated as in normal chain-of-thought decoding: the agent extends the trajectory with its best guesses for the next reasoning step, producing N = 3 alternatives (since top-k = 3).

Then, for each candidate next step x_n, the probability estimation model M (the same LLaMa-3-8B-Instruct) computes:

$h = p_M(x_n \mid T, r)$

This is the model's estimated probability of generating candidate step x_n when conditioned on both the trajectory T and the rationale r. In other words: "If a language model knew both the reasoning history AND the implicit rationale connecting the history to the future, how likely would it be to produce this specific next step?"

What this computes operationally: For each candidate, the system runs M in scoring mode (not generation mode) to compute the log-probability of the candidate token sequence given the augmented context [T, r]. This is a single forward pass per candidate that yields a scalar score — the same type of operation as computing perplexity, but here it measures how well the rationale-plus-trajectory context predicts the candidate continuation.

Why this form works as a heuristic. The rationale r captures the logical structure that a correct next step should follow. If the rationale says "the next calculation should start from 35, not 58," then candidate steps that compute "35 - 2 = 33" will have high probability under M conditioned on this rationale, because the model can see the alignment between the instruction (start from 35) and the action (subtracting 2 from 35). Candidate steps that compute "58 - 2 = 56" will have lower probability because they contradict the rationale. The language model M acts as an implicit verifier: it doesn't explicitly check correctness, but its probability estimates naturally favor continuations that are consistent with the rationale, because that's what a well-trained language model does — it assigns higher probability to text that is coherent with its context.

Why "implicit" — the agent never sees the rationale. The rationale influences the selection of the next step (line 19 picks the candidate with highest h) but never appears in the trajectory T itself. The agent generated its candidates without knowledge of the rationale, and the rationale only served as a scoring criterion. This means the trajectory remains clean — it contains only the reasoning steps, not the meta-level commentary — which matters for two reasons: (1) it prevents the trajectory from being cluttered with potentially imperfect rationales that could confuse the agent in later steps, and (2) it keeps the trajectory interpretable for humans reading the final output.

Explicit Supervision (Lines 8, 15)

In explicit supervision, the rationale is temporarily appended to the trajectory before the agent generates candidates. The agent generates M_Agent(T, r) — candidate next steps conditioned on both the trajectory and the rationale. The rationale is then scored using the agent's own probability estimates:

$h = p_{M_{\text{Agent}}}(x_n \mid T, r)$

This is the agent model's own probability of generating each candidate given the rationale-augmented context. The difference from implicit supervision is both who generates (the agent generates with rationale context, rather than generating blind) and who scores (the agent's own probabilities, rather than a separate model M).

Why this form underperforms implicit supervision (Table 4). The paper's manual analysis (Section 5.3) reveals that explicit supervision is less robust to rationale errors. When RATIONALYST generates an imperfect or slightly inaccurate rationale, explicit supervision directly incorporates that flawed rationale into the agent's generation context, which can mislead the agent into producing an incorrect next step. The agent treats the rationale as ground-truth context and reasons from it, so a mistake in the rationale propagates directly into the reasoning. Implicit supervision, by contrast, uses the rationale only as a soft scoring heuristic — even if the rationale is imperfect, it can still provide useful signal (a somewhat relevant rationale will still assign higher probability to better candidates than to clearly wrong ones), and the agent's candidate generation is unaffected because it never saw the potentially flawed rationale.

The stopping condition (line 21). The reasoning loop continues until a stopping condition is met. The condition is dataset-dependent and typically involves detecting a final answer string — e.g., "The answer is:" or "The final answer is:" — which can be specified in the system prompt for M_Agent (Appendix C, Figures 7 and 8). When the generated next step contains this pattern, the trajectory is considered complete. This is a standard approach in chain-of-thought reasoning systems.

Computational cost analysis (Section 5.6). The paper analyzes the inference overhead of RATIONALYST. Using vLLM for deployment with LLaMa-3-8B-Instruct as both agent and RATIONALYST, the time overhead is within 5% of baseline despite requiring two model calls per reasoning step (one for rationale generation, one for candidate scoring). This efficiency is achieved through:

1. Async batched execution: rationale generation and candidate scoring can be parallelized where dependencies allow (the candidates can be generated simultaneously once the rationale is ready, and the scoring of N candidates is trivially parallelizable).

2. Short rationale generation: rationales are typically much shorter than full reasoning steps — often just a sentence or two — so the generation cost per rationale is small.

Memory requirements approximately double when using the same-size model for both roles (two copies of LLaMa-3-8B), but the paper notes this overhead "becomes proportionally smaller with larger agent models" — if the agent is LLaMa-3-70B, adding an 8B RATIONALYST represents a much smaller relative increase in memory and compute.

---

Design Choices and Their Justifications

Why separate models for rationale generation and reasoning. The paper's architecture separates M_Ra (rationale generator) from M_Agent (reasoner) and M (probability estimator), even though in practice all three can be the same underlying model (LLaMa-3-8B-Instruct) with different prompts or roles. This separation is motivated by the well-documented difficulty LLMs have in evaluating their own outputs (Huang et al., 2023; Jiang et al., 2024b): a model that generates a reasoning step and then evaluates that same step suffers from self-consistency biases. By having RATIONALYST generate rationales and M score candidates, the evaluation comes from a different "perspective" — RATIONALYST is prompted to generate bridging logic, not to solve the problem, so its output is not simply the reasoner's own self-assessment. The paper provides evidence for this in Section 5.4: vanilla LLaMa-3-8B-Instruct used as its own process verifier (generating and evaluating steps) actually degrades performance slightly (77.6% → 77.4% on GSM8K), while RATIONALYST — the same base model but fine-tuned specifically for rationale generation — improves it to 81.6%.

Why implicit supervision over explicit. The empirical comparison in Table 4 shows implicit supervision consistently outperforms explicit across both GSM8K and ECQA. The paper's analysis attributes this to error robustness: implicit supervision uses rationales as soft heuristics, so imperfect rationales still provide directional guidance; explicit supervision treats rationales as authoritative context, so errors propagate. This is a design choice that trades off the potential benefit of directly informing the agent (which could produce higher-quality candidates when rationales are perfect) for the robustness of soft filtering (which degrades gracefully when rationales are imperfect). Given that rationale generation is not 100% accurate (the paper reports 95% accuracy matching the filtration threshold), the robustness-first approach is empirically justified.

Why top-k = 3 and temperature = 0.7. The paper uses temperature 0.7 and top-k sampling with k = 3 during inference (Section 4.2). This represents a balance: temperature 0 (deterministic) would produce only one candidate, making the rationale-based selection irrelevant (there's nothing to choose between). Higher k would increase diversity but also increase computational cost (more candidates to score) and potentially introduce low-quality candidates that could confuse the selection heuristic. The choice of 3 candidates creates a manageable decision problem: the rationale needs to discriminate between only 3 alternatives, which is easier than ranking a larger set. The temperature 0.7 introduces enough variability to produce meaningfully different candidates while keeping generation reasonably focused.

Why 95% precision threshold for GSM8K/ECQA filtration. The paper explicitly prioritizes precision over recall in rationale filtering: "Since it's straightforward to scale up the extraction of rationales from unlabelled data for filtration, we prioritize maximizing the precision of our filtered rationales, even if it means extracting fewer of them" (Section 4.1). This choice makes sense given the training objective — RATIONALYST is learning to imitate the filtered rationales, so training on high-precision data (95% accurate) means the model learns a more reliable mapping from context to good rationales. Training on lower-precision data (more rationales but with more errors) might increase diversity but would also teach the model to sometimes generate incorrect rationales, which could degrade inference-time performance given that the approach is sensitive to rationale quality (as shown by the explicit vs. implicit comparison). The volume tradeoff is acceptable because The Pile can generate a large absolute number of high-quality rationales even at low retention rates — 2.9% of 853k is still 24.7k rationales from Pile-CC alone.

Why use the same model for extraction, filtering, training, and inference. The paper uses LLaMa-3-8B-Instruct for every role: it extracts rationales, computes filtering losses, serves as the base for RATIONALYST fine-tuning, acts as the agent during evaluation, and provides probability estimates for candidate scoring. This is a deliberate choice to demonstrate that the approach works through self-supervision alone — no external knowledge from stronger models is injected. The authors note (Section 5.1) that using stronger models (GPT-4, LLaMa-3-70B) for extraction would likely improve results, but the current setup proves the method's viability without such dependence. This is important for two reasons: (1) it shows the approach can work in resource-constrained settings where only one model is available, and (2) it isolates the contribution of the rationale mining methodology from the contribution of using a more capable teacher model (which would confound comparisons with distillation-based approaches).

4. Key Insights and Innovations

Innovation 1: Reframing Process Supervision from an Annotation Problem to a Data Mining Problem

The paper's most fundamental conceptual move is redefining where process supervision signals come from. Prior work on process-based verifiers (Lightman et al., 2023; Wang et al., 2024a; Luo et al., 2024) treats the bottleneck as human annotation: to get step-level correctness labels, you need experts to judge intermediate reasoning — expensive, slow, and domain-specific. Even automated approaches like Math-Shepherd (Wang et al., 2024a) circumvent human labels only by exploiting ground-truth answers or executable code, which restricts them to domains with formal correctness criteria. The implicit assumption across this line of work is that process supervision is fundamentally annotation-constrained: you can have it in narrow domains where you can afford to label steps, but scaling it across diverse reasoning tasks requires solving the annotation cost problem.

RATIONALYST demolishes this framing. The paper's key insight — which distinguishes it from both annotation-based verifiers and from prior rationale extraction work — is that implicit rationales already exist in unlabeled text as latent logical structure, and they can be surfaced through a self-supervised signal (future-text predictability) that requires zero human labels. This isn't an incremental improvement in annotation efficiency; it's a category shift in how the problem is understood. Process supervision becomes a data mining task: the supervision signal is embedded in the web-scale corpora that LLMs are already trained on, and the challenge is extraction and filtering rather than annotation.

What makes this reframing intellectually distinctive is that it inverts the relationship between scale and supervision quality. In the annotation paradigm, more supervision means more human labor — a linear cost that makes broad-domain process supervision economically infeasible. In RATIONALYST's paradigm, more supervision means mining more of the already-existing web text — unlabeled data is abundant, and the extraction process is automated. The 65k rationales from The Pile cost the same to extract whether they cover one domain or a dozen; the only constraint is the presence of reasoning-rich text. This makes process supervision scalable in a way that annotation-based approaches simply are not, and that scalability is what enables the cross-domain generalization demonstrated in Table 2 — RATIONALYST improves accuracy on scientific reasoning (ARC, +3.1%), logical reasoning (ProofWriter, +4.3%), and multi-task reasoning (MMLU-Pro, +5.7%) despite never extracting rationales from those domains.

The comparison to Quiet-Star (Zelikman et al., 2024) sharpens what's novel here. Quiet-Star also mines rationales from unlabeled text using future-text prediction as a self-supervised signal — the extraction mechanism is similar. But Quiet-Star's architectural goal is fundamentally different: it trains the reasoning model to generate rationales for itself as internal "thinking tokens," an integrated approach where the same model both produces and uses rationales. RATIONALYST's architectural separation — training a dedicated supervisor model independent of the reasoner — is not just an implementation detail. It represents a conceptual claim about how rationale knowledge should be operationalized: as a reusable capability that can supervise any reasoner, not as an enhancement that's baked into a specific model. This matters because it decouples rationale quality from reasoner capability — you can improve the supervisor without retraining the reasoner, scale them independently, and apply the same supervisor to multiple reasoners. The paper's evidence for the value of this separation is the poor performance of vanilla LLaMa-3-8B-Instruct as its own process supervisor (Table 5: 77.4% vs. 77.6% baseline), which directly demonstrates the self-evaluation problem that RATIONALYST's architecture circumvents.

Innovation 2: The Implicit Rationale as a Distinct Category of Reasoning Signal

Prior work on reasoning with LLMs has largely conflated two types of text that appear in chain-of-thought outputs: the computational steps that execute reasoning (e.g., "58 - 23 = 35") and the meta-level commentary that explains why a step is taken ("Now we should calculate the sum of chips in April and May"). The dominant paradigm — whether in few-shot prompting (Wei et al., 2022), fine-tuning on reasoning traces (Lewkowycz et al., 2022), or verifier training (Cobbe et al., 2021) — treats both as unified "reasoning steps" and trains models to produce them jointly. The assumption is that a good reasoning model will naturally generate appropriate commentary alongside computational steps.

RATIONALYST makes a category distinction that prior work elides: computational steps answer what to do; implicit rationales answer why to do it. This distinction is not merely taxonomic — it has practical consequences for how reasoning systems should be built. An implicit rationale captures the logical principle, goal, or constraint that justifies the transition between steps, but it doesn't (and shouldn't) contain the step's execution. The paper's anti-leakage measure — explicitly instructing the extraction model to exclude answers from rationales — enforces this separation: the rationale says "calculate the sum" without saying what the sum is.

Why is this distinction important? Because it means rationales are transferable across reasoning instances and domains in ways that computational steps are not. "Now we should calculate the sum" applies to any problem where two quantities have been computed separately and need to be combined; it doesn't depend on the specific numbers or the problem's surface form. This transferability is what enables RATIONALYST to provide useful supervision on Math problems (Table 2: +4.5%) and ARC science questions (+3.1%) using rationales extracted primarily from GSM8K, ECQA, and The Pile — the meta-level reasoning patterns generalize even when the domain-specific computations differ.

The paper's comparison with Quiet-Star on rationale interpretability (Section 5.5) provides concrete evidence for the value of this distinction. Quiet-Star optimizes rationales using final-answer accuracy as a reward signal, which can produce rationales that are effective for task performance but opaque to humans — the model discovers shortcuts or non-intuitive patterns that predict correct answers without articulating understandable logic. RATIONALYST's training objective — imitate rationales filtered by future-text predictability — doesn't optimize for final-answer accuracy directly, which the paper argues produces more human-interpretable rationales. This is a tradeoff: RATIONALYST may sacrifice some task optimization for interpretability. The innovation here is not just generating interpretable rationales (many systems do that) but identifying that the training signal matters for interpretability — optimizing for future-text prediction produces different (and more understandable) rationales than optimizing for final-answer correctness, even when both signals are self-supervised.

Innovation 3: Implicit Supervision as a Robustness Mechanism Against Rationale Errors

The paper's internal comparison between implicit and explicit supervision (Section 5.3, Table 4) reveals something non-obvious: how you use a rationale matters more than whether the rationale is correct. Both supervision modes receive the same rationales from RATIONALYST (which are approximately 95% accurate on GSM8K-type patterns, per Section 4.1). Yet implicit supervision — where the rationale is used only as a scoring signal and never shown to the agent — consistently outperforms explicit supervision, where the rationale is inserted into the agent's generation context.

This is a finding with implications beyond RATIONALYST's specific architecture. The dominant approach in verifier-guided reasoning (Snell et al., 2024; Lightman et al., 2023) is to provide the verifier's output as explicit context or to use it to prune the search space directly — what RATIONALYST calls explicit supervision. The paper's results suggest that when the supervisor is imperfect (which it always will be, at some rate), soft probabilistic guidance outperforms hard contextual injection. The rationale doesn't need to be correct to be useful; it needs to provide enough directional signal to discriminate between candidates. An imperfect rationale that says "the next step should involve subtraction" will still assign higher probability to subtraction-based candidates than addition-based ones, even if the rationale's specific framing is slightly off. Explicit supervision, by contrast, bakes the rationale's errors directly into the reasoning context, where they can cascade.

This insight reframes the problem of building robust reasoning systems. Rather than pursuing perfect verifiers — an asymptotic goal that becomes harder as tasks get more complex — the finding suggests that graceful degradation under verifier error is an architectural property that can be achieved through how supervision signals are integrated. Implicit supervision achieves this by treating the rationale as a prior over candidate quality rather than as ground-truth context. This connects to broader ideas in probabilistic reasoning (soft evidence vs. hard constraints) but applies them specifically to the LLM reasoning context, where the dominant tendency has been toward hard integration of verifier outputs.

The paper's cross-method comparison in Table 5 provides converging evidence for this interpretation. Vanilla LLaMa-3-8B-Instruct used for process supervision — essentially explicit supervision, where the model evaluates its own partial trajectories — degrades performance by 0.2% on GSM8K (77.6% → 77.4%), consistent with prior findings that LLMs struggle at self-evaluation (Huang et al., 2023; Jiang et al., 2024b). RATIONALYST with implicit supervision achieves 81.6% on the same task, a 4.0% improvement over baseline. The gap between -0.2% and +4.0% is not explained by rationale quality alone (both use LLaMa-3-8B as the evaluator); the integration mechanism — implicit vs. explicit, soft vs. hard — is doing substantial work. This makes implicit supervision not just a design choice for RATIONALYST but a generalizable principle for building verifier-guided reasoning systems.

Innovation 4: Web-Scale Rationale Diversity as a Generalization Mechanism

The ablation in Section 5.2 (Table 3) is deceptively simple — removing The Pile rationales from training — but its implications challenge a common assumption in reasoning research. The default approach to improving reasoning on a target task is to train on data from that task or closely related domains: fine-tune on GSM8K to improve math reasoning, fine-tune on ECQA to improve commonsense reasoning. The assumption is specificity beats diversity: reasoning patterns are domain-specific enough that training data should match the target distribution.

RATIONALYST's ablation undermines this assumption. Training on only the 14k reasoning-dataset rationales (GSM8K + ECQA) already provides in-domain supervision that improves performance. But adding 65k rationales from The Pile — covering code, academic text, technical Q&A, encyclopedic content, and forum discussions — provides a consistent additional gain across all seven benchmarks, including those with no relation to The Pile's content. The gains are not uniform: they're largest on MMLU-Pro (−4.1% drop when Pile rationales are removed) and ProofWriter (−2.5%), and smaller but still present on GSM8K (−1.3%) and ECQA (−0.7%). This pattern suggests that web-scale rationale diversity provides something that in-domain rationale training alone does not: exposure to a broader range of logical patterns, reasoning structures, and inference types that transfer across domains.

What makes this intellectually significant is that it identifies diversity of reasoning patterns as a distinct axis for improving process supervision, separate from volume of in-domain data. The Pile rationales are not better than GSM8K rationales in any domain-specific sense — they're extracted from less structured text, with a weaker filtration criterion (τ_f = 0 vs. 1.2 for GSM8K), and likely contain more noise. But their diversity appears to matter. A model trained only on mathematical and commonsense reasoning patterns learns those patterns well but may be brittle when encountering the logical structures of scientific argumentation (ARC) or formal deduction (ProofWriter). The Pile rationales expose the model to reasoning as it appears in code documentation, medical literature, technical explanations, and informal argumentation — a broader distribution of what "logical connection between statements" can look like.

This finding connects to the broader observation in language model training that data diversity improves generalization (evidenced by the success of web-scale pre-training itself), but applies it specifically to the meta-reasoning skill of generating bridging rationales. It suggests that the space of useful rationales is large and that even noisy, imperfect rationales from diverse sources contribute positively to a model's ability to supervise reasoning across domains. This has practical implications: future efforts to build process supervisors should consider not just the quality and quantity of training rationales but their domain diversity, potentially justifying the inclusion of seemingly out-of-domain data that expands the distribution of reasoning patterns the supervisor has encountered.

5. Experimental Analysis

Evaluation Methodology

• Dataset. RATIONALYST is evaluated on 7 benchmarks spanning four reasoning categories: mathematical (GSM8K, MATH), commonsense (ECQA, HellaSwag), logical (ProofWriter), scientific (ARC), and multi-task (MMLU-Pro). For GSM8K, the standard test split is used (1,319 examples evaluated). For MATH, 5,000 test examples across all difficulty levels and topics. For ECQA, the validation split is used due to test-set evaluation server constraints (17,944 examples). For HellaSwag, 10,000 examples from the standard evaluation setup. For ProofWriter, only proofs with depth > 5 are evaluated (600 examples) to assess complex logical reasoning. For ARC, the Challenge subset is used (1,172 examples). For MMLU-Pro, the full 12,000-example evaluation set across all reasoning categories. Each dataset uses few-shot demonstrations calibrated for its domain (8-shot for GSM8K, 5-shot for MATH, 6-shot for ECQA, 4-shot for HellaSwag, 2-shot for ProofWriter, 4-shot for ARC, 5-shot for MMLU-Pro). Source: Table 6 and Appendix A.1.

• Base model(s). The agent model M_Agent is LLaMa-3-8B-Instruct, the official instruction-tuned version of Meta's LLaMa-3 at 8 billion parameters. For RATIONALYST (M_Ra), the base model is the same LLaMa-3-8B-Instruct, fine-tuned on the extracted rationale dataset. For probability estimation during inference (M), the same LLaMa-3-8B-Instruct is used in scoring mode (not generation). This choice means all results reflect self-supervision without importing capabilities from larger or proprietary models — a deliberate design to isolate the contribution of the rationale mining methodology from teacher-model scaling effects. The model family is chosen as representative of current open-weight instruction-tuned LLMs at a scale accessible to academic researchers.

• Metrics. Exact match accuracy is used across all tasks, following standard evaluation protocols. For mathematical tasks (GSM8K, MATH), this means the final numerical answer or expression must exactly match the ground truth. For multiple-choice tasks (ECQA, HellaSwag, ARC, MMLU-Pro), the predicted answer choice (A/B/C/D/E) must match the correct selection. For ProofWriter, the logical conclusion must exactly match the ground-truth entailment judgment. No partial credit is awarded. Accuracy is reported as a percentage of correctly answered evaluation examples.

• Baselines. The paper compares against: (1) No-verifier baseline: LLaMa-3-8B-Instruct with standard few-shot chain-of-thought prompting, no process or outcome supervision — this is the agent model M_Agent reasoning independently without RATIONALYST's guidance. (2) Process supervision with LLaMa-3-8B-Instruct: the same base model prompted to rerank partial reasoning trajectories at each step, using the prompt and in-context examples in Appendix E (Figure 10). The model assigns a reward from 0 to 3 judging the likelihood that a partial trajectory is correct. (3) Process supervision with GPT-4: GPT-4 prompted identically to rerank partial reasoning trajectories, representing a much stronger general-purpose model used for process verification. (4) Outcome supervision with LLaMa-3-8B-Instruct + fine-tuning: LLaMa-3-8B-Instruct fine-tuned on the training sets of each reasoning dataset to classify whether a complete solution (question + full answer) is correct, following the outcome-based verifier approach of Cobbe et al. (2021). This verifier is trained on matching data — i.e., for GSM8K evaluation, the verifier is fine-tuned on GSM8K training data. (5) RATIONALYST without The Pile: an ablation variant trained only on rationales from GSM8K and ECQA (~14k rationales), excluding all Pile-derived data, to isolate the contribution of web-scale rationale extraction.

• Generation budget / compute accounting. The paper measures compute in terms of inference-time overhead relative to the baseline. For each reasoning step, RATIONALYST requires one additional model call (rationale generation) plus scoring of N = 3 candidate next steps. Using vLLM for batched inference, the time overhead is reported as within 5% of the baseline (Section 5.6), achieved through async batched execution and the fact that rationales are typically short. Memory requirements approximately double when using LLaMa-3-8B-Instruct for both agent and RATIONALYST roles, though the paper notes this becomes proportionally smaller with larger agent models. The generation budget is not explicitly sweated — there is no ablation of number of candidates (N=3 is fixed), temperature (0.7 is fixed), or number of reasoning steps. This is a simpler setup than the compute-optimal scaling frameworks in prior work (Snell et al., 2024; Wu et al., 2024) where generation budget is the primary axis of analysis; RATIONALYST focuses on accuracy at a single operating point rather than scaling behavior.

• Cross-validation / statistical protocol. There is no explicit cross-validation or statistical significance testing reported. The paper does not mention standard deviations, confidence intervals, or multiple runs with different random seeds. The reported accuracy improvements (Table 2) are single-point estimates from one evaluation pass. For the ablation comparing RATIONALYST with and without The Pile (Table 3), all datasets are re-evaluated but no statistical protocol is described. This is a notable omission — with test set sizes ranging from 600 (ProofWriter) to 17,944 (ECQA), the reliability of small absolute improvements (e.g., +2.1% on HellaSwag with 10,000 examples) may differ substantially from dataset to dataset. The paper's manual annotation for rationale quality (100 examples split 50/50 positive/negative) is the only place where human-judged quality assessment is used, and this is for threshold calibration rather than result validation.

Main Quantitative Results

RATIONALYST Improves Accuracy Across Diverse Reasoning Tasks (Table 2)

The headline result is that RATIONALYST improves accuracy over the no-verifier baseline by an average of 3.9% across the 7 benchmarks, with individual gains ranging from +2.1% (HellaSwag) to +5.7% (MMLU-Pro). The specific per-dataset results reported in Table 2 are:

These results use implicit supervision with RATIONALYST trained on the full rationale dataset (GSM8K + ECQA + The Pile). The paper emphasizes two patterns in these numbers. First, improvement is not limited to the datasets used for rationale extraction: gains appear on ARC (+3.1%), ProofWriter (+4.3%), HellaSwag (+2.1%), and MMLU-Pro (+5.7%) despite none of these being sources of training rationales. This supports the claim that extracted rationales are "broadly applicable across different reasoning tasks" (Section 5.1). Second, the largest absolute gains appear on the hardest tasks for the baseline model: MMLU-Pro (+5.7%, from a low baseline of 39.6%) and MATH (+4.5%, from 28.0%) benefit more than already-high-performing tasks like ProofWriter (86.4% baseline leaves less room for improvement). This is consistent with the hypothesis that process supervision is most valuable when the agent model's unaided reasoning is unreliable — on easy tasks where the agent rarely makes errors, there are fewer incorrect trajectories for the rationale-based guidance to correct.

However, the paper does not break down performance by difficulty within individual datasets — for instance, we don't know whether MATH gains concentrate in easier subcategories (Algebra, Prealgebra) or harder ones (Precalculus, Number Theory). This limits insight into whether RATIONALYST, like the PRM-based search in Snell et al. (2024), shows difficulty-dependent behavior where benefits vanish on the hardest problems. The MMLU-Pro result is notable because that benchmark is explicitly designed to be more challenging than standard MMLU, with expert-level questions across multiple disciplines — the +5.7% gain suggests RATIONALYST's web-scale rationale training provides broadly useful meta-reasoning patterns that transfer across disciplines.

Web-Scale Rationales Provide Consistent Additional Gains (Table 3)

The ablation removing The Pile from training (keeping only ~14k GSM8K + ECQA rationales) produces a consistent accuracy drop across all seven benchmarks, establishing that web-scale rationale diversity provides benefits beyond in-domain rationale training. The specific drops reported in Table 3 are:

The gains from The Pile rationales are not uniform: they correlate negatively with the availability of in-domain rationale training data. GSM8K (−1.3%) and ECQA (−0.7%) show the smallest drops because RATIONALYST without The Pile was already trained on rationales from their training sets — the Pile adds marginal diversity but the in-domain signal is strong. In contrast, MMLU-Pro (−4.1%), ProofWriter (−2.5%), and ARC (−1.9%) show substantially larger drops because these tasks were never represented in the rationale extraction from reasoning datasets. The Pile rationales act as the primary source of supervision patterns for these out-of-domain tasks, so removing them has outsized impact.

This pattern is the strongest evidence for the paper's claim that "web-scale data likely provides exposure to more diverse reasoning types and content, including specialized knowledge, complex real-world scenarios, and interdisciplinary connections not present in the more focused datasets" (Section 5.2). The MMLU-Pro result is particularly striking: losing The Pile rationales drops accuracy by 4.1 percentage points, meaning the majority of RATIONALYST's benefit on that benchmark (5.7% total improvement over baseline) comes from web-scale rationales rather than from the reasoning-dataset rationales. This is consistent with MMLU-Pro's design as a multi-disciplinary benchmark spanning law, physics, biology, computer science, and other domains — the diverse reasoning patterns in The Pile (StackExchange for technical reasoning, GitHub for procedural logic, PubMed for scientific inference) plausibly provide more relevant supervision than math-and-commonsense rationales alone.

A limitation in interpreting these results: the paper doesn't ablate individual Pile subdomains, so we don't know which components of The Pile (StackExchange? GitHub? Wikipedia?) drive the gains on specific tasks. A subdomain-level ablation would clarify whether the benefit comes from general diversity or from specific subdomains that closely match certain evaluation tasks.

Implicit Supervision Outperforms Explicit Supervision (Table 4)

The paper compares two methods for using RATIONALYST's generated rationales during inference, testing them on the two datasets where rationale extraction was performed (GSM8K and ECQA) to isolate the effect of the supervision mechanism. The results reported in Table 4:

Implicit supervision (rationale used as scoring heuristic; agent never sees it) outperforms explicit supervision (rationale inserted into agent's generation context) by 2.8 percentage points on GSM8K and 2.3 points on ECQA. The paper attributes this gap to error robustness: "When RATIONALYST generates an imperfect rationale, the probability-based heuristic used in implicit supervision can still provide useful guidance even if the rationale itself is not ideal" (Section 5.3). In explicit supervision, an imperfect rationale becomes part of the generation context and can mislead the agent into producing incorrect next steps.

Note the accuracy numbers here differ from Table 2: GSM8K shows 80.3% for RATIONALYST without The Pile in Table 3, which matches the "Implicit Supervision" row here (80.3%), consistent with the ablation using the GSM8K-only model. The full RATIONALYST (with Pile) achieves 81.6% on GSM8K in Table 2. The paper is comparing supervision methods using the GSM8K-only and ECQA-only trained models to avoid confounding with The Pile's additional rationales.

This experiment is important because it validates a key architectural choice. A natural alternative design would be the explicit approach — simply prepend the rationale to the trajectory and let the agent generate conditioned on it. The fact that this underperforms implicit scoring suggests that how the rationale information enters the reasoning process matters as much as the information content of the rationale itself. The paper's manual analysis (Section 5.3) indicates that rationale errors are the key factor: explicit supervision treats rationales as authoritative, implicit treats them as advisory.

The paper does not explore whether the performance gap between implicit and explicit supervision changes with rationale quality. If rationales were 100% accurate, would explicit supervision catch up or surpass implicit? This isn't tested, but the paper's 95% precision threshold and the 2.8-point gap on GSM8K suggest that even a 5% error rate in rationales is enough to make implicit supervision clearly preferable. This has practical implications for verifier-guided reasoning more broadly: soft integration (probability-based scoring) may be inherently more robust than hard integration (context injection) whenever the verifier is imperfect.

RATIONALYST Outperforms Competing Verifiers (Table 5)

Table 5 compares RATIONALYST against several verifier baselines on the two datasets where rationale extraction was performed (GSM8K and ECQA):

Several comparisons are informative:

RATIONALYST vs. vanilla LLaMa-3-8B-Instruct process supervision: The vanilla model used as a process verifier actually degrades performance slightly on both GSM8K (77.6% → 77.4%, -0.2%) and ECQA (72.6% → 71.5%, -1.1%). This is consistent with prior findings that LLMs struggle to evaluate their own outputs — the model is being asked to generate reasoning steps and then judge those same steps' quality, and the verification task appears to be difficult for the untuned model. The paper attributes this to "the model's struggle to differentiate among its own generated outputs, a phenomenon observed in recent studies (Jiang et al., 2024b; Huang et al., 2023)" (Section 5.4). RATIONALYST, by contrast, was specifically trained for the rationale generation and scoring task, and improves performance substantially. This comparison directly validates the paper's architectural choice to separate the supervisor and reasoner roles.

RATIONALYST vs. GPT-4 process supervision: Even without The Pile, RATIONALYST (80.3%) slightly outperforms GPT-4's process supervision (80.0%) on GSM8K, and is essentially tied on ECQA (74.5% vs. 74.7%). With The Pile, RATIONALYST pulls ahead clearly on both (81.6% vs. 80.0% on GSM8K; 76.2% vs. 74.7% on ECQA). This is a notable result: an 8B parameter model fine-tuned on self-extracted rationales outperforms a vastly larger proprietary model in providing process supervision. The paper hypothesizes that "this advantage arises from RATIONALYST's specialized design for providing supervision, in contrast to GPT-4's general-purpose training" (Section 5.4). GPT-4 is a stronger model overall, but it hasn't been specifically trained to generate the kind of bridging meta-rationales that RATIONALYST specializes in — being a better language model doesn't necessarily make you a better process supervisor for a specific reasoning framework. The comparison also operates at different computational costs: GPT-4 inference is substantially more expensive and slower than running a local 8B model.

RATIONALYST vs. outcome-based verifier fine-tuned on matching data: RATIONALYST without The Pile (80.3%) outperforms the outcome-based verifier (79.2%) on GSM8K by 1.1 points, despite the outcome verifier being trained on the exact same GSM8K training data that RATIONALYST's rationales were extracted from. On ECQA, the comparison is closer: 74.5% vs. 74.3%. With The Pile included, RATIONALYST's advantage widens to 2.4 points on GSM8K and 1.9 points on ECQA. The paper attributes this to "the richer feedback provided by process-based supervision compared to outcome-based approaches" (Section 5.4) — providing guidance at each step rather than only at the end enables more fine-grained trajectory steering. This comparison is important because the outcome verifier represents the dominant paradigm in verifier-guided reasoning (Cobbe et al., 2021; Snell et al., 2024), and the process-based approach showing stronger performance with less domain-specific training (RATIONALYST uses extracted rationales rather than final-answer correctness labels) supports the paper's central thesis about the value of process-level supervision signals from unlabeled data. However, this comparison is only on two datasets — whether the advantage holds across the full 7-benchmark suite is not tested, as outcome verifiers were not trained for the other tasks.

What these comparisons collectively show: RATIONALYST occupies a distinctive position in the verifier design space — it achieves better process supervision than a much larger general-purpose model (GPT-4) and better accuracy than an outcome-based verifier trained on matching data, all without requiring human annotations or external teacher models. The key differentiator appears to be the combination of (a) process-level rather than outcome-level supervision, (b) specialized training for rationale generation rather than general-purpose reasoning, and (c) web-scale data for diversity rather than domain-restricted training data.

RATIONALYST Generates Interpretable Rationales on Unseen Data (Section 5.5, Appendix D)

The paper provides qualitative evidence that RATIONALYST's rationales are human-understandable on MATH test set problems, which were not part of the rationale extraction datasets. Appendix D (Figure 9) shows a complete reasoning trajectory with rationales for a system-of-equations problem. The rationales generated are of the form: "Let's first rewrite the second equation to isolate y in terms of x" → "Substitute the expression for y into the first equation" → "We should simplify the equation and solve for x" → "Use the value of x to calculate y" → "Now we can just multiply the values of x and y to find the product." These rationales are readable, logically ordered, and correctly identify the next operation without revealing the computational result — exactly the meta-reasoning behavior the system was designed to produce.

The paper compares these with rationales from Quiet-Star (Zelikman et al., 2024) on the same problems (no examples provided, but described in Section 5.5). The claim is that Quiet-Star's rationales are less interpretable because its training optimizes for final-answer accuracy as a reward, which "might inadvertently develop shortcuts or non-intuitive patterns that optimize for accuracy but not necessarily for clarity or human understanding." This comparison is qualitative — no human evaluation study with multiple annotators or interpretability metrics is conducted, so the claim about superior interpretability remains anecdotal. The paper acknowledges that Quiet-Star's approach "while effective for improving task performance, does not explicitly prioritize human interpretability," framing this as a deliberate tradeoff in RATIONALYST's design.

RATIONALYST Introduces Minimal Computational Overhead (Section 5.6)

The paper reports that using vLLM for efficient deployment, the time overhead of RATIONALYST is within 5% of the baseline despite requiring two model calls per reasoning step. This is attributed to (1) async batched execution, where rationale generation and candidate scoring are parallelized where possible, and (2) short rationale length — rationales are typically much shorter than the reasoning steps they guide. Memory requirements double when using LLaMa-3-8B-Instruct for both the agent and RATIONALYST roles (two 8B models loaded simultaneously), but the paper notes this overhead "becomes proportionally smaller with larger agent models" — e.g., when pairing an 8B RATIONALYST with a 70B agent, the memory overhead is approximately 11% rather than 100%.

Ablation Studies and Robustness Checks

Removing The Pile from training (Table 3): Removing web-scale rationales from training causes consistent accuracy drops across all seven benchmarks, ranging from −0.7% (ECQA) to −4.1% (MMLU-Pro). The largest drops occur on tasks furthest from the reasoning-dataset extraction domains (MMLU-Pro, ProofWriter, ARC), suggesting The Pile's diverse reasoning patterns are most valuable for out-of-domain generalization. This ablation is discussed in detail in the Main Quantitative Results section above.

Implicit vs. explicit supervision (Table 4): Tested only on GSM8K (using the GSM8K-only trained RATIONALYST) and ECQA (using the ECQA-only trained model) to isolate the supervision mechanism. Implicit supervision outperforms explicit by 2.8 points on GSM8K and 2.3 points on ECQA. The paper's manual analysis attributes this to implicit supervision's greater robustness to rationale errors — imperfect rationales still provide useful scoring signal in the implicit approach but can mislead the agent when inserted as explicit context. Discussed in detail in the Main Quantitative Results section above. A missing ablation: the paper does not test whether the implicit/explicit gap changes with rationale quality (e.g., by varying the filtration threshold τ_f to control training rationale accuracy, or by comparing rationales from different extraction sources). This would clarify whether the gap is primarily about error rates or about a fundamental difference in how the two mechanisms use even perfectly accurate rationales.

Combining RATIONALYST with self-consistency (Appendix H): The paper reports additional experiments on GSM8K and MATH using 64-generation self-consistency (majority voting over 64 sampled reasoning chains) both with and without RATIONALYST's guidance. Self-consistency improves accuracy on both benchmarks, and RATIONALYST provides additional gains on top of self-consistency (specific numbers not provided in the main paper; Appendix H states "we observed improved accuracy on both benchmarks with the help of RATIONALYST"). This demonstrates that RATIONALYST is complementary to test-time compute scaling techniques — it provides per-step guidance that can be combined with generation-time diversity strategies. The paper notes it chose arithmetic reasoning benchmarks (GSM8K, MATH) rather than multiple-choice tasks for this experiment because "using self-consistency on multiple-choice tasks could artificially inflate performance."

Filtration threshold calibration via manual annotation (Section 4.1): The choice of τ_f = 1.2 for GSM8K and τ_f = 0.5 for ECQA was calibrated using 100 manually annotated rationale examples (50 positive, 50 negative) to achieve 95% precision. The paper doesn't ablate the threshold — e.g., what happens at 90% precision (higher recall, more training data, potentially noisier) or 99% precision (higher quality, less data)? This leaves open the question of how sensitive the approach is to filtration quality. The Pile uses τ_f = 0 (no filtering beyond the sign check), and the consistent gains from including those unfiltered rationales in Table 3 suggest that even noisy rationales provide useful training signal — but whether the 5% error rate in the GSM8K/ECQA data is optimal or simply conservative isn't explored.

Pre-filtering threshold for The Pile (Section 4.1): The cosine similarity threshold α = 0.3 for selecting reasoning-rich documents from The Pile is not ablated. Changing α would alter the number and composition of documents available for rationale extraction — a higher threshold would select fewer but potentially more reasoning-dense documents; a lower threshold would include more diverse but possibly less relevant text. The paper doesn't explore this tradeoff, and the choice of 0.3 appears to be a pragmatic setting based on available compute rather than a principled optimization.

Number of candidate generations (N=3): The paper uses top-k sampling with k=3 throughout, meaning 3 candidate next steps are generated at each reasoning step. This value is not ablated — e.g., what happens with N=2 (less diversity, faster) or N=5 (more diversity, more computation)? The choice of 3 balances diversity against computational cost, but it's possible that certain tasks benefit more from larger candidate pools (if the correct step is unlikely to appear in the top-3) or that smaller pools suffice for easier tasks. This is particularly relevant because the rationale must discriminate between candidates — if the correct candidate isn't among the generated options, no amount of scoring can recover it.

Model scale: All experiments use LLaMa-3-8B-Instruct for all roles. The paper does not test whether RATIONALYST's benefits scale with model size — would a 70B RATIONALYST provide substantially better supervision? Would the agent benefit more from RATIONALYST if the agent itself were larger? The paper explicitly identifies this as future work (Section 6: "Scaling RATIONALYST with stronger models and increased computational resources is a logical next step"), so the current results establish feasibility at the 8B scale but don't characterize the scaling behavior.

Data mixture proportions: The training data mixes rationales from GSM8K (~3.4k), ECQA (~11.3k), and The Pile subdomains (~65k total). The exact proportions are not specified, and there's no ablation of the mixture ratio. It would be informative to know whether the relative proportions matter (e.g., does the model overfit to GSM8K patterns if GSM8K rationales are oversampled?) and whether the optimal mixture differs for different evaluation tasks.

Critical Assessment

Claim 1: RATIONALYST improves reasoning accuracy by an average of 3.9% across diverse reasoning tasks.

The experiments demonstrate this, but with important scope limitations. The 3.9% average gain is computed over 7 benchmarks that span mathematical, commonsense, scientific, logical, and multi-task reasoning — a reasonably diverse set that supports the "diverse reasoning tasks" claim. The individual gains (Table 2) are all positive and range from 2.1% to 5.7%, so there's no cherry-picking of benchmarks where the method works while ignoring those where it doesn't.

However, several aspects of the evidence merit scrutiny:

Single agent model, single scale. All results use LLaMa-3-8B-Instruct as M_Agent. We don't know whether RATIONALYST would provide similar gains for a different agent model (e.g., a base (non-instruct) model, a model from a different family like Mistral or Qwen, or a model of different scale). The paper claims RATIONALYST can supervise "any general-purpose LLM" (Section 3.3), but this claim is not tested — the agent is the same model family as the rationale extractor and RATIONALYST. There's a plausible risk that RATIONALYST's guidance is calibrated to LLaMa-3-8B-Instruct's specific reasoning patterns and error modes, and would transfer less effectively to a model with different behaviors.

Statistical reliability is unreported. With test set sizes ranging from 600 (ProofWriter) to 17,944 (ECQA) and no standard deviations or confidence intervals reported, we cannot assess whether the improvements are statistically significant or could arise from sampling variance. On ProofWriter (600 examples), a +4.3% absolute improvement represents approximately 26 additional correct answers — a meaningful absolute change, but without knowing the variance of the baseline (which depends on the agent's sampling temperature of 0.7), we can't assess whether this is reliably above noise. The paper would be strengthened by reporting confidence intervals or at minimum running multiple evaluation seeds.

No breakdown by difficulty within datasets. The paper doesn't report whether gains are concentrated on easier or harder problems within each benchmark. This matters because prior work on process supervision (Lightman et al., 2023; Snell et al., 2024) has shown that supervision benefits vary substantially with problem difficulty, and RATIONALYST's own hypothesis — that rationales fill gaps in incomplete reasoning — predicts larger benefits on problems where the agent's reasoning is more likely to be missing steps. MATH, with its difficulty levels from 1 (easiest) to 5 (hardest), would be an ideal testbed for this analysis, but results are reported only in aggregate.

No comparison to simply fine-tuning the agent on more reasoning data. An important missing baseline is: what if, instead of training RATIONALYST as a separate supervisor, you simply fine-tuned the agent model on the same 79k extracted rationales as additional chain-of-thought training data? This would test whether the benefit comes from the architectural separation (supervisor + agent) or simply from having more reasoning examples. The paper briefly addresses this in Section 6 ("The value of fine-tuning on previously encountered text is likely lower than the value of fine-tuning on newly incorporated rationales"), but this is a claim rather than an experimental result.

Claim 2: RATIONALYST outperforms significantly larger verifiers like GPT-4.

Supported on the tested benchmarks, but the comparison has nuances. Table 5 shows RATIONALYST (81.6% on GSM8K) outperforming GPT-4's process supervision (80.0%) by 1.6 points, and RATIONALYST (76.2% on ECQA) outperforming GPT-4 (74.7%) by 1.5 points. These are genuine comparisons — both verifiers are providing process supervision on the same reasoning trajectories from the same agent model.

However, note what's being compared: GPT-4 prompted for process supervision vs. RATIONALYST fine-tuned specifically for rationale generation. This is not GPT-4's best possible performance as a verifier — GPT-4 could potentially be fine-tuned for the same task, or prompted more extensively, or used in a chain-of-verification architecture. The comparison demonstrates that a specialized small model can beat a general-purpose large model on a specific supervision task, which is valuable but doesn't establish that RATIONALYST's methodology (implicit rationale extraction) is what produces the advantage — the advantage could come from task-specific fine-tuning regardless of the rationale extraction aspect.

Additionally, GPT-4 is compared only on GSM8K and ECQA, not the other 5 benchmarks. We don't know whether the advantage holds on ProofWriter, ARC, or MMLU-Pro, where the reasoning patterns are different from the extraction domains.

The comparison to outcome-based verifiers is also limited to two datasets. RATIONALYST outperforms outcome-based verifiers fine-tuned on matching training data on both GSM8K and ECQA (Table 5), but outcome verifiers for MATH, ProofWriter, ARC, or MMLU-Pro were not trained. This limits the strength of the "process supervision beats outcome supervision" claim — it's demonstrated for math word problems and commonsense QA, but not for the full diversity of tasks where RATIONALYST is evaluated.

Claim 3: Web-scale rationale extraction provides broadly transferable process supervision that generalizes beyond extraction domains.

Reasonably well-supported by the ablation evidence, with interpretability caveats. The comparison between RATIONALYST (full) and RATIONALYST (without The Pile) in Table 3 provides direct evidence: removing The Pile rationales causes performance drops on all benchmarks, with the largest drops on the tasks furthest from the GSM8K/ECQA extraction domains (MMLU-Pro: -4.1%, ProofWriter: -2.5%). This demonstrates that web-scale rationales contribute to generalization.

However, two factors limit the strength of this demonstration:

The "generalization" is tested on benchmarks that are still structured reasoning tasks. All seven evaluation benchmarks are standard reasoning datasets with clear questions and answer formats. This is a far cry from the open-ended, implicit-reasoning-in-wild-text scenario that motivates the paper (the Harry Potter example in Figure 1). The claim that RATIONALYST captures "broadly applicable" reasoning patterns is supported for academic reasoning benchmarks, but these benchmarks already contain explicit reasoning chains (in their training sets) — the implicit rationale is being applied to make those chains more complete, not to surface reasoning in genuinely unstructured text. The paper's motivating example (LLM concluding theft will be rewarded) is from an open-ended ethical reasoning scenario that is never tested in the evaluation.

No ablation of individual Pile subdomains. The paper can't say whether the generalization benefit comes from StackExchange (which has Q&A format similar to reasoning tasks), GitHub (which has procedural logic), PubMed (scientific reasoning), or simply from the volume of additional training data (65k more examples, regardless of content). A subdomain ablation would clarify whether content diversity or data volume drives the gains. If StackExchange alone accounted for most of the improvement, the "broadly transferable" claim would be weaker than if gains were spread across subdomains.

The "process supervision" generalization is tested on reasoning, but the rationale extraction is from pre-training text. There's an implicit mismatch: rationales extracted from The Pile are bridging statements for general text continuity (predicting what sentence comes next in a Wikipedia article or GitHub README), but they're being evaluated on their ability to supervise step-by-step mathematical or logical reasoning. The paper doesn't analyze whether the Pile rationales that survive filtration and contribute to training are actually about logical/mathematical reasoning (e.g., StackExchange math posts) or about general discourse coherence (e.g., topic transitions in articles). If the latter, the mechanism of transfer is unclear — how does learning to predict "the next paragraph will discuss methodology" help with "the next step should subtract 2 from 35"?

Claim 4: RATIONALYST generates interpretable rationales without optimizing for task accuracy at the cost of human understanding.

Supported only by qualitative examples, not systematic evaluation. Section 5.5 and Appendix D show examples of RATIONALYST's rationales on MATH, which are indeed interpretable. The comparison with Quiet-Star is described but no Quiet-Star rationales are shown, and no human evaluation study is conducted. The claim that RATIONALYST's rationales are "more human-understandable" rests entirely on the authors' qualitative judgment and the argument that training for future-text prediction (RATIONALYST's objective) is less prone to developing opaque shortcuts than training for final-answer accuracy (Quiet-Star's objective). This argument is plausible but untested — one could equally imagine that future-text prediction could produce rationales that are effective for prediction but not necessarily interpretable (e.g., highly compressed patterns that a language model can use but a human would find cryptic).

A minimal robustness check would involve: (1) showing a human evaluation with multiple annotators comparing RATIONALYST and Quiet-Star rationales for interpretability, correctness, and helpfulness; (2) demonstrating that human readers can use RATIONALYST's rationales to better understand or verify the agent's reasoning (a downstream interpretability task). Without this, the interpretability claim is suggestive but not demonstrated.

Missing experiments that would strengthen the paper

Scaling behavior with compute budget. The paper uses a fixed N=3 candidate next steps and doesn't explore how RATIONALYST's benefits scale with more or fewer candidates. A compute-optimal analysis (similar to Snell et al., 2024; Wu et al., 2024) would reveal whether RATIONALYST's per-step guidance becomes more or less valuable as the generation budget increases — do the gains compound with more candidates (the rationale helps select the best from a larger pool) or saturate (the top-3 already capture the needed diversity)?

Transfer to different agent model families. Testing RATIONALYST with, e.g., a Mistral or Qwen agent would validate the claim that it can supervise "any general-purpose LLM." Without this, we don't know whether RATIONALYST's guidance is model-specific.

Difficulty-stratified analysis. Breaking down MATH, MMLU-Pro, or ARC results by difficulty level would reveal whether RATIONALYST helps most on easy, medium, or hard problems — a crucial characterization for understanding where and when to deploy it.

Direct comparison to fine-tuning the agent on the same extracted rationales. This would isolate whether the benefit comes from the separated supervisor architecture or simply from having more training data with reasoning content.

Ablation of individual Pile subdomains. This would reveal whether the web-scale generalization benefit is about diversity per se or about specific subdomains that happen to match evaluation tasks.

Larger-scale models. The paper explicitly identifies scaling to LLaMa-3-70B or GPT-4 for extraction as future work (Section 6). The current results at 8B represent a lower bound — stronger extraction models would likely produce higher-quality rationales and potentially larger gains.

Summary assessment

The experiments provide solid evidence for the paper's central innovation: implicit rationales can be extracted from unlabeled text using future-text predictability as a self-supervised signal, and a model trained on these rationales can provide useful process supervision that generalizes across reasoning tasks. The 3.9% average improvement across diverse benchmarks, the consistent contribution of web-scale rationales, and the outperformance of GPT-4 as a process supervisor are all genuine results that support the approach's viability.

The evidence is weaker where the paper makes broader claims about interpretability, generalizability to arbitrary agent models, and the precise mechanisms by which web-scale rationales transfer. The evaluation is thorough in breadth (7 benchmarks) but shallow in depth (single operating point, no difficulty stratification, no statistical testing, no model-family transfer experiments, no scaling analysis). The comparisons to prior work are well-chosen but incomplete — in particular, the absence of outcome verifier comparisons on more than two datasets limits the generality of the "process beats outcome" finding, and the absence of a direct comparison to simply training the agent on the same extracted rationales leaves open the possibility that a simpler architecture would achieve similar gains. These limitations do not undermine the paper's contribution but define the boundaries of what has been demonstrated and point to clear directions for follow-up work.

6. Limitations and Trade-offs

The Assumption That Difficulty Estimation Would Be Cheap or Unnecessary

The assumption or constraint. The paper does not estimate, condition on, or adapt to question difficulty. Every prompt receives the same inference procedure: generate a rationale, score N=3 candidates, select the best one, and repeat. The compute-optimal reasoning literature that the paper cites as related work (Snell et al., 2024; Wu et al., 2024) has demonstrated that the effectiveness of process supervision is highly difficulty-dependent — beam search can harm easy problems through verifier over-optimization, sequential revisions help easy problems but not hard ones, and no amount of test-time compute helps on problems outside the base model's capability range. RATIONALYST applies a uniform strategy across all prompts, which means it is almost certainly over-spending computation on some problems and under-spending on others relative to what a difficulty-conditioned allocation would achieve.

The consequence. The paper reports a uniform N=3 candidate scoring approach and a uniform rationale generation at every step. There is no mechanism to skip rationale generation when the agent's reasoning is already clearly on track (wasting computation on easy steps) or to allocate more candidates when the agent is uncertain (under-exploring difficult branch points). The practical consequence is that the reported accuracy improvements may understate what the method could achieve with adaptive allocation — or conversely, the computational efficiency (≤5% overhead reported in Section 5.6) is achieved by applying a fixed, relatively lightweight supervision scheme that may not scale to the harder regime where more expensive supervision would be most needed. The method has no way to dynamically say "this step is critical; let me generate more candidates and score them more carefully."

What evidence exists in the paper. The paper does not analyze difficulty-stratified performance within any benchmark — MATH results are reported in aggregate (32.5% overall), MMLU-Pro in aggregate (45.3% overall), etc. Section 5.6 reports ≤5% time overhead for the uniform procedure, but this number would change under an adaptive scheme. The connection to test-time compute scaling is acknowledged in Section 6 ("RATIONALYST's framework is compatible with more sophisticated test-time compute techniques. Its heuristics can be integrated into existing algorithms like beam-search or lookahead search") but this is framed as future compatibility, not as a current capability or analysis.

Mitigation status. The paper does not address this limitation experimentally or architecturally. The uniform application of RATIONALYST is presented as the method itself, not as a simplification to be relaxed later. Section 6 suggests integration with test-time compute scaling as future work, which implicitly acknowledges that the current fixed-budget approach could be improved. No difficulty estimation mechanism is proposed, and no ablation studies test whether RATIONALYST's benefits vary across difficulty levels — data that would directly inform whether adaptive allocation is worth pursuing.

---

Single Agent Model, Single Model Family — The "Any General-Purpose LLM" Claim Is Untested

The assumption or constraint. The paper explicitly states that during inference "any general-purpose LLM (the 'agent model' or M_Agent) can be employed for reasoning across various problems" (Section 3.3). However, all experiments use LLaMa-3-8B-Instruct as the agent — the same model family used for rationale extraction (M), filtering (M), RATIONALYST training (MRa), and probability estimation during inference (M). This creates a tight coupling between the supervisor and the reasoner that goes beyond both being LLaMa-3-8B: they share the same pre-training distribution, the same instruction-tuning procedure, the same tokenizer, and likely similar failure modes and reasoning patterns. RATIONALYST was explicitly trained to generate rationales that improve future-text prediction under this specific model's probability estimates — the filtration criterion Li = Li(ε) − Li(r) uses LLaMa-3-8B-Instruct as the scoring model M, so a rationale is considered "good" only insofar as it helps LLaMa-3-8B-Instruct predict subsequent text. There is no guarantee that a rationale optimized for LLaMa-3-8B-Instruct's predictive patterns would transfer to a model with different architectural inductive biases, different pre-training data, or different tokenization.

The consequence. If a practitioner deploys RATIONALYST with a non-LLaMa agent — say, Mistral, Qwen, Gemma, or a fine-tuned domain-specific model — the guidance quality may degrade in unknown ways. The rationales were selected for their ability to reduce perplexity under LLaMa-3-8B-Instruct; a different model may assign different probabilities to candidate continuations given the same rationale, potentially ranking candidates differently or failing to benefit from the same meta-reasoning patterns. More subtly, the implicit supervision mechanism (line 13 of Algorithm 1) uses LLaMa-3-8B-Instruct as M to estimate p_M(x | T, r) — if the agent is a different model, the scoring distribution and the generation distribution are mismatched. The rationale scores candidate next steps based on what LLaMa-3-8B-Instruct finds probable, but the selected step is executed by a different model that may not share those probability judgments. This is a form of distribution shift between the scoring model and the generation model that the paper does not characterize.

What evidence exists in the paper. The paper contains no experiments using a non-LLaMa agent model. The claim "any general-purpose LLM" is asserted in Section 3.3 without experimental support. All baselines in Section 5.4 also use LLaMa-3-8B-Instruct as the agent, so the comparisons to GPT-4 process supervision (Table 5) test different verifiers on the same agent, not different agents with the same verifier. The ablation on The Pile (Section 5.2) varies the supervisor's training data but keeps the agent fixed. The paper acknowledges (Section 8) that "the comprehensiveness of our experiments" is a limitation and plans to "extend our experiments to a broader range of reasoning tasks," but does not specifically mention testing with different agent models.

Mitigation status. Not addressed experimentally. The architectural separation of MRa and M_Agent is presented as a design advantage (Section 3.2: "this focus enables us to implement a simpler and more reliable method, as we don't need to directly integrate rationale extraction with 'reasoner' training"), but the claimed benefit — that the supervisor can work with any reasoner — is not tested. This could be verified with a relatively modest experiment: run the same RATIONALYST checkpoint with 2-3 different agent models (e.g., Mistral-7B, LLaMa-3-8B (non-instruct), and Qwen-7B) on a subset of benchmarks and measure whether the accuracy improvements are consistent across agents.

---

Filtration Relies on a Model That Is Also the Training Target — A Form of Self-Fulfilling Signal

The assumption or constraint. The rationale filtration criterion Li = Li(ε) − Li(r) ≥ τf uses LLaMa-3-8B-Instruct (called M in this role) as the model that estimates future-text probability. This is the same model that is subsequently fine-tuned to become RATIONALYST (MRa). The paper states this explicitly: "The same M is subsequently used to train MRa" (Section 3.1). This means the supervision signal for determining which rationales are "good" comes from the very model that will later be trained on those rationales. The filtration doesn't use an external quality signal — it uses the base model's own probability estimates to select rationales, then trains a fine-tuned version of that same model to generate those rationales.

The consequence. This creates a potential self-reinforcing loop where the model selects rationales that align with its own predictive preferences, then learns to generate those same types of rationales. If the base model has systematic weaknesses in predicting certain types of text (e.g., it underestimates the probability of logical connectives, or overestimates the probability of narrative transitions), the filtration will systematically prefer rationales that exploit those model-specific patterns rather than rationales that are genuinely helpful for reasoning. A rationale that reduces LLaMa-3-8B-Instruct's perplexity may do so for model-specific reasons (the rationale matches the model's learned surface statistics) rather than for task-general reasons (the rationale captures genuine logical structure). When the model is then trained to imitate those rationales, it learns to reproduce patterns that were selected because they were easy for this specific model to leverage for prediction, which may not correspond to patterns that would help a different model or even the fine-tuned model itself after training. This is analogous to the "reward hacking" or "over-optimization" problem in RLHF, but at the data curation stage: the selection criterion is aligned with the model's own biases.

What evidence exists in the paper. The paper provides indirect evidence that there may be some robustness to this concern: the rationale quality after training "closely matches the filtration accuracy we specified for training data through our filtration parameters" (Section 4.1), meaning the model learns to generate rationales that human annotators would judge as similarly accurate. However, human judgment of rationale quality may not correlate with actual usefulness for guiding reasoning — a rationale can be logically correct (passing human evaluation) but suboptimal for the scoring task (failing to discriminate between correct and incorrect candidates). The performance gains in Table 2 demonstrate that the rationales are useful overall, but don't establish whether a filtration criterion using a different (or stronger) model would produce rationales that are more useful.

Mitigation status. The paper does not address this as a limitation or compare alternative filtration models. The choice to use the same model for extraction, filtration, and training is presented as a deliberate design demonstrating self-supervision (Section 5.1: "our results do not leverage external knowledge from stronger models"), but the potential circularity is not discussed. An obvious robustness check — comparing rationales filtered by LLaMa-3-8B-Instruct with the same rationales scored by a held-out model (e.g., LLaMa-3-8B base, or Mistral-7B) — would characterize how much the filtration is model-specific, but this is not performed. The paper positions using stronger models for extraction/filtration as a future scaling opportunity (Section 6), which implicitly acknowledges that the current self-supervised signal could be improved upon, but does not frame the self-reinforcing nature of the current setup as a limitation.

---

The Pile Extraction Process Is Not Shown to Be Necessary for the Reported Gains — Volume vs. Diversity Is Unresolved

The assumption or constraint. The paper's central methodological contribution is extracting rationales from web-scale unlabeled data (The Pile). The ablation in Section 5.2 (Table 3) shows that removing The Pile rationales from training reduces accuracy across all benchmarks, which the paper interprets as evidence that "web-scale data likely provides exposure to more diverse reasoning types" (Section 5.2). However, the ablation conflates two variables: data diversity (The Pile contains qualitatively different reasoning patterns than GSM8K/ECQA) and data volume (The Pile contributes ~65k rationales, approximately 4.6× the ~14k from reasoning datasets). The performance drop when removing The Pile could be partly or entirely due to the reduction in total training data, rather than to the specific content of The Pile rationales. The paper does not include a control where the reasoning-dataset rationales are upsampled to match The Pile's volume (~79k total from GSM8K/ECQA alone), which would isolate the diversity effect from the volume effect.

The consequence. If the gains from The Pile are primarily a data volume effect, then the paper's narrative about "web-scale rationale diversity" and "broadly applicable" cross-domain transfer is overstated — similar gains might be achievable by simply extracting more rationales from reasoning datasets (or by augmenting the existing reasoning-dataset rationales via paraphrasing or synthetic generation) without the complexity of mining web-scale unlabeled text. This matters practically because The Pile extraction is expensive: it requires pre-filtering 800GB of text via semantic embedding similarity, generating rationales for 323k documents across 6 subdomains, and then filtering the resulting ~1.08M candidate rationales down to 65k (a 6% yield). If a simpler procedure on in-domain data achieves comparable results, the method's complexity budget is poorly allocated. It also matters scientifically: the paper's claim to have discovered that "implicit rationales in pre-training data can be leveraged" depends on The Pile providing unique value beyond what in-domain rationale data would provide at equivalent volume.

What evidence exists in the paper. The pattern of drops in Table 3 is suggestive but not dispositive. The largest drops occur on tasks furthest from GSM8K/ECQA domains (MMLU-Pro: -4.1%, ProofWriter: -2.5%), which is consistent with a diversity explanation — The Pile provides supervision patterns for logical and multi-disciplinary reasoning that GSM8K/ECQA don't cover. The smaller drops on GSM8K (-1.3%) and ECQA (-0.7%) are consistent with a volume explanation — these tasks already have in-domain rationales, so additional data (whether diverse or not) provides diminishing returns. However, without a volume-matched control, neither interpretation can be confirmed. The paper does not report per-subdomain ablation of The Pile, nor does it analyze the content overlap between The Pile rationales and the evaluation benchmarks to determine whether specific subdomains (e.g., StackExchange for MATH, GitHub for ProofWriter) are driving the cross-domain transfers.

Mitigation status. Not addressed. The ablation in Table 3 is presented as evidence for web-scale rationale diversity, but the volume confound is not acknowledged or controlled. The paper's Appendix G notes that The Pile's StackExchange subdomain "retains the highest percentage of rationales, likely due to its question-answering format aligning well with our reasoning tasks and containing more inherent reasoning" — which actually undercuts the diversity argument by suggesting that the most useful Pile subdomain is the one that most resembles the evaluation data. A simple volume-matched experiment (upsampling GSM8K/ECQA rationales to 79k total, or downsampling The Pile to 14k) would substantially strengthen or qualify the diversity claim, but this is left to future work.

---

The 3.9% Average Improvement Is Reported Without Statistical Characterization — Reliability at Small Test Set Sizes Is Unknown

The assumption or constraint. The paper reports accuracy improvements as single-point estimates from one evaluation pass, with no standard deviations, confidence intervals, or multi-seed evaluations. The test set sizes vary widely: from 600 examples (ProofWriter, depth > 5 subset) to 17,944 (ECQA, validation split). The agent model uses temperature 0.7 and top-k=3 sampling during inference (Section 4.2), which introduces stochasticity — different runs with different random seeds will produce slightly different accuracy numbers even with the same RATIONALYST checkpoint. The paper does not report how many evaluation runs were performed or how much variance exists across runs.

The consequence. For the smallest test set (ProofWriter, 600 examples), the reported +4.3% improvement (86.4% → 90.7%) represents approximately 26 additional correct answers. With stochastic sampling at temperature 0.7, the variance of the baseline accuracy could easily be ±1-2 percentage points across runs, meaning the true improvement might be anywhere from ~2% to ~6%. For larger test sets (ECQA, 17,944 examples), the +2.6% improvement (72.6% → 75.2%) represents ~466 additional correct answers — a large absolute number where sampling variance is less of a concern, but still unknown without variance characterization.

This matters particularly for the paper's comparisons where the margins are small: RATIONALYST outperforms GPT-4 process supervision by 1.6 points on GSM8K (81.6% vs. 80.0%) and by 1.5 points on ECQA (76.2% vs. 74.7%). On the full RATIONALYST vs. RATIONALYST without The Pile comparison (Table 3), the drops of -0.7% (ECQA), -1.1% (MATH), and -1.2% (HellaSwag) could plausibly be within the noise floor if variance is non-trivial. Without significance testing, these comparisons are suggestive rather than conclusive, and a practitioner trying to decide whether The Pile extraction is worth the computational cost cannot assess whether the 1.1-point gain on MATH (for example) is reliably above the run-to-run variance.

What evidence exists in the paper. The paper provides no variance estimates, no confidence intervals, no mention of multiple evaluation seeds, and no statistical tests. The evaluation methodology (Appendix A) specifies the number of few-shot demonstrations per dataset and the temperature/top-k settings, but does not mention the number of evaluation runs or any procedure for estimating uncertainty. The manual annotation for filtration threshold calibration (Section 4.1) uses 100 examples — the only place where any form of statistical thinking (setting a threshold to achieve 95% precision) is applied, and even there, the confidence interval around the 95% precision estimate given 100 samples is not reported.

Mitigation status. Not addressed. This is a standard reporting practice in much of the LLM evaluation literature (single-point accuracy estimates without variance characterization are common), but it is particularly consequential here because (a) the method introduces additional stochasticity through temperature sampling and top-k selection, (b) some test sets are small enough that variance could meaningfully affect the interpretation of results, and (c) several of the paper's key comparative claims rest on margins of 1-2 percentage points. Reporting 95% confidence intervals (via bootstrap or multi-seed evaluation), or at minimum noting the standard deviation across 3-5 runs, would substantially strengthen the reliability of the reported improvements.

---

RATIONALYST's Gains Are Not Compared to Simply Fine-Tuning the Agent on the Same Extracted Rationales

The assumption or constraint. The paper's architecture separates the supervisor (MRa, trained on extracted rationales) from the reasoner (M_Agent, a general-purpose LLM). This separation is the core architectural claim — RATIONALYST provides "process supervision for any 'reasoner'" (Section 2) rather than baking the rationale knowledge into the reasoner itself. However, the paper never tests whether this separation is actually necessary to achieve the reported gains. An obvious alternative design would be to fine-tune the agent model directly on the same 79k extracted rationales — i.e., train the agent to generate both chain-of-thought steps and meta-level rationales (or better chain-of-thought steps informed by the rationales) and then use standard decoding. This would be a simpler system: no separate supervisor model, no two-model inference procedure, no rationale generation and candidate scoring loop. If this simpler approach achieved comparable accuracy improvements, the architectural complexity of the supervisor-agent separation would be unjustified.

The consequence. If the gains from RATIONALYST come primarily from having access to the information in the 79k rationales (via training) rather than from the online process supervision mechanism (generating rationales dynamically during inference), then the method's claimed innovation — a separate supervisor providing step-by-step guidance — is partially misattributed. The benefit would be from data augmentation (having more and more diverse training data with reasoning content) rather than from architectural innovation (the supervisor-agent separation). This matters for deployment: a model fine-tuned on the rationales and then used with standard chain-of-thought decoding would have simpler infrastructure requirements (one model instead of two), lower latency (no extra rationale generation and scoring steps), and lower memory usage (one model loaded instead of two). A practitioner choosing between approaches needs to know whether the online supervision provides value beyond what offline training on the same data would achieve.

What evidence exists in the paper. The closest the paper comes to this comparison is in Section 6 ("Is training on extracted rationales necessary?"), where it argues: "The value of fine-tuning on previously encountered text is likely lower than the value of fine-tuning on newly incorporated rationales. Second, implicit rationales encapsulate the reasoning process. Pre-training on these rationales enhances reasoning more effectively than focusing on the whole document." These are arguments, not experimental results. The paper does not report accuracy for an agent model fine-tuned on the 79k rationales as additional chain-of-thought training data. The outcome-based verifier comparison (Table 5) is different — that verifier is trained to judge complete solutions (question + answer → correct/incorrect), not to generate better reasoning steps. The process supervision baselines (LLaMa-3-8B-Instruct and GPT-4) are prompted, not fine-tuned. So there is genuinely no baseline where the same data used to train RATIONALYST is instead used to directly improve the agent.

Mitigation status. Not addressed experimentally. The paper's argument in Section 6 is theoretical, and the limitation section (Section 8) does not mention this missing comparison. Given that the rationales are extracted and filtered at significant computational cost, and that the paper claims value comes specifically from how they are used (process supervision rather than outcome supervision or data augmentation), demonstrating that the architectural choice matters — rather than just the data — is a high-priority experiment that would substantially strengthen the paper's central thesis. The fact that it is absent is a significant gap in the experimental validation of the approach's core design principle.

7. Implications and Future Directions

How This Work Changes the Landscape

RATIONALYST shifts the framing of process supervision for LLM reasoning from an annotation-constrained problem to a data-mining opportunity. Prior to this work, the dominant narrative — established by Lightman et al. (2023) and reinforced by subsequent PRM work — was that process supervision requires human judgment of intermediate steps, making it expensive to scale across domains. Even automated approaches like Math-Shepherd (Wang et al., 2024a) circumvented human labels only by exploiting ground-truth final answers or executable code, restricting process supervision to domains with formal correctness criteria. The implicit assumption across this line of work was that no human labels means no process supervision — you either pay the annotation cost or you don't get step-level guidance.

RATIONALYST breaks this assumption not by finding a cheaper way to get human labels, but by identifying a different source of supervision entirely: the implicit logical connections that already exist in unlabeled web text. The core methodological move — using future-text perplexity reduction as a self-supervised signal for rationale quality — demonstrates that the information needed for process supervision is latent in the pre-training data that LLMs already consume. The 79k rationales extracted from The Pile and reasoning datasets without any human step-level annotations, and their demonstrated ability to improve accuracy across 7 benchmarks (average +3.9%, Table 2), establishes that process supervision can be scalable in a way that annotation-based approaches cannot match, because unlabeled text is effectively unlimited while human annotation is linear in cost.

This is not a paradigm shift — it doesn't overturn the value of human-annotated process supervision or render PRMs obsolete. It is better characterized as opening a parallel track: process supervision can now be pursued through two complementary routes. The human-annotation route (Lightman et al., 2023; Luo et al., 2024) remains valuable for domains where precision matters most and annotation budget is available. The data-mining route (RATIONALYST) becomes viable when annotation budget is constrained, when broad domain coverage is needed, or when the target reasoning patterns are well-represented in available unlabeled text. The two routes are not mutually exclusive — rationales extracted via RATIONALYST's method could seed or augment human-annotated PRM training data, combining the scale of automated extraction with the precision of human verification.

The work also provides a reconciliation mechanism for a tension in the verifier literature. Multiple studies have found that LLMs struggle to evaluate their own outputs — Huang et al. (2023) showed that LLMs "cannot self-correct reasoning yet," Jiang et al. (2024b) demonstrated self-refinement difficulties, and the current paper replicates this: vanilla LLaMa-3-8B-Instruct used as its own process supervisor degrades accuracy by 0.2% on GSM8K (Table 5, 77.6% → 77.4%). Yet process supervision as a concept clearly works — Lightman et al.'s PRM achieves strong results, and RATIONALYST itself improves accuracy by 4.0% on the same task. RATIONALYST's architectural separation explains this contradiction: the problem isn't process supervision; it's self-evaluation. By training a dedicated supervisor model on a different task (rationale generation rather than answer verification) and using its outputs as soft scoring heuristics rather than hard correctness judgments, the system circumvents the self-evaluation problem. The supervisor isn't being asked "is my own output correct?" — a task LLMs demonstrably fail at — but rather "given this bridging rationale, which candidate continuation is most probable?" — a task that the language model's probability estimates are naturally suited for. This reframes the verifier problem from "build a better correctness judge" to "build a better context model," which is a different technical challenge with different solutions.

The work also makes two specific research directions less attractive than they previously appeared. First, the finding that implicit supervision (+2.8 points over explicit on GSM8K, Table 4) consistently outperforms explicit context injection suggests that hard integration of verifier outputs into generation — the dominant paradigm in much verifier-guided reasoning work — may be fundamentally suboptimal when verifiers are imperfect. The robustness advantage of soft probability-based guidance over hard context injection is not specific to RATIONALYST and may generalize to other verifier architectures. Second, the finding that web-scale rationale diversity provides benefits beyond in-domain rationale training (Table 3, drops of 0.7–4.1% when The Pile is removed) suggests that domain-specific verifier training alone — the standard approach of training a verifier on the target task's data — leaves performance on the table. A verifier trained only on GSM8K reasoning patterns will be less effective on ProofWriter or ARC, even if those tasks share underlying logical structures, because the verifier hasn't been exposed to the diversity of ways those logical structures manifest in different domains. This implies that verifier training should prioritize domain diversity over domain specificity, a counterintuitive implication given the standard fine-tuning paradigm where matching the target distribution is considered optimal.

Follow-Up Research This Work Enables

Jointly optimizing rationale quality and supervision mechanism under a test-time compute budget. RATIONALYST uses a fixed procedure: one rationale per step, N=3 candidates, implicit soft scoring. The paper acknowledges compatibility with "more sophisticated test-time compute techniques" (Section 6) but doesn't explore how the per-step budget should be allocated between rationale generation quality, candidate diversity, and scoring thoroughness. A concrete follow-up would implement RATIONALYST's heuristics within the compute-optimal framework of Snell et al. (2024): for a given total generation budget, determine how many reasoning steps should receive rationale supervision, how many candidates should be generated per supervised step, and whether the implicit scoring threshold should be adapted based on the rationale's estimated quality (e.g., via the rationale's own generation probability or the score gap between top candidates). The key experiment is a budget-normalized comparison: does RATIONALYST's per-step guidance compound with beam search (where candidates are scored and the top-k trajectories are pursued) or does the overhead of rationale generation consume budget that would be better spent on wider parallel sampling? MATH would be the natural testbed, since its difficulty stratification enables analyzing whether compute-optimal allocation shifts from rationale-heavy (for medium problems where guidance helps) to rationale-light (for easy problems where the agent is already reliable, or hard problems where no amount of guidance helps).

Scaling the extraction model M to test whether rationale quality is the primary bottleneck. The paper uses LLaMa-3-8B-Instruct for all roles and explicitly notes (Section 5.1) that "future work might change M to stronger models, with the expectation that higher-quality rationales will lead to better performance." This is more than a scaling experiment — it tests whether RATIONALYST's current performance is bottlenecked by extraction quality (the rationales aren't good enough) or by integration architecture (even perfect rationales would provide limited additional benefit given how they're used). A concrete design: extract rationales using LLaMa-3-70B-Instruct or GPT-4o, keep the filtration criterion identical (future-text perplexity under the extraction model), train RATIONALYST-70B on the resulting higher-quality rationales, and compare against the 8B version on the full benchmark suite. If gains are large and roughly proportional to extraction model capability, the bottleneck is extraction quality and the method scales naturally with stronger base models. If gains saturate, the bottleneck is in how rationales are integrated into reasoning (e.g., the implicit scoring mechanism can only extract so much value from a rationale regardless of quality), indicating that architectural improvements to the supervision mechanism should take priority over scaling extraction.

Characterizing the generalization mechanism of web-scale rationales through subdomain ablation and content analysis. The paper's key claim — that The Pile rationales provide "broadly applicable" supervision that transfers across reasoning domains — is supported by the aggregate ablation (Table 3) but the mechanism is opaque. A rigorous follow-up would ablate The Pile by subdomain: train separate RATIONALYST variants each with one Pile subdomain removed (or with only one subdomain included) and measure per-benchmark accuracy. This would answer: does StackExchange (Q&A format, technical reasoning) drive MATH and ARC gains while GitHub (procedural logic) drives ProofWriter gains? Or is the benefit genuinely from diversity per se — no single subdomain matters much, but the aggregate variety across all subdomains produces smooth improvements? A complementary content analysis would embed the extracted The Pile rationales and the evaluation benchmarks in the same semantic space (using the same MPNet-base embedder used for pre-filtering) and measure whether rationales that are semantically closer to a given benchmark's reasoning patterns are more heavily weighted in training (via attention analysis or influence functions). If rationales close to ARC science reasoning patterns predominantly influence ARC performance, the mechanism is nearest-neighbor transfer; if rationales from all subdomains contribute roughly equally regardless of semantic distance, the mechanism is general meta-reasoning skill acquisition. These have different implications for how to select unlabeled data for future rationale extraction — targeted selection of domain-adjacent text vs. maximal diversity sampling.

Direct comparison to fine-tuning the agent on the same extracted rationales to isolate the value of online supervision. This is the most important missing experiment for validating RATIONALYST's core architectural claim. The procedure: take the same 79k filtered rationales, construct training sequences where the agent model learns to generate both the reasoning step AND the implicit rationale (or generates the rationale-conditioned reasoning step), fine-tune LLaMa-3-8B-Instruct on these sequences, and evaluate with standard chain-of-thought decoding (no separate supervisor, no online rationale generation, no candidate scoring). If this achieves comparable accuracy to RATIONALYST's two-model inference procedure, then RATIONALYST's value is primarily in data augmentation — the extracted rationales are useful training data regardless of how they're deployed — and the architectural complexity of online supervision is unnecessary. If RATIONALYST substantially outperforms, then the dynamic, per-step rationale generation provides value beyond what can be baked into the agent's weights, validating the supervisor-agent separation. A nuanced result (online supervision helps on harder problems but not easier ones) would point toward hybrid approaches where the agent handles routine reasoning independently and invokes the supervisor only at uncertain branch points.

Testing transfer to non-LLaMa agent models to determine whether rationale guidance is model-agnostic. The claim that RATIONALYST can supervise "any general-purpose LLM" (Section 3.3) is untested. A straightforward experiment: take the trained RATIONALYST checkpoint, pair it with 3–4 different agent models (Mistral-7B-Instruct, Qwen-7B-Chat, LLaMa-3-8B (non-instruct base model), and optionally a 70B model), and evaluate on the same benchmark suite. Measure not just aggregate accuracy but also the alignment between the scoring model M and the agent: when RATIONALYST scores a candidate as high-probability, does the agent model also assign it high probability? A mismatched scoring-generation pair would manifest as the selected candidate being high-probability under M (LLaMa-3-8B-Instruct) but low-probability under the agent, producing guidance that looks good to the supervisor but leads to poor agent execution. If transfer works well across diverse agent families, the architectural separation is genuinely valuable for deployment flexibility. If transfer is poor, RATIONALYST's guidance is implicitly coupled to LLaMa-3-style models and the method should be applied with matched extraction and agent models.

Preference optimization for rationale quality to test whether the self-supervised signal can be refined. The paper uses standard SFT on filtered rationales and notes in limitations (Section 8) that "we could potentially improve performance through preference tuning (e.g., DPO) where the model learns to distinguish between valid and invalid rationales." This is a well-motivated extension because the filtration process naturally produces paired data: for each position in the text, multiple rationales were generated by M, and the filtration criterion assigns each a score (L_i value). Rationales with high L_i (large perplexity reduction) are "preferred," those with low or negative L_i are "dispreferred." This creates a natural DPO training set without additional annotation: pair high-scoring and low-scoring rationales for the same context, and train RATIONALYST to prefer the former. The key question is whether DPO improves rationale quality beyond what the SFT baseline achieves, and whether improved rationale quality translates to improved downstream reasoning accuracy. A negative result (DPO improves rationale quality ratings but doesn't improve reasoning accuracy) would suggest that the current SFT rationales are already above the quality threshold needed for effective supervision, and that further rationale improvements provide diminishing returns — valuable information for allocating research effort between improving rationales vs. improving how they're used.

Practical Applications and Downstream Use Cases

Cost-efficient process supervision for broad-domain reasoning applications. Organizations deploying LLMs for multi-domain reasoning — educational tutoring platforms covering math, science, and logic; customer support systems handling diverse query types; research assistants that must reason across disciplines — currently face a choice between expensive process supervision (hiring domain experts to annotate step-level correctness for each domain) or outcome-only verification (cheaper but less effective). RATIONALYST offers a third path: extract rationales once from a large unlabeled corpus (or use the publicly released 79k rationales directly), fine-tune a supervisor model, and deploy it across all domains simultaneously. The 3.9% average accuracy improvement across 7 diverse benchmarks, achieved without any domain-specific human annotation of intermediate steps, translates directly to reduced error rates in production. For an educational platform processing 100,000 student queries per day across math, science, and logic, a 3.9% accuracy improvement could mean thousands fewer incorrect answers requiring human review. The ≤5% inference overhead means this improvement comes at minimal additional latency or compute cost — the supervisor can run on the same GPU as the agent with async batching.

Lightweight deployment where larger verifier models are infeasible. The comparison in Table 5 shows that RATIONALYST (8B parameters) outperforms GPT-4 as a process supervisor on GSM8K (81.6% vs. 80.0%) and ECQA (76.2% vs. 74.7%). For applications where using GPT-4 as a verifier is cost-prohibitive, latency-sensitive, or privacy-constrained (GPT-4 requires sending data to OpenAI's API), RATIONALYST provides a locally-deployable alternative that actually achieves better process supervision on the tested benchmarks. A concrete deployment scenario: a healthcare organization building an LLM-powered clinical reasoning assistant that must operate on-premises (no external API calls due to HIPAA). The organization can fine-tune RATIONALYST on extracted rationales from medical literature (PubMed, which is already in The Pile and contributed 18.6k rationales in this paper's extraction), deploy it alongside their agent model, and get process supervision quality exceeding what they'd get from cloud-based GPT-4 — all while keeping patient data within their infrastructure. The memory overhead (one additional 8B model) is manageable for on-premises deployments where a single A100 or H100 already hosts the agent model.

Improving self-improvement and data generation pipelines. Many self-improvement methods (STaR, ReST^EM, rejection sampling fine-tuning) use LLMs to generate training data for themselves, then fine-tune on the generated solutions that pass some correctness filter. These pipelines are bottlenecked by the quality of the generated reasoning traces — if the model generates flawed intermediate steps that happen to lead to a correct final answer (or vice versa), the training data quality degrades. RATIONALYST can be integrated into these pipelines as a generation-time filter: during data generation, use RATIONALYST to score candidate reasoning steps and select the most logically coherent trajectories, producing higher-quality training data even when final-answer verification is noisy or unavailable. The web-scale rationale training means this filtering can work across diverse generation tasks without per-task verifier training. A concrete use case: an organization using LLMs to generate step-by-step solutions for a new domain (say, financial reasoning problems) where ground-truth intermediate steps don't exist. RATIONALYST provides process-level guidance during generation based on reasoning patterns extracted from general web text, improving the quality of the generated solutions that will be used for subsequent fine-tuning. The paper's finding that The Pile rationales improve performance most on out-of-domain tasks (MMLU-Pro: +5.7%, -4.1% when Pile removed) suggests this cross-domain transfer would be particularly valuable for novel domains with no existing reasoning datasets.

Interpretability tool for human-in-the-loop reasoning verification. RATIONALYST's rationales are generated in natural language and are human-readable (Section 5.5, Appendix D). For high-stakes applications where human reviewers must verify LLM reasoning — legal analysis, medical diagnosis support, scientific research — the rationales provide an audit trail that explains not just what the model did at each step, but why it chose that step over alternatives. A human reviewer can examine the reasoning trajectory alongside the generated rationales and quickly identify where the model's logic was sound vs. where it made an unjustified leap. This addresses a key limitation of outcome-based verification: a correct final answer with flawed intermediate reasoning is dangerous because it erodes trust and makes errors harder to catch. With RATIONALYST, the reviewer can see that step 3 was selected because the rationale "the next calculation should start from 35, not 58" correctly identified an alternative candidate (using 58) as logically inconsistent — even if the reviewer isn't a domain expert, they can verify the logical coherence of the reasoning chain. The paper's claim that RATIONALYST's rationales are more interpretable than Quiet-Star's (Section 5.5) is qualitative and needs human-evaluation validation, but if confirmed, this interpretability advantage makes RATIONALYST particularly suited for applications where reasoning transparency is required for regulatory compliance or user trust.