Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems

ArXiv: 2408.16293

🎯 Pitch

Pretraining a language model on data containing deliberate errors followed by corrections—rather than on error-free data alone—boosts its inherent reasoning accuracy from 78% to 95% on the hardest problems, with no cost at inference. Crucially, the ability to correct mistakes cannot be bolted on later via fine-tuning; the model's linear probes can already detect errors with near-perfect accuracy, but only pretraining on correction traces teaches the model to fix them on the fly.

1. Executive Summary

This paper studies whether language models can learn to correct reasoning mistakes immediately during generation—rather than after multi-round prompting—by incorporating retry data (erroneous solution steps followed immediately by [BACK] and corrections) directly into pretraining. Using the synthetic iGSM dataset with GPT2-12-12 models, the authors demonstrate that pretraining on retry data with error rates up to 50% consistently improves reasoning accuracy over pretraining on the same number of tokens from error-free data, with no need for label masking on the mistakes and no increase in the model's tendency to generate errors during inference—on the hardest out-of-distribution problems, accuracy jumps from 78% to 95%. The paper further shows that error correction is a fundamentally different skill from error detection: while models pretrained on error-free data can detect mistakes with 99% accuracy using a lightweight rank-8 probe (enabling retry upon regret), they cannot acquire the ability to correct those mistakes through parameter-efficient fine-tuning like LoRA, irrespective of rank or masking configuration, establishing that retry data must be integrated at the pretraining stage rather than deferred to alignment.

2. Context and Motivation

The Core Problem: Language Models Make Mistakes But Can't Self-Correct in Real Time

Language models have achieved remarkable performance on reasoning tasks, including mathematical problem-solving. Yet even the strongest models—including GPT-4—still make logical errors. The paper opens with a real example (Figure 1) where GPT-4o solves a complex box-stacking weight problem, makes an error (failing to account for a box K), produces a wrong answer, and then, only after being prompted to verify its work, identifies the mistake and produces the correct solution. This sequence captures the central puzzle that motivates the entire paper.

The fundamental asymmetry is this: if a model can correct its mistake when prompted after generation, why didn't it correct the mistake immediately when it first occurred? More precisely, the paper frames this as two interrelated questions:

"(1) Why does it make those mistakes to begin with? (2) Why doesn't it correct the mistakes immediately during generation, instead of waiting until after?"

The first question has received substantial attention in prior work. The paper cites studies showing that many reasoning mistakes arise from distribution shift—the training data differs from the test prompts, so even models trained on perfectly error-free data can produce errors at inference time. But the second question—why models can't catch errors as they happen rather than requiring external prompting—has been far less studied, and this is the gap the paper aims to fill.

Why Immediate Self-Correction Matters: Tokens and Complexity

The practical implications of immediate self-correction are substantial. As illustrated in Figure 1:

• Inference token savings. When a model makes a mistake at step 3 of a 10-step solution and doesn't correct it, it continues generating the remaining 7 steps on a faulty foundation. All those subsequent tokens are wasted computation. If the model could recognize the error immediately—A ⇒B, oh I made a mistake, actually A ⇒C—it would save those tokens and produce the correct answer in a single generation pass.

• Simplified inference architecture. The alternative approach—multi-round prompting where the model generates a complete solution, then is prompted to verify it, then generates a corrected solution—requires calling the model multiple times, managing conversation state, and designing effective verification prompts. If error correction is baked into the model's autoregressive generation, the entire complexity collapses to a single forward pass.

• Alignment with the "general intelligence" aspiration. The paper explicitly references the goal of a single model performing all tasks using the same autoregressive decoding algorithm. Multi-round prompting with external verification breaks this uniformity—it's a scaffolding that compensates for a capability the model lacks internally. Teaching the model to self-correct during generation internalizes that capability.

The Training Data Hypothesis: An Alignment Problem

The paper's central conjecture is that the inability to self-correct during generation is fundamentally an alignment problem—not a capability problem. The reasoning is straightforward:

"There might be little to no training data in the language model's corpus to teach it to correct errors immediately. After all, people are unlikely to write an erroneous sentence followed by immediate correction in internet articles."

When humans write content for the internet—articles, books, forum posts—they typically edit out mistakes before publishing. The final text that enters training corpora is polished and error-free. Even when errors do appear, they are rarely followed by the format [mistake], oh I made a mistake, actually [correction]. So language models are never exposed to the very pattern needed to learn immediate self-correction.

This is a clear and testable hypothesis: if we could provide training data containing errors and immediate corrections, would the model learn to self-correct during generation? The paper's investigation is built around testing this.

The Danger and Uncertainty: Why the Answer Isn't Obvious

Even if we can construct this "retry" training data, is it obviously beneficial? The paper identifies two serious concerns that make the answer non-trivial:

Harm of next-token prediction on mistaken data. Standard autoregressive pretraining minimizes the negative log-likelihood of every token in the training sequence. If the training data contains the erroneous step A ⇒B, the model will be trained to predict A ⇒B when it encounters the preceding context. This could teach the model to generate mistakes, not just correct them.

One might try to mitigate this by label masking—excluding the erroneous tokens from the loss computation so the model isn't rewarded for predicting them. But the paper points out a subtle difficulty: in the retry data format where errors and corrections are interleaved (unlike post-hoc error correction where the entire wrong solution can be masked), separating the error from the correction for masking purposes is non-trivial. The error tokens are embedded within a sequence that also contains the correction and the subsequent correct reasoning.

Unclear benefit over perfectly error-free data. Even if label masking solves the problem of learning mistakes, is there any reason to think that training on A ⇒B, [BACK], A ⇒C is better than training directly on A ⇒C? After all, the goal is for the model to produce correct solutions, and the retry data introduces a distribution shift compared to what we want at test time. If we mask the error portion, we're essentially back to training on error-free data. So where would the benefit come from?

The paper's answer—developed through experiments in Section 4—is counterintuitive: the model doesn't learn to generate more errors; instead, it learns a general error-correction capability that activates only when an error would otherwise occur, and this capability improves overall accuracy beyond what error-free training achieves. But this result is far from obvious a priori, which is why controlled experiments are necessary.

Prior Work and Its Limitations

The paper situates itself within a broader literature on improving LLM reasoning accuracy, but identifies specific shortcomings in existing approaches:

Verifier-based approaches ignore the generation process. Works like Cobbe et al. (2021), Yang et al. (2022), Li et al. (2023), and Zhang et al. (2024) train separate verifier models to score candidate solutions, then select the best one. These approaches improve accuracy but operate after generation—they don't change the model's internal reasoning process. The model still generates mistakes; a verifier just catches them later. The paper's approach of embedding error correction in pretraining aims to change what the model generates in the first place.

Multi-round self-correction is fragile and prompt-dependent. Several works, including Madaan et al. (2024), Weng et al. (2022), and Pan et al. (2023), explore prompting language models to verify and correct their own outputs. The paper's Figure 1 illustrates both the promise and the limitation: GPT-4o can self-correct when prompted, but the correction required an external user saying "Verify your answer. This is not correct." and then "Go over the problem statement sentence by sentence..." The paper notes:

"GPT-4o can insist that its solution is correct and makes no correction (using the 2024-08-06 version with temperature 0)."

This fragility means self-correction through prompting is unreliable in practice—it depends on the specific prompt formulation, the model's willingness to acknowledge errors, and the user's ability to recognize when the model's answer is wrong (which may not be possible in many deployment scenarios).

The gap between error detection and error correction is unexplored. Perhaps most critically for the paper's motivation, prior work had not systematically distinguished between the ability to detect an error and the ability to correct it. The paper's Result 0—drawing on probing work from the companion paper Ye et al. (2024)—shows that models pretrained on error-free data already have near-perfect internal representations for error detection (99% accuracy on the can_next probe). Yet these same models regularly produce errors during generation. This dissociation—"the model knows it made a mistake but can't stop itself from making it"—is the core phenomenon the paper seeks to understand and address.

No controlled study of retry data integration. Prior work on incorporating corrections into training (Qu et al., 2024; Singh et al., 2024) focused on post-hoc multi-turn correction trajectories, not immediate error-within-solution correction. And no prior work had systematically varied retry rate, compared masking strategies, or studied the pretrain-vs-finetune distinction in a controlled setting. The paper fills this gap using synthetic data that allows perfect control over error insertion.

The Synthetic Data Justification

A critical methodological choice is the use of the iGSM dataset from Ye et al. (2024) rather than real math problems or LLM-generated errors. The paper provides a specific justification:

"While it is tempting to use language models such as GPT-4 to synthesize such data, there is no evidence that GPT-4 can robustly generate errors or make those corrections consistently... To avoid complicating our conclusions with the success rate and reliability of GPT-4, we need a different setting where we can 100% reliably generate errors and corrections."

This is important because it frames the paper as a controlled scientific study rather than an engineering demonstration. If the paper had used GPT-4 to generate retry data and found improvements, we wouldn't know whether the improvement came from the retry format itself or from GPT-4's high-quality corrections providing additional training signal. By using synthetic data with programmatically inserted errors (where correctness is guaranteed by the data generation procedure), the paper isolates the effect of the retry format.

The iGSM dataset is specifically designed to isolate logical reasoning by removing arithmetic difficulty (operations are modulo 23) and common-sense knowledge requirements. The authors emphasize:

"We choose this setting because mathematical reasoning errors are among the most widely observed errors made by language models."

And they note that even GPT-4 fails on these problems for larger op values (> 10), confirming the dataset is non-trivially challenging for logical reasoning.

How This Paper Positions Itself

The paper's positioning can be understood along several axes:

It is a "physics of language models" study, not an engineering contribution. The paper explicitly acknowledges:

"We do not claim that the synthetic data used here can directly aid in building future LLMs."

Instead, the goal is to discover guiding principles—what types of data, integrated at what training stage, enable models to acquire specific reasoning capabilities. The synthetic setting provides the control needed to answer these mechanistic questions.

It addresses a specific, underexplored question within the self-correction landscape. While much work asks "can models verify/correct their outputs?", the paper asks the more specific question: "can models learn to correct errors immediately during autoregressive generation, and must this capability be learned during pretraining or can it be added at fine-tuning?"

It connects to broader debates about pretraining vs. fine-tuning. The finding that error correction cannot be acquired through LoRA fine-tuning (Section 5) has implications beyond this specific task. It suggests that certain reasoning capabilities may require the deep weight changes that only pretraining-scale optimization can provide, informing the architectural decisions of future LLM training pipelines.

It anticipates the needs of future LLMs. The paper notes that commercial LLMs already use synthetic data and that "future LLMs are rumored to use Q⋆." The implication is that as models become more capable at reasoning, the bottleneck will shift from "can the model solve this problem?" to "can the model recover from its own rare mistakes efficiently?", and understanding how to teach error correction will become increasingly important.

3. Technical Approach

3.1 Reader Orientation

The "system" in this paper is not a deployed application or an inference-time framework but rather a controlled experimental pipeline for constructing training data, pretraining language models, and evaluating their ability to learn error correction. The core problem it addresses is: can we teach a language model—during pretraining—to recognize when it has made a reasoning mistake and immediately correct that mistake within the same autoregressive generation, rather than requiring external prompting after the fact? The "shape" of the solution is a data-centric intervention: the authors take a corpus of perfectly correct math solutions, programmatically inject errors and corrections into those solutions at controlled rates to create retry data, then compare models pretrained on this retry data against models pretrained on the original error-free data using identical compute budgets.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has four major components:

1. Synthetic problem/solution generator (iGSM dataset). Produces math reasoning problems with step-by-step solutions and a known dependency structure that specifies which parameters can legally be computed at each step. This generator is the data source for all experiments and is not modified by this paper.

2. Retry data construction module. Takes perfectly correct solutions from component 1 and, with probability retry_rate at each solution step, inserts a deliberately wrong parameter (one that cannot legally be computed next) followed by the special token [BACK]. This produces training data where errors and their immediate corrections are interleaved within the solution text.

3. Language model pretraining (GPT2-12-12). Trains a standard autoregressive transformer on either error-free data or retry data, using matched compute budgets (equal total tokens). The model learns through standard next-token prediction—no architectural modifications, no auxiliary losses, and (in the default setting) no label masking on error tokens.

4. Evaluation and analysis suite. Measures test accuracy on in-distribution and out-of-distribution problems; counts how often the model uses [BACK] during generation (retry frequency); counts how many unnecessary operations or parameters appear in solutions; and compares against alternative approaches including beam search, "retry upon regret" (Section 3), LoRA fine-tuning (Section 5), and fake retry data variants (Section 6).

Information flows linearly: the iGSM generator creates problems → the retry data module transforms solutions → the language model is pretrained on the transformed data → the resulting model is evaluated on held-out problems.

3.3 Roadmap for the Deep Dive

• First, the iGSM dataset structure and the specific types of reasoning mistakes it induces, because the retry data construction depends critically on knowing which parameters are "legal" vs. "illegal" at each step.
• Second, the retry data construction procedure—how errors are selected, inserted, and how the retry rate parameter controls the density of mistakes in training data.
• Third, the model architecture and pretraining protocol, including the crucial "equal tokens" comparison framework that makes the retry-vs-error-free comparison fair.
• Fourth, the evaluation methodology and what is being measured, because the paper's claims rest on specific metrics (accuracy, retry count, unnecessary operations) that require careful definition.
• Fifth, the probing-based error detection setup (Result 0) and the "retry upon regret" generation procedure (Result 1), since these provide the baseline demonstrating that error detection is "easy" while error correction requires retry data.
• Sixth, the LoRA fine-tuning protocol used to test whether error correction can be acquired post-pretraining, and the full fine-tuning ("continued pretraining") baseline that shows error correction requires deep weight changes.
• Seventh, the fake retry data construction methods (retry_weak and retry_miss), which are the paper's proposal for how retry-like data could be generated for real-world math problems without requiring semantic understanding.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a controlled empirical study whose core idea is that inserting errors and immediate corrections into pretraining data—at controlled rates and without label masking—teaches language models an error-correction capability that improves reasoning accuracy, and that this capability cannot be acquired through lightweight post-hoc fine-tuning from an error-free pretrained model.

The iGSM Dataset and the Structure of Reasoning Mistakes

The paper builds directly on the iGSM dataset introduced in Ye et al. (2024). Understanding the dataset's structure is essential because the retry data construction depends on knowing exactly which solution steps are legal at each point in the generation.

What iGSM problems look like. Each problem consists of a natural-language description of relationships between entities (e.g., "The number of each Riverview High's Film Studio equals 5 times as much as the sum of...") followed by a question (e.g., "How many Backpack does Central High have?"). The underlying structure is defined by two graphs:

• Structure graph. Describes the hierarchical relationships between entity types (e.g., a School District contains Schools, which contain Classrooms). All parameters mentioned in the problem belong to specific entity types arranged in this hierarchy.

• Dependency graph. Specifies the computational dependencies between parameters. A parameter A can be computed only when all parameters it depends on—its predecessors in the dependency graph—have already been computed. The dependency graph is a directed acyclic graph (DAG), and any valid solution must compute parameters in a topological order.

The key parameter controlling problem difficulty is op, the number of binary operations required to compute the final answer. The paper uses four dataset families:

• iGSM-med_pq/qp: op ≤ 15 for training; op ∈ {20, 21, 22, 23} for OOD testing.
• iGSM-hard_pq/qp: op ≤ 21 for training; op ∈ {28, 29, 30, 31, 32} for OOD testing.

Here, pq means the problem description comes before the question; qp means the question comes before the problem description. The reask variant (e.g., iGSM-med_op=20,reask_pq) modifies the test distribution by re-sampling the queried parameter, which changes which parameters are necessary to compute.

Why arithmetic and common sense are removed. The iGSM dataset restricts arithmetic to integers modulo 23 (so 12 + 13 = 2 in mod-23, for example) and removes common-sense reasoning (e.g., no "a candle burns and its length decreases"). The operations are further broken into binary operations—12 + 13 + 7 is written as (12 + 13) + 7—so the difficulty of the problems lies entirely in determining the correct order of computation, not in arithmetic precision or world knowledge.

The specific type of mistake studied: "skipping steps." The paper identifies the most common reasoning mistake in this setting:

"The most common reasoning mistake occurs when the model generates a [param] that is not yet ready for computation (i.e., the model has not determined the values of all the parameters that [param] depends on, also known as 'skipping steps', a frequent error even in GPT-4)."

Concretely, when generating a solution step by step, each step begins with "Define [param A] as X; so [computation]." The model must ensure that all parameters referenced in the computation of A have already been defined in previous steps. If the model writes "Define Central High's Film Studio as B; so B = p + W = ..." but p and W have not yet been defined, the step is a "skip" error because the model is trying to compute a parameter whose dependencies haven't been satisfied.

The iGSM dataset makes this error type completely verifiable: for any solution step, one can check against the dependency graph whether the parameter being defined is "legal" to compute next. The paper calls this predicate can_next(A) ∈ {true, false}—it is true if all of A's predecessors have already appeared in the solution, and false otherwise.

Why this matters for retry data construction. Because can_next is computable from the dependency graph, the paper can programmatically insert errors into solutions with 100% reliability: at any step, select a parameter A for which can_next(A) = false, insert "Define A as" into the solution text, and we have guaranteed a mistake. This is what enables the controlled experiments that would be impossible with real LLM-generated errors (which are unreliable and inconsistent).

Retry Data Construction

The retry data is the paper's core experimental intervention. The construction procedure transforms a perfectly correct solution into one containing errors and their corrections.

The base solution format. Every iGSM solution follows a rigid step-by-step structure. Each step is a single sentence of the form:

Define [param] as [variable]; so [computation].

For example: "Define Dance Studio's School Daypack as p; so p = 17." The steps appear in dependency order, so by the time a parameter is defined, all its dependencies have been computed.

The retry insertion algorithm. For each solution step, with independent probability retry_rate ∈ [0, 1):

1. Before the correct "Define [param] as" sentence, insert a wrong parameter—a parameter A for which can_next(A) = false at the current point in the solution.

2. Select A uniformly at random from all parameters that cannot be computed next, excluding parameters already appearing in the solution.

3. Insert the text "[param A] as [BACK]" into the solution, where [BACK] is a special token marking the error and indicating a retry.

The process is repeated: after inserting one error, with probability (retry_rate)^2, a second error is inserted at the same position, and so on. This means at a given step with retry_rate = 0.5, there is a 50% chance of one error, a 25% chance of two errors, a 12.5% chance of three errors, etc.

A complete example (retry_rate = 0.5). Figure 4(a) shows a solution where half the steps have errors:

Define Dance Studio's School Daypack as p; so p = 17. Define Film Studio's School Daypack as [BACK]. Define Film Studio's Messenger Backpack as W; so W = 13. Define Central High's Classroom as [BACK]. Define Central High's Backpack as [BACK]. Define Central High's Film Studio as B; so B = p + W = 17 + 13 = 7. ...

The strikethroughs in the figure are for illustration only—the actual training data contains normal English text without formatting markers. The model must learn from the sequence of tokens that a [BACK] means the preceding parameter name was a mistake and that the same position can be retried with a different parameter.

Why the [BACK] token is critical. The [BACK] token serves as an explicit signal that the model should "undo" the preceding attempted parameter definition and try again. Without it, the training data would simply be sequences of correct and incorrect statements interleaved arbitrarily, and the model would have no way to distinguish between "this parameter was incorrect, discard it" and "this parameter was correct, continue." The [BACK] token makes the error-correction structure explicit and learnable.

The retry rate parameter. The paper explores retry_rate values from 0.01 to 0.5. The key property is:

"Within a reasonable range, the more mistakes the better."

A higher retry_rate means the training data contains more examples of error-correction patterns per solution, which improves the model's ability to learn the correction skill. However, it also lengthens each solution (since errors add tokens), which means that for a fixed token budget, the model sees fewer distinct math problems. The retry rate that balances these effects is an empirical question answered in the experiments.

Label masking: masking the error tokens from the loss. A natural concern is that standard autoregressive pretraining—which minimizes the negative log-likelihood of every token—would train the model to produce the erroneous parameter names, since they appear in the training data. The paper tests this by comparing two training configurations:

1. No masking (standard autoregressive): The loss is computed on all tokens, including the erroneous parameter names and the [BACK] tokens. The model is trained to predict "Define [wrong param] as" when it encounters the preceding context.

2. With masking: The loss is not computed on the erroneous tokens (the wrong parameter name and "as"). The model still sees these tokens in the input context (so it can learn from them) but is not penalized for failing to predict them. This is implemented by setting the label positions corresponding to error tokens to the ignore index in the cross-entropy loss.

The paper's finding—that masking is unnecessary—is one of the most striking results:

"Masking mistakes is unnecessary. We observe that it is generally not needed to introduce label masking on the error data even for large retry rate = 0.5."

This means the model, even when trained with standard next-token prediction on error-filled data, does not learn to generate errors. The paper attributes this to the statistical structure of the data: at each solution step, there is still a higher probability (1 - retry_rate) of producing a correct step, and the model is incentivized to prefer correct steps.

The equal-tokens comparison framework. A critical experimental design choice is that all comparisons between retry and error-free pretraining use the same number of training tokens. This means:

• A model pretrained on retry data with retry_rate > 0 sees fewer distinct math problems than a model pretrained on error-free data, because the retry solutions are longer (errors add tokens).

• Within each problem, the retry model sees more tokens—some of which are errors—but across the entire training run, the total number of gradient updates and the total computational cost are identical.

• This makes the comparison directly fair: any accuracy improvement from retry data cannot be attributed to the model simply seeing more tokens or more training steps.

How retry data fits into pretraining sequences. The pretraining data is constructed by randomly generating math problems and their solutions (either error-free or with retry), concatenating them together, and truncating on the right to fit within the context window (768 tokens for iGSM-med, 1024 for iGSM-hard). If a problem-solution pair is longer than the context window, it is discarded entirely. During evaluation, problems are not discarded based on their retry-augmented length—the model is evaluated on the same test distribution regardless of how it was trained—but solutions longer than 768 (or 1024) tokens in their ground-truth error-free form are discarded during evaluation, following Ye et al. (2024).

Model Architecture and Pretraining Protocol

The paper uses a consistent model architecture and training protocol across all experiments, allowing clean comparisons between data conditions.

Model architecture: GPT2-12-12 with Rotary Positional Embedding. The model is based on the GPT-2 architecture but with a modification:

"We use the GPT2 architecture [20], replacing its absolute positional embedding with modern rotary positional embedding [7, 24], still referred to as GPT2 for short."

Specifically, the model is GPT2-12-12: 12 transformer layers, 12 attention heads per layer, and a hidden dimension of 768 (since GPT2 uses 64 × number_of_heads for the hidden size). The authors note they also experimented with the Llama architecture (with gated MLP layers) and saw no benefit, so the simpler GPT2 is used throughout.

Tokenizer and context length. The default GPT2Tokenizer is used. For pretraining, the context length is 768 tokens for iGSM-med and 1024 tokens for iGSM-hard. For evaluation, the context length is extended to 2048 for both datasets, allowing the model to generate longer solutions at test time than what it saw during training—a mild form of length generalization.

Pretraining hyperparameters. The authors adopt the pretraining parameters from Ye et al. (2024) directly, without re-tuning for retry data:

For iGSM-med:

• Optimizer: AdamW with mixed-precision fp16
• Betas: (0.9, 0.98)
• Learning rate: 0.002
• Weight decay: 0.05
• Batch size: 512
• Context length: 768
• Training steps: 100,000
• Learning rate schedule: cosine decay down to 0.01× initial, with 1000 steps of linear ramp-up

For iGSM-hard:

• Optimizer: AdamW with mixed-precision fp16
• Betas: (0.9, 0.98)
• Learning rate: 0.002
• Weight decay: 0.03
• Batch size: 256
• Context length: 1024
• Training steps: 200,000
• Learning rate schedule: cosine decay down to 0.01× initial, with 1000 steps of linear ramp-up

Training data generation. New data is generated on-the-fly throughout training—there is no fixed dataset, no repetition of examples, and no concern about overfitting to specific problems. The authors state:

"We did not limit the amount of training data; we generated new data on-the-fly. We do not explore sample complexity in this paper."

This is important because it means all experiments operate in a regime where the model sees an effectively infinite stream of unique problems, and differences in accuracy reflect differences in what the model learns per token, not differences in how many tokens it needs to memorize a fixed dataset.

Test-time generation parameters. During evaluation, the model generates solutions using either:

• beam=1, dosample=false (greedy decoding): At each step, the model selects the single most likely next token.
• beam=4, dosample=true (beam search with multinomial sampling): The model maintains 4 candidate beams and samples from the probability distribution at each step, selecting the 4 most likely continuations.

For original (error-free) training, additional beam sizes of 16 and 32 are tested. The authors note:

"We discover it is better to keep dosample=false while beam=1 and dosample=true while beam > 1."

This is an empirical finding, not a theoretical claim—greedy decoding works best with a single beam, while multinomial sampling is needed to get diversity when multiple beams are maintained.

The best-accuracy reporting convention. For the main accuracy tables (Figure 4(b), Figure 8), each cell reports the best accuracy across:

• 2 different random seeds for pretraining
• 2 different beam configurations (beam=1 or beam=4)

This means the reported numbers are optimistic relative to any single run, but since the same convention is applied to all data conditions, the comparisons between conditions remain valid.

Evaluation Methodology and Metrics

The paper evaluates models along several dimensions, with accuracy being the primary metric but behavioral statistics (retry counts, unnecessary operations) providing crucial insight into what the model has learned.

Primary metric: solution accuracy. Accuracy is not determined by simply comparing the final answer integer (which ranges from 0 to 22 in modulo-23 arithmetic). Instead:

"We have written a parser to make sure the model's intermediate solution steps are fully-correct."

This means a solution is counted as correct only if:

1. Every step computes a parameter that is legally computable at that point (no skip errors).
2. Every computation is arithmetically correct.
3. The final answer matches the ground truth.

This strict grading eliminates the possibility that the model gets the right answer through a wrong process—a crucial consideration given that the paper's goal is to study reasoning correctness, not answer-matching.

Accuracy statistics. Each cell in the accuracy tables is the average over 4096 math problems of that specific type. So for the op=23 column in iGSM-med_pq, the reported number is the fraction of 4090 problems with op=23 that the model solved completely correctly.

Retry count statistics. The paper measures how often the model uses the [BACK] token during its own generation. This is reported separately for:

• Correct solutions (Figure 5(a)): Among solutions that the model ultimately gets right, what is the average number of [BACK] tokens used?
• Wrong solutions (Figure 5(b)): Among solutions the model gets wrong, what is the average number of [BACK] tokens used?

These statistics reveal that models pretrained on retry data "hardly retry" on problems they solve correctly—for retry_rate = 0.2, the average retry count is less than 0.3 even on hard problems. The model uses [BACK] primarily when it is genuinely stuck, not as a routine part of its generation.

Unnecessary operation and parameter counts. Following Ye et al. (2024), the paper counts:

• Unnecessary operations: Extra computation steps beyond the minimal solution.
• Unnecessary parameters: Parameters that are computed but not needed for the final answer.

A model that has learned to reason efficiently should produce solutions with zero unnecessary operations and zero unnecessary parameters (i.e., shortest-path solutions). The paper uses these metrics to verify that retry data does not degrade solution efficiency.

The out-of-distribution (OOD) evaluation. For each dataset family, the op values in the training and test sets are disjoint. For example, iGSM-med_pq trains on op ≤ 15 but tests on op ∈ {20, 21, 22, 23}. This means the model must generalize to problems requiring more reasoning steps than it ever saw during training—a genuine test of reasoning generalization, not memorization. The reask variants provide an additional distribution shift by changing which parameter is queried.

Probing-Based Error Detection (Result 0)

Before studying error correction, the paper establishes a baseline capability that models pretrained on error-free data already possess: near-perfect internal error detection.

The can_next probing task. The probing setup is adopted directly from Ye et al. (2024). For a parameter A at a specific point in the solution, can_next(A) ∈ {true, false} indicates whether A can legally be computed next (i.e., all its dependencies are satisfied). The probe is a classifier that takes the model's last hidden layer representation after the token sequence "Define [param A] as" and predicts whether can_next(A) is true or false.

Probing architecture and training. The probe uses the V-probing technique from Ye et al. (2024):

• The pretrained model is frozen.
• A linear classification head is added on top of the output layer.
• A small rank-r update is applied to the input embedding layer.
• Only the linear head and the rank-r update are trainable; the rest of the model is fixed.

Two versions are used:

Version 1 (small parameter change):

• Rank r = 8 for the embedding update.
• Learning rate: 0.002 (no ramp-up, linear decay to 0).
• Weight decay: 0.01.
• Batch size: 256 for iGSM-med, 128 for iGSM-hard.
• Training steps: 100,000.
• This uses 50% of the pretraining tokens for iGSM-med and 25% for iGSM-hard—a substantial amount of fine-tuning data.

Version 2 (fewer fine-tuning samples, more trainable parameters):

• Rank r = 32 for the embedding update.
• Additional rank-2 update on query and value matrices in all 12 layers (this adds 768 × 4 × 2 × 12 trainable parameters).
• Batch size: 8.
• Learning rate: best between {0.001, 0.0005} (same linear scheduling).
• Weight decay: 0.
• Training steps: 16,000.
• This uses only 0.25% of the pretraining tokens.

Probing accuracy results. The paper reports:

"When A ranges over all possible parameters, the probing 99% accurately predicts can_next(A), meaning the model knows if A can be computed next, even for the hardest op = 32 problems."

And specifically on error cases:

"When the model makes a mistake, the first sentence with a mistake usually has can_next(A) = false. Probing shows the model has ∼60% chance of knowing can_next(A) = false, indicating it often knows it has made a mistake, right after stating the parameter name in full."

Why this is called "regret." The internal state after producing "Define [wrong param] as" contains information (extractable via linear probing with 99% accuracy in general, and 60% on mistakes) that the parameter was wrong. The model "knows" it made a mistake—its internal states register the error—but this knowledge does not prevent the mistake from occurring. This dissociation between detecting and preventing errors is what the paper calls "regret."

What this establishes for the paper. Result 0 demonstrates that error detection is a capability near-inherently present in models pretrained on error-free data. The model doesn't need special training to know when it has made a mistake; it already has that information internally. The bottleneck is not detection but something else—something that retry data aims to teach.

Retry Upon Regret (Result 1)

Given that the model can detect errors (Result 0), the natural question is: can we use this detection to improve accuracy during generation? The "retry upon regret" procedure does exactly this.

The retry upon regret generation algorithm. During autoregressive generation:

1. The model generates a solution sentence normally (e.g., "Define Central High's Film Studio as B; so B = ...").

2. After completing the sentence, the can_next probe evaluates whether the parameter just defined was legal.

3. If the probe says can_next = false (i.e., the model just made a mistake), the generation reverts to the end of the previous sentence—effectively erasing the erroneous sentence from the generation history—and regenerates from that point.

4. The regeneration uses multinomial sampling (beam=1, dosample=true) to introduce randomness, so the model has a chance to generate a different (hopefully correct) parameter this time.

5. This retry process continues with a maximum of 10 total retries (or 50 for the "versionP50" variant) for the entire solution, to prevent infinite loops.

The three detection quality levels tested:

• version1 and version2: Use the can_next probing with 99% accuracy (versions 1 and 2 described above). These represent realistic, imperfect error detectors.

• versionP: Uses the ground-truth can_next function (perfect accuracy). This represents an upper bound—what retry upon regret could achieve with a perfect error detector.

• versionP50: Same as versionP but allows up to 50 retries instead of 10, representing an even more generous upper bound.

Results: retry upon regret helps but is limited. The accuracy table in Figure 3(b) shows:

• Comparing version1/2 with beam32 (the strongest beam search baseline): retry upon regret provides a modest improvement, about 2 percentage points for the hardest cases (op=23 for iGSM-med, op=32 for iGSM-hard).

• Comparing version1/2 with versionP: the jump from 99% to 100% detection accuracy yields a much larger accuracy improvement, indicating that the bottleneck is detection reliability—a single false negative (missed error) can cause the entire solution to be wrong.

• Even versionP (perfect detection, 10 retries) does not achieve the accuracy levels that retry data pretraining achieves, establishing that error correction requires more than just detection plus re-sampling.

Why retry upon regret is limited. The paper identifies the fundamental issue:

"The idea of 'retry upon regret' increases the inference complexity because one needs to keep an error detector model alongside and keep checking the correctness of the generated solution steps. In the event of an error, the model needs to regenerate using randomness (possibly multiple times) until it passes the error detector."

And critically:

"Error correction is a skill that can be fundamentally different from beam search or retry based on the model's randomness."

The model pretrained on error-free data, when forced to regenerate after an error, relies entirely on randomness to produce a different (and hopefully correct) next step—the same mechanism as beam search, just triggered by an error detector instead of by beam expansion. There is no learned "correction strategy" that would allow the model to intelligently select a different parameter. Retry data pretraining, in contrast, teaches the model to actively correct errors as part of its learned generation policy.

LoRA Fine-Tuning Protocol (Result 7)

To test whether error correction can be acquired after pretraining, the paper uses LoRA (Low-Rank Adaptation) to fine-tune a model that was pretrained on error-free data, using retry data as the fine-tuning corpus.

Why LoRA? LoRA is the dominant parameter-efficient fine-tuning method in practice. It freezes the pretrained weights and adds trainable low-rank matrices that modify the model's behavior. If error correction could be learned through LoRA, then practitioners could take any pretrained model and fine-tune it on retry data without the expense of retraining from scratch.

LoRA configuration sweep. The paper tests an extensive range of LoRA configurations:

• Rank on query/value matrices: r ∈ {4, 8, 16, 32, 64, 128, 256}. The notation qv[rank] in the results (e.g., qv4e8) indicates the LoRA rank on query/value matrices.

• Rank on embedding matrix: 2r (twice the query/value rank). The e[rank] notation indicates the embedding rank.

• The total trainable parameters range from very small (r=4, adding only a few thousand parameters) to r=256, which the paper notes is "almost equivalent to full finetuning, given that the hidden dimension is 768 for GPT2-12-12."

LoRA fine-tuning hyperparameters:

For iGSM-med:

• Learning rate: 0.001 (lower than the 0.002 used for pretraining, following the convention that fine-tuning uses smaller learning rates)
• Weight decay: 0.05
• Batch size: 256
• Context length: 768
• Training steps: 200,000

For iGSM-hard:

• Learning rate: 0.001
• Weight decay: 0.05
• Batch size: 128
• Context length: 1024
• Training steps: 200,000

The training steps (200,000) provide the same number of tokens as pretraining for iGSM-med and half the pretraining tokens for iGSM-hard. The authors note that the training curves plateau, so additional steps would not help.

Full fine-tuning ("continued pretraining") baseline. For comparison, the paper also tests full fine-tuning (all parameters trainable) on retry data, using the same hyperparameters as the original pretraining except:

For iGSM-med:

• Learning rate: 0.001
• Weight decay: 0.05
• Batch size: 512
• Context length: 768
• Training steps: 100,000

For iGSM-hard:

• Learning rate: 0.001
• Weight decay: 0.03
• Batch size: 256
• Context length: 1024
• Training steps: 200,000

This full fine-tuning uses the same number of tokens as the original pretraining, meaning the model sees a total of 2T tokens (T from error-free pretraining + T from retry full fine-tuning).

The double-time pretraining comparison. To fairly evaluate full fine-tuning, the paper also tests pretraining directly on retry data for twice the tokens (pretrain double-time). This controls for the fact that full fine-tuning sees twice the total tokens. The pretrain double-time configuration uses:

• Weight decay reduced to 0.03 (from 0.05) for iGSM-med and to 0.02 (from 0.03) for iGSM-hard.
• Training steps doubled: 200,000 for iGSM-med, 400,000 for iGSM-hard.

Result assessment protocol. For LoRA and full fine-tuning experiments:

• Fine-tuning starts from 2 different pretrained models (2 random seeds for the error-free pretraining).
• For each, evaluation uses 4096 problems per cell.
• The reported accuracy is the best across the 2 seeds and the 2 beam choices (beam=1/4), consistent with the pretraining experiments.

Fake Retry Data Construction Methods (Result 8)

Real retry data requires knowing which parameters are legal at each step—information that is available for synthetic iGSM problems but not for real-world math solutions. The paper explores two methods for creating retry-like data that require no semantic understanding of the solution.

Retry_weak: inserting future steps as fake errors. The idea is simple: at each step, instead of requiring that the inserted error be genuinely illegal to compute, simply insert a sentence that appears later in the solution as the "error."

The algorithm:

1. At the current step, randomly select one of the sentences that appears later in the correct solution.
2. Insert that sentence (or just the parameter name) with [BACK] as if it were an error.
3. With probability (retry_rate)^2, insert a second future sentence, and so on.

The intuition is:

"Intuitively, it encourages the model not to skip steps."

Even though a parameter from later in the solution might technically be computable at the current step (the dependencies could be satisfied), its appearance as an "error" teaches the model a conservative strategy: don't try to compute parameters before you're sure about all the intermediate ones. This doesn't teach error correction in the exact sense, but it teaches a similar behavioral pattern—if you're about to jump ahead, retry and follow the correct order.

Concrete example from the paper. In the example of Figure 2, the correct solution order is:

Dance Studio's School Daypack → Film Studio's Messenger Backpack → Central High's Film Studio → Film Studio's School Daypack → Film Studio's Backpack → Central High's Backpack

At the position where "Central High's Film Studio" should be defined, retry_weak might insert "Film Studio's Backpack" (which appears later) as the error, followed by [BACK], followed by the correct "Central High's Film Studio."

Retry_miss: inserting unused parameters as fake errors. A slightly more sophisticated approach: at each step, insert as the "error" a parameter that appears in the problem statement but has not yet been used in the solution—and may not be needed at all. In the example, at the ♢♢♢ position, this could include parameters like "Riverview High's Film Studio" that reference entity types mentioned in the problem but are not part of any dependency chain needed for the answer.

The intuition:

"It encourages the model not only 'not to skip steps' but also not to compute unnecessary parameters."

This is slightly harder to implement than retry_weak because it requires consulting the problem statement to find unused parameters, but it is still much easier than the perfect retry data which requires full dependency graph verification.

Why retry_weak works while retry_miss doesn't. The results in Figure 8 show that retry_weak produces accuracy improvements comparable to perfect retry data, while retry_miss actually hurts accuracy. The paper doesn't provide a definitive mechanistic explanation, but the implication is that teaching "don't skip steps" (retry_weak) captures the essential benefit, while teaching "don't compute unnecessary parameters" (retry_miss) may create confusion—the model might learn to be overly cautious about parameter selection when what it really needs is better step-ordering discipline.

Practical implications. The retry_weak method is trivially implementable for any problem with a step-by-step solution: one only needs to identify future sentences and insert them. The paper speculates:

"While future LLMs may not be trained directly on such retry_weak data, our results suggest that it can be beneficial to, for instance, use auxiliary models to rewrite math data to include fake mistakes of this type."

This provides a concrete, implementable recommendation for practitioners: even if you can't reliably generate genuine errors and corrections, simply inserting out-of-order steps as "mistakes" to be "corrected" can teach the model the discipline of step-by-step reasoning.

Summary of Design Choices and Their Justifications

• Synthetic iGSM data over LLM-generated errors: Guarantees 100% reliability of error insertion and correction; isolates the effect of the retry format from confounds about error quality.

• Equal-tokens comparison framework: Ensures retry data pretraining is not advantaged by seeing more training tokens; any accuracy gain reflects genuine learning efficiency, not more compute.

• Standard autoregressive training (no masking) as the default: The empirical finding that masking is unnecessary simplifies the training pipeline and demonstrates that the model naturally prefers correct steps over errors, even when trained on error-containing data.

• Probing with minimal trainable parameters (rank-8 embedding update): Demonstrates that error detection information is already in the pretrained model's representations, not learned by the probe—the small rank proves the information is linearly accessible.

• LoRA sweep from r=4 to r=256: Tests the full spectrum from minimal adaptation to near-full-finetuning; the failure of even high-rank LoRA to teach error correction proves the capability requires deep weight changes, not just surface-level adaptation.

• Retry_weak as a practical proxy for perfect retry data: Addresses the real-world obstacle that genuine error detection requires semantic understanding of the solution; provides a simple, implementable alternative that captures the essential benefit.

• Context length extension at test time (768→2048): Allows the model to generate longer solutions than it saw during training, testing whether learned reasoning strategies (including error correction) generalize to longer problem sequences.

4. Key Insights and Innovations

Innovation 1: Error Correction Is a Fundamentally Different Skill from Error Detection, Not Just a More Difficult Version of It

The paper's most conceptually significant finding is the dissociation between error detection and error correction as distinct capabilities with fundamentally different learnability profiles. This is not simply a matter of one being harder than the other—the evidence shows they require qualitatively different kinds of training signal and weight modification to acquire.

What the field assumed before this work. The dominant paradigm for improving LLM reasoning has treated verification and self-correction as points on a spectrum: if a model can verify its output (an easier task), it should be able to use that verification signal to improve its output through prompting or fine-tuning. This assumption underlies the entire "self-correction" literature (Madaan et al., 2024; Weng et al., 2022; Pan et al., 2023), where the model is prompted to first generate, then verify, then revise. The implicit model is that verification provides a signal, and the model can use that signal—through in-context learning or lightweight adaptation—to fix its mistakes. If the model knows it made an error, why wouldn't it be able to correct it?

What this paper shows instead. Result 0 and Result 1 together establish the dissociation empirically, and Result 7 provides the mechanistic evidence for why:

• Result 0 (probing): A model pretrained on error-free data already has near-perfect internal representations for error detection. The can_next probe achieves 99% accuracy using a rank-8 embedding update—meaning the information is essentially linearly decodable from the model's existing hidden states. The model "knows" when it has made a mistake at the moment it makes it.

• Result 1 (retry upon regret): But giving the model access to this error detection signal during generation—forcing it to regenerate when the probe detects an error—yields only marginal accuracy gains (roughly 2 percentage points on the hardest problems). Even a perfect error detector (versionP) with 50 retries cannot achieve what pretraining on retry data achieves.

• Result 7 (LoRA fine-tuning): The most decisive evidence comes from the fine-tuning experiments. Applying LoRA to a model pretrained on error-free data, using the exact same retry data that works when included in pretraining, fails to teach error correction across a sweep of LoRA ranks from 4 to 256 (the latter being nearly full fine-tuning). Small LoRA ranks actually hurt accuracy relative to the original error-free model, and even the largest ranks produce accuracy far below what pretraining on retry data achieves. Meanwhile, full fine-tuning (all parameters) can teach error correction, but only with as many retry tokens as the original pretraining consumed—making it indistinguishable from continued pretraining.

The implication is profound: error correction is not simply using the error detection signal to select a different action. If it were, beam search (which explores multiple actions) or "retry upon regret" with perfect detection (which flags the error and re-samples) would work nearly as well as retry pretraining. The fact that they don't—and that LoRA cannot bridge the gap—suggests that error correction requires the model to restructure its internal computation in a way that only full-scale pretraining (or equivalent continued pretraining) can achieve.

Why this reframes the problem. Prior work on self-correction has focused on the prompting mechanism: how to elicit verification and revision from the model. This paper's dissociation suggests that the bottleneck is not in the elicitation but in the model's learned representations. A model that has never seen error-correction patterns during pretraining has no internal circuitry for executing the correction behavior, regardless of how it is prompted or what auxiliary signals are provided. The verification capability is present (the model can classify errors), but the correction capability—the ability to back up, identify the correct alternative, and continue—requires having been trained on sequences that exercise exactly that pattern.

This is not an incremental refinement of existing self-correction approaches. It is a fundamental claim about the origin of cognitive capabilities in language models: certain reasoning behaviors cannot be "unlocked" through prompting or lightweight adaptation because they require representational structures that only pretraining-scale optimization over appropriate data can build. The paper's evidence is unusually clean because the synthetic setting allows the authors to hold everything constant—same data, same task, same compute—while varying only whether error-correction examples appear during pretraining or fine-tuning.

The practical consequence is the paper's strongest prescriptive claim:

"Error correction is a skill that can be very different from the original (error-free) reasoning and may not be acquired during a LoRA finetune stage, even with a sufficient number of finetune (retry) samples."

This means that for future LLMs, teaching self-correction is not an alignment-phase concern—it must be addressed at the pretraining stage, when the model's fundamental reasoning structures are being formed.

Innovation 2: Training on Mistakes Does Not Teach Models to Make Mistakes—It Teaches Them When to Self-Correct

A persistent anxiety in machine learning is that training models on data containing errors will cause them to reproduce those errors at inference time. This concern is especially acute for autoregressive language models, where the next-token prediction objective directly incentivizes the model to generate whatever patterns appear in the training data. The paper confronts this concern head-on and provides a striking counter-narrative: not only does training on error-filled data not cause the model to generate more errors, but higher error rates in training data (up to 50%) produce monotonically better reasoning accuracy at test time, with the model actively choosing to generate fewer retries than the training data would suggest.

What the field assumed. The default intuition—which the paper explicitly acknowledges as a "potential harm" in the introduction—is that training on data with mistakes is dangerous. If the model sees "Define Central High's Classroom as [BACK]" during training, the next-token prediction loss will push it to predict those same tokens when it encounters similar contexts. The natural mitigation is label masking: exclude the erroneous tokens from the loss so the model isn't rewarded for predicting them. This intuition has shaped how the field handles imperfect training data—either filter it out or mask the problematic portions.

What this paper shows instead. The results in Section 4 systematically dismantle this concern:

• Result 2 (accuracy increases with retry rate): On the hardest out-of-distribution problems, accuracy jumps from 78% (error-free training) to 95% (retry rate = 0.5 training, no masking). The relationship is monotonic: within the tested range, higher retry rates produce better accuracy. This is true despite the fact that higher retry rates mean the model sees fewer distinct problems (because solutions are longer) and a higher fraction of tokens are error-related.

• Result 3 (masking is unnecessary): Adding label masking to prevent the model from learning to predict error tokens produces no consistent improvement in accuracy. The results with and without masking are essentially identical in Figure 4(b), even at retry_rate = 0.5 where half of all solution steps contain errors. The model trained with standard autoregressive loss on error-filled data performs as well as or better than the model trained with carefully masked errors.

• Result 4 (the model rarely retries during inference): The behavioral statistics in Figure 5(a) are perhaps the most surprising: a model trained with retry_rate = 0.2 (meaning 20% of training steps contain errors) uses the [BACK] token an average of less than 0.3 times per correct solution, even on the hardest problems. The model is not copying the error pattern from its training data—it is selectively applying error correction only when genuinely needed.

• Result 5 (solution efficiency is preserved): Figure 6 shows that models trained on retry data still produce shortest-path solutions with essentially zero unnecessary operations and zero unnecessary parameters, matching the efficiency of models trained on error-free data.

Why this works: the statistical intuition. The paper offers a partial explanation rooted in the statistics of the training data. At each solution step with retry_rate = p, the probability of generating a correct step (without error) is 1 - p, while the probability of each specific erroneous parameter is substantially smaller because there are many possible wrong parameters to choose from. For p = 0.5, the model is still incentivized to prefer correct steps in most contexts. The paper draws a parallel to language model learning on context-free grammars with mistakes (Allen-Zhu and Li, 2023), where models trained on noisy data can still learn the underlying clean grammar at lower generation temperatures.

But the empirical result goes beyond this statistical explanation. Models trained on retry data don't just avoid generating errors—they generate fewer errors than models trained on error-free data, despite having been exposed to far more error examples during training. This suggests the model is learning something transferable: the skill of recognizing and avoiding error-prone situations, not just the pattern of error-then-correction.

Significance as a methodological contribution. This finding has direct implications for how training data should be constructed for future LLMs. If training on carefully constructed error-correction data is not only safe but actively beneficial—and if standard autoregressive training works without special masking—then the barrier to incorporating such data is far lower than previously assumed. The paper's advocacy in the conclusion follows directly from this evidence:

"We strongly discourage using fine-tuning to teach a model to correct errors... We advocate for adding mistakes and corrections at the pretrain level."

This is not an incremental gain claim. It is a reversal of a widely-held assumption about the dangers of training on imperfect data, backed by controlled experiments that isolate the retry data variable.

Innovation 3: The "Retry Upon Regret" Baseline Formally Establishes That Beam Search and Error-Aware Resampling Cannot Substitute for Learned Error Correction

Section 3's "retry upon regret" experiments might initially appear to be a minor warmup result, but they serve a crucial function that goes beyond providing a performance baseline: they formally rule out the hypothesis that error correction can be achieved by combining error detection with random resampling. By systematically comparing beam search, retry with realistic error detection (99% accurate), and retry with perfect detection, the paper establishes a clear hierarchy: detection + resampling < learned correction, and the gap is not due to detector accuracy.

What the field assumed. A natural engineering intuition—which the paper's authors likely anticipated from reviewers—is that error correction is "just" detection plus re-generation. If you can detect errors, and you can generate multiple candidates (beam search), you should be able to correct errors by flagging them and sampling alternatives until you pass the detector. This intuition underlies approaches like tree-of-thought reasoning and verifier-guided search, where models explore multiple reasoning paths and select among them.

What this paper shows instead. The "retry upon regret" results in Figure 3(b) systematically test this hypothesis at multiple levels of idealization:

• Beam search (beam=16, beam=32) on error-free trained models produces essentially no improvement over greedy decoding on the hardest problems (the curves are flat). This confirms that the model's internal probability distribution does not place sufficient mass on correct alternatives for beam search to find them—the error is not a matter of sampling variance.

• Retry upon regret with realistic detection (version1, version2) provides marginal gains (roughly 2 percentage points on the hardest cases) but cannot approach what retry pretraining achieves. Even with 99% accurate error detection and up to 10 regeneration attempts per error, the accuracy improvement is small.

• Retry upon regret with perfect detection (versionP) shows significantly higher accuracy than realistic detection, revealing that detector accuracy is indeed a bottleneck—a single false negative (missed error) can cause the entire solution to fail, and false positives cause wasted regeneration.

• Retry upon regret with perfect detection and 50 retries (versionP50) establishes the upper bound for what detection + resampling can achieve, and even this falls short of retry pretraining accuracy for most difficulty levels.

The diagnostic value of this comparison. The key insight from this experiment is not the accuracy numbers themselves but the shape of the gap between versionP and retry pretraining. If error correction were simply detection + re-sampling, then versionP (perfect detection, multiple regeneration attempts) should match retry pretraining. The fact that it doesn't—that even knowing with 100% certainty that an error occurred and having 50 chances to fix it cannot bridge the gap—proves that retry pretraining teaches the model something beyond "detect error, try again."

What retry pretraining teaches, the paper argues, is a positive correction strategy: the model learns to identify which specific alternative to generate when an error is detected, rather than blindly sampling from its distribution and hoping. This is the difference between "I know this is wrong, let me try something random" (retry upon regret) and "I know this is wrong, and I know that the correct next parameter should be Y because I have Z information available" (retry pretraining).

Why this reframes the research direction. This finding redirects effort away from improving error detection (which the paper shows is already near-perfectly learnable) and toward improving the correction mechanism itself. The limiting factor is not knowing that a mistake was made, but having the internal computational capacity to recover from it. This is a conceptual shift that has implications for architecture design (what mechanisms enable error recovery during autoregressive generation?), training data composition (what patterns teach effective recovery strategies?), and evaluation (should we measure correction success rate separately from first-attempt accuracy?).

Innovation 4: Fake Retry Data (Retry_Weak) Provides a Practical, Semantic-Free Method for Teaching Step-Ordering Discipline Without Requiring Error Understanding

The paper's final major contribution is a methodological one with clear practical implications: the retry_weak data construction method demonstrates that the benefits of retry training can be largely captured without any semantic understanding of what constitutes a genuine error. By simply inserting later solution steps as "errors" and then "correcting" them with the proper step, the model learns the discipline of following correct step ordering—and this discipline alone accounts for most of the accuracy gains seen with perfect retry data.

What distinguishes this from the naive approach. The natural way to construct retry data would be to identify genuine logical mistakes (parameters that cannot legally be computed next) and insert those as errors. This requires full semantic parsing of the solution and access to the underlying dependency structure—information that is available for synthetic iGSM problems but not for real-world math solutions. The paper's retry_weak method eliminates this requirement entirely: to construct retry_weak data, one only needs to know the order of sentences in the correct solution, not why that order is correct.

The retry_weak algorithm is trivially implementable:

1. Take a correct step-by-step solution.
2. At each step, with probability retry_rate, insert one of the later sentences as an "error" followed by [BACK].
3. Then insert the actual correct sentence for that step.

The inserted "error" might, in fact, be a perfectly legal computation at that point in the solution—the dependencies might already be satisfied. But by treating it as an error and immediately replacing it with the correct step for that position, the training data teaches the model a behavioral pattern: do not generate steps that "belong" later in the solution.

Evidence that retry_weak captures the essential benefit. Figure 8 shows that retry_weak training achieves accuracy gains approaching those of perfect retry data, particularly on the hardest OOD problems. For iGSM-med op=23, retry_weak with retry_rate = 0.5 achieves roughly 94% accuracy versus 95% for perfect retry data and 78% for error-free training. The gap between retry_weak and perfect retry is small, suggesting that the primary benefit of retry training is teaching step-ordering discipline, with genuine logical error correction providing a modest additional gain.

In contrast, retry_miss (which inserts unused parameters as errors) actually hurts accuracy. The paper's interpretation—that retry_miss teaches unnecessary caution about parameter selection while retry_weak teaches the specific skill of step ordering—is a post-hoc explanation rather than a proven mechanism, but the empirical contrast is clear.

Why this matters beyond the synthetic setting. The paper explicitly connects this finding to real-world LLM training:

"While future LLMs may not be trained directly on such retry_weak data, our results suggest that it can be beneficial to, for instance, use auxiliary models to rewrite math data to include fake mistakes of this type."

This provides an immediately actionable recommendation: if you have a corpus of correct step-by-step solutions (from textbooks, educational websites, or LLM-generated training data), you can construct retry-like training data by programmatically reordering steps—no error understanding required. The paper is careful not to claim this will work on real problems (the experiments are entirely synthetic), but the principle—that teaching models to follow correct ordering through fake error-correction patterns is effective—is a concrete hypothesis that can be tested on natural data.

A note on what this is and is not. Retry_weak is not a novel algorithm in the sense of being technically sophisticated—it's a straightforward data augmentation trick. Its significance lies in the diagnostic insight it provides: it reveals that the benefit of retry training is not primarily about teaching the model to recognize complex logical dependency violations. Rather, it is about teaching a simpler behavioral pattern—"don't jump ahead, follow the order"—that the model can learn from shallow cues. This insight could redirect efforts to construct retry-like data for real-world applications, since it suggests perfect error identification is unnecessary for capturing most of the benefit.

Innovation 5: The Paper Provides a Conceptual Framework for Understanding When Reasoning Skills Must Be Learned During Pretraining vs. When They Can Be Added Later

While the paper does not present this as an explicit theoretical framework, its systematic comparison of pretraining, LoRA fine-tuning, and full fine-tuning (continued pretraining) for both error detection and error correction establishes a taxonomy of reasoning skill learnability that transcends the specific task studied.

The taxonomy, implicit in the results. The paper's experiments reveal three distinct categories of reasoning-related capabilities:

1. Detection skills (almost free). Error detection—knowing whether a step is correct—is a capability that models acquire naturally from error-free pretraining. The can_next probe achieves 99% accuracy with a rank-8 embedding update and can be trained to similar accuracy with just 0.25% of the pretraining data (version2 probing). This suggests that detection skills emerge as a byproduct of learning to reason correctly; the model's internal representations automatically encode verification information.

2. Correction skills (require deep weight changes during pretraining-scale optimization). Error correction cannot be acquired through LoRA fine-tuning from an error-free pretrained model, even with high ranks, extensive retry data, and label masking. It can be acquired through full fine-tuning, but this requires as many retry tokens as the original pretraining and is effectively continued pretraining. This suggests that correction skills require restructuring the model's internal representations in ways that only large-scale gradient updates (not low-rank adaptations) can achieve.

3. Behavioral shaping (can be added through data augmentation of pretraining). The retry_weak results suggest that certain reasoning behaviors—specifically, following correct step ordering—can be taught through relatively simple data patterns that don't require the model to understand why the ordering is correct. These behaviors may be more about learning surface-level patterns (what sequences of steps are typical) than about learning deep logical constraints.

Why this taxonomy matters beyond this paper. The distinction between detection, correction, and behavioral shaping has direct implications for LLM training pipeline design:

• If a capability is a "detection skill," practitioners can rely on it being present in pretrained models and need only lightweight adaptation to surface it for specific tasks.

• If a capability is a "correction skill," it must be incorporated at the pretraining stage—post-hoc fine-tuning will not suffice. This has implications for data curation strategies: if you want your model to be able to recover from its own reasoning errors, you need error-correction patterns in the pretraining corpus, not just in the fine-tuning data.

• If a capability can be achieved through "behavioral shaping" (like retry_weak), then data augmentation strategies that create superficially error-like patterns may be sufficient, without requiring semantically accurate error identification.

The limits of the framework. The paper provides this taxonomy implicitly through its experimental results; it does not develop a mechanistic theory of why correction requires deep weight changes while detection does not. The probing literature (including the companion paper Ye et al., 2024) provides some intuition—detection may correspond to reading out information already present in the model's computational traces, while correction may require rewiring the model's internal planning mechanisms—but this remains speculative. The taxonomy is therefore best understood as an empirical generalization that is well-supported within the iGSM setting but requires testing in other domains to be considered a general principle.

Significance. This is the paper's most forward-looking contribution. While the specific retry data format and iGSM experiments may not directly transfer to production LLM training, the categorization of skills by their learnability stage (pretraining vs. fine-tuning) provides a conceptual tool for thinking about what data to include in pretraining corpora. As the paper notes in its conclusion, commercial LLMs already use synthetic data, and "future LLMs are rumored to use Q⋆"—understanding which capabilities must be built during pretraining versus which can be added later is essential for efficient allocation of increasingly expensive training resources.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the iGSM dataset from Ye et al. (2024), a synthetic, program-generated set of grade-school-level math reasoning problems. The dataset removes arithmetic difficulty (computations are modulo 23) and common-sense requirements, isolating logical reasoning as the sole challenge. Two main families are used: iGSM-med (training on op ≤ 15, OOD testing on op ∈ {20, 21, 22, 23}) and iGSM-hard (training on op ≤ 21, OOD testing on op ∈ {28, 29, 30, 31, 32}), each with pq and qp variants (problem-before-question vs. question-before-problem) plus reask variants that modify the test distribution by re-sampling the queried parameter. The dataset has over 90 trillion unique solution templates, and new data is generated on-the-fly throughout training—there is no fixed dataset, no repetition, and no risk of memorization-based overfitting.

• Base model(s). All experiments use GPT2-12-12 (12 layers, 12 attention heads, 768-dimensional hidden states), modified from the original GPT-2 architecture by replacing absolute positional embeddings with rotary positional embeddings (RoPE). The authors note they also tested the Llama architecture with gated MLP layers and observed no benefit, so the simpler GPT2 variant is used throughout. The model size (~85M parameters for the standard GPT2-small configuration) is deliberately small to enable extensive controlled experiments under practical compute constraints while still being capable of non-trivial reasoning on the iGSM tasks.

• Metrics. The primary metric is fully-correct solution accuracy: a solution is counted as correct only if every intermediate step computes a parameter that is legally computable at that point (no skip errors), every computation is arithmetically correct, and the final answer matches ground truth. This is verified by a custom parser, not by simple answer-integer comparison. Secondary behavioral metrics include retry count (average number of [BACK] tokens used per solution, computed separately for correct and wrong solutions), unnecessary operations (extra computation steps beyond the minimal solution), and unnecessary parameters (parameters computed but not needed for the final answer). Each accuracy cell is averaged over 4096 evaluation problems of the specified type.

• Baselines. The paper uses several baselines, all evaluated on models pretrained on error-free iGSM data: (1) Original greedy/beam search generation with beam=1 (greedy), beam=4, beam=16, and beam=32, where beam > 1 uses multinomial sampling to maintain diversity across beams. (2) Retry upon regret version1, using the can_next probe with a rank-8 embedding update (50% of pretraining tokens for fine-tuning, 99% probing accuracy) to detect errors and trigger regeneration with a maximum of 10 retries. (3) Retry upon regret version2, using a probe with rank-32 embedding update plus rank-2 query/value updates across all layers but trained on only 0.25% of pretraining tokens (also 99% accuracy). (4) Retry upon regret versionP, using the ground-truth can_next function as a perfect error detector (10 retries). (5) Retry upon regret versionP50, same as versionP but with a maximum of 50 retries. These baselines collectively establish what error detection alone can achieve without retry data pretraining.

• Generation budget / compute accounting. The paper's core fairness principle is equal training tokens: all comparisons between retry and error-free pretraining use the same number of total training tokens. For iGSM-med, this means 100,000 steps at batch size 512 with context length 768. For iGSM-hard, it means 200,000 steps at batch size 256 with context length 1024. Because retry data solutions are longer than error-free solutions (errors add tokens), a model pretrained on retry data sees fewer distinct math problems within the same token budget—the paper explicitly accounts for this. At inference time, the compute budget is measured in beam size (1, 4, 16, or 32) for the original baselines, and in maximum retry count (10 or 50) for the "retry upon regret" baselines. For LoRA and full fine-tuning experiments, the token budgets are matched to pretraining (200,000 steps for LoRA, 100,000 for full fine-tuning on iGSM-med; 200,000 for both on iGSM-hard), and the "pretrain double-time" condition uses twice the tokens.

• Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional sense (the test sets are fixed by the iGSM data generation procedure). Instead, each accuracy cell is the best result across 2 random seeds for pretraining (to account for training stochasticity) and 2 beam configurations (beam=1 with greedy decoding, or beam=4 with multinomial sampling). The authors note: "We discover it is better to keep dosample=false while beam=1 and dosample=true while beam > 1." This best-of-4 reporting convention is applied uniformly to all conditions—both retry-trained and error-free-trained models—so comparisons between conditions are valid, though absolute numbers are optimistic relative to any single training run. Each cell represents the average over 4096 evaluation problems, providing reasonably tight estimates (standard error on a proportion with n=4096 is at most ~0.8 percentage points at 50% accuracy, smaller near the extremes).

Main Quantitative Results

Probing and Retry Upon Regret (Section 3, Results 0-1)

Result 0: Error detection is near-perfectly decodable from pretrained model states. The can_next probing achieves 99% accuracy on both iGSM-med (tested up to op = 23) and iGSM-hard (tested up to op = 32), as established in Ye et al. (2024) and replicated here. The paper emphasizes that this high accuracy is achieved with version1's rank-8 embedding update—a minimal parameter change—meaning the information is essentially linearly present in the pretrained model's hidden states. On the subset of cases where the model actually makes a mistake (generates a parameter with can_next = false), the probe achieves ~60% accuracy, which is significantly above the 50% random-guess baseline given the probe's 99% accuracy on the full distribution. The paper interprets this as evidence of a "regretful" pattern: after generating an erroneous parameter, the model's internal state often registers the error.

Result 1: Retry upon regret improves accuracy over beam search, but gains are limited. Figure 3(b) presents the comprehensive comparison. The headline findings are:

• Beam search alone is ineffective: On iGSM-med op = 23 (the hardest OOD condition in that dataset), the original error-free model achieves roughly 78% accuracy with beam=1, and beam=32 provides essentially no improvement—the accuracy curve is flat across beam sizes. On iGSM-hard op = 32, the pattern is similar: accuracy remains around 78% regardless of beam size. This confirms that the model's internal probability distribution does not place meaningful mass on correct alternatives when it makes a skip error; the error is not a sampling variance problem.

• Retry upon regret provides modest gains: On iGSM-med op = 23 in the qp variant, retry upon regret version1/2 (with beam=1) achieves approximately 80% compared to 78% for the original beam=1—a gain of about 2 percentage points. VersionP (perfect detection, 10 retries) achieves approximately 84%, and versionP50 (perfect detection, 50 retries) reaches approximately 88%. The pattern is similar on iGSM-hard op = 32: version1/2 improves by roughly 2 points over the original, versionP improves further, and versionP50 pushes accuracy higher still.

• Detection accuracy is a bottleneck: The substantial gap between version1/2 (99% detection) and versionP (100% detection) reveals that even a 1% error rate in the detector significantly limits the gains from retry upon regret. A single false negative—failing to detect an actual error—dooms the entire solution, since the model continues generating on a faulty foundation. False positives (flagging a correct step as wrong) cause wasted regeneration but are generally less harmful.

• Even perfect detection cannot match retry pretraining: VersionP50 establishes the upper bound for what detection + resampling can achieve, and these numbers are still substantially below what retry data pretraining achieves (e.g., 88% vs. 95% on iGSM-med op = 23). This is the key evidence that error correction is not simply detection plus re-sampling—retry pretraining teaches a positive correction strategy that "retry upon regret" does not capture.

The paper also notes practical limitations: retry upon regret "increases the inference complexity because one needs to keep an error detector model alongside and keep checking the correctness of the generated solution steps" and "alters the generation process, which might not align with the 'general intelligence' framework, where the goal is typically for one model to perform all tasks using the same (autoregressive) decoding algorithm."

Pretraining with Retry Data (Section 4, Results 2-6)

Result 2: Retry data pretraining improves accuracy, and higher retry rates help more. Figure 4(b) presents the central accuracy comparison between models pretrained on error-free data ("original") and models pretrained on retry data with retry_rate ∈ {0.05, 0.1, 0.2, 0.4, 0.5}, all using the same number of total training tokens. The headline results, focusing on the hardest OOD conditions where gains are largest:

• iGSM-med op = 23, qp variant: Original accuracy is approximately 78%. Retry pretraining with retry_rate = 0.05 achieves approximately 84%. With retry_rate = 0.1, accuracy rises to approximately 89%. With retry_rate = 0.2, approximately 92%. With retry_rate = 0.5, the model reaches approximately 94–95%. The relationship is monotonic: higher retry rates produce better accuracy across the tested range.

• iGSM-hard op = 32, qp variant: Original accuracy is approximately 78%. Retry pretraining with retry_rate = 0.05 achieves approximately 85%. With retry_rate = 0.2, approximately 93%. With retry_rate = 0.5, approximately 95%. Again, the relationship is monotonic.

• iGSM-med op = 20 (reask), qp variant: This represents a distribution shift from changing the queried parameter. Original accuracy is approximately 91–92%. Retry pretraining with retry_rate = 0.5 pushes this to approximately 96–97%.

• In-distribution performance: On training-range op values (e.g., op = 15 for iGSM-med), all models achieve near-perfect accuracy (99–100%), showing that retry data does not degrade performance on problems the model already handles well. The gains are concentrated in the OOD regime where error-free models struggle.

The paper emphasizes that these gains come despite the retry-trained models seeing fewer distinct math problems (because solutions with errors are longer, so fewer fit in the same token budget). The higher retry_rate, the greater the problem-count reduction—yet accuracy still improves, indicating that the error-correction patterns are highly valuable per token of training data.

Result 3: Label masking on error tokens is unnecessary. Figure 4(b) shows results both with and without label masking for each retry_rate. The "with mask" rows train with standard autoregressive loss but exclude the erroneous tokens (the wrong parameter name and "as") from the loss computation. The key observation is that the with-mask and without-mask rows are nearly identical across all retry_rate values and all test conditions. For instance, on iGSM-med op = 23, retry_rate = 0.5 with masking achieves approximately 94.5% vs. 94.8% without masking—a negligible difference. The paper concludes: "Masking mistakes is unnecessary. We observe that it is generally not needed to introduce label masking on the error data even for large retry rate = 0.5."

This is a practically significant finding because it means retry data can be incorporated into standard autoregressive pretraining pipelines without any modification to the training objective or data preprocessing—no error-boundary detection, no special token handling, no auxiliary loss terms.

Result 4: Retry-trained models rarely use [BACK] during correct generation. Figure 5(a) reports the average number of retries per test problem among correctly solved problems. For models pretrained with retry_rate = 0.2 (without masking), the average retry count is less than 0.3 even for the hardest OOD conditions (op = 23 or op = 32). For retry_rate = 0.1, it is less than 0.15. For retry_rate = 0.05, it is less than 0.1. Even at retry_rate = 0.5, the average retry count without masking is 2–4 on the hardest problems, and masking reduces this to 0.5–1.5. This means the model, despite being trained on data where 50% of steps contained errors, generates correct solutions with very few retries—it is not copying the error pattern from training.

Figure 5(b) shows that in contrast, on wrong solutions, the model uses [BACK] extensively—trying and failing to correct its mistakes until the solution is ultimately incorrect. For retry_rate = 0.5 on iGSM-hard op = 32 without masking, wrong solutions average 5–8 retries. This asymmetry—few retries on correct solutions, many retries on wrong ones—indicates the model is using [BACK] strategically when it genuinely struggles, not as a default generation pattern.

Result 5: Retry-trained models still produce shortest solutions with no unnecessary computation. Figure 6 reports the average number of unnecessary operations and unnecessary parameters in correctly generated solutions. For all retry_rate values and all test conditions, both unnecessary operations and unnecessary parameters are extremely close to zero (overwhelmingly <0.05 per solution, often 0.00). This matches the error-free baseline, demonstrating that retry data does not cause the model to produce longer or less efficient solutions. The model still learns to identify the minimal set of necessary parameters and compute them in the correct order—the retry skill is deployed only when an error would otherwise occur.

Result 6: Error correction via retry pretraining fundamentally differs from beam search or detection-based retry. The paper makes this as a comparative claim rather than presenting a separate figure. The comparison is between Figure 4(b) (retry pretraining) and Figure 3(b) (beam search and retry upon regret):

• Beam search (beam=32) on error-free models provides essentially no accuracy gain on the hardest problems.
• Retry upon regret with 99% accurate detection provides roughly 2 percentage points of gain.
• Retry upon regret with perfect detection and 50 retries provides roughly 10 percentage points of gain (reaching ~88% on iGSM-med op = 23).
• Retry pretraining with retry_rate = 0.5 provides roughly 17 percentage points of gain (reaching ~95%).

The conclusion: "Error correction is a skill that can be fundamentally different from beam search or retry based on the model's randomness." The model learns something from retry data—an active correction strategy—that cannot be replicated by simply detecting errors and re-sampling.

Fine-Tuning with Retry Data (Section 5, Result 7)

Result 7: LoRA fine-tuning from error-free models cannot teach error correction; full fine-tuning can, but is equivalent to continued pretraining. Figure 7 (in the main text) and Figure 10 (in Appendix D.3, with additional retry_rate values) present the comprehensive fine-tuning results. All experiments use a model pretrained on error-free data and then fine-tuned on retry data.

LoRA results: The paper sweeps LoRA ranks from r = 4 to r = 256 on query/value matrices (with embedding rank 2r), spanning from tiny adaptation to near-full-finetuning. The key findings:

• Small LoRA ranks hurt accuracy: On iGSM-med op = 23, a model pretrained on error-free data achieves approximately 78%. LoRA fine-tuning with r = 4 (and retry data with retry_rate = 0.2) reduces accuracy to approximately 63% without masking and 72% with masking. This degradation is consistent across small ranks and is worse without label masking—the model actually becomes less accurate than before fine-tuning.

• Higher LoRA ranks provide modest improvements at best: As LoRA rank increases, accuracy generally improves, but even at r = 256 (which the paper notes is "almost equivalent to full finetuning, given that the hidden dimension is 768"), accuracy on iGSM-med op = 23 reaches only approximately 84–85% with masking, compared to approximately 92% for pretraining directly on retry data with the same retry_rate = 0.2. The gap is substantial and robust across rank choices.

• Label masking matters much more for LoRA than for pretraining: Unlike pretraining, where masking made no difference, LoRA fine-tuning shows consistent benefits from label masking, especially at lower ranks. At r = 4 with retry_rate = 0.2, masking improves accuracy from ~63% to ~72% on iGSM-med op = 23. This suggests that when model capacity for change is constrained (low-rank adaptation), training on unmasked errors is actively harmful because the model learns to produce errors without having the representational flexibility to also learn the correction strategy.

• The pattern is consistent across retry rates: Figure 10 shows similar results for retry_rate ∈ {0.05, 0.1, 0.2, 0.5}. Higher retry rates with large LoRA ranks can occasionally outperform the original model (e.g., retry_rate = 0.5, r = 256 with masking on iGSM-med op = 23 reaches approximately 88%, compared to 78% for the original), but this still substantially lags behind direct pretraining with the same retry data (which achieves 95%).

Full fine-tuning ("continued pretraining") results: Full fine-tuning (all parameters trainable) using retry data can achieve accuracy comparable to or exceeding direct pretraining on retry data. For instance, on iGSM-med op = 23, full fine-tuning with retry_rate = 0.2 (both with and without masking) achieves approximately 92%, matching or slightly exceeding the direct pretraining result. However, the paper's Figure 10 comparison against "pretrain double-time" (pretraining on retry data for twice as many tokens) clarifies that this is simply the effect of seeing more training tokens: the error-free pretrained model has already seen T tokens, and full fine-tuning adds another T tokens of retry data, for a total of 2T tokens—roughly matching what "pretrain double-time" sees. There is no significant advantage to the two-stage process over simply pretraining on retry data from the start for the full duration.

The core conclusion: "Error correction is a skill that can be very different from the original (error-free) reasoning and cannot be acquired during a LoRA finetune stage from language models pretrained only using error-free data." The paper explicitly contrasts this with error detection (Result 0), which required only a rank-8 embedding update to achieve 99% accuracy. Detection is an "almost free" capability that emerges from error-free pretraining; correction requires deep representational changes that only pretraining-scale optimization (or equivalent full-model continued training) can achieve.

Fake Retry Data (Section 6, Result 8)

Result 8: Retry_weak (inserting future steps as fake errors) captures most of the benefit of perfect retry data; retry_miss (inserting unused parameters) hurts accuracy. Figure 8 presents the accuracy comparison for models pretrained on retry_weak and retry_miss data against the original error-free and perfect retry baselines.

Retry_weak results: The retry_weak method—which at each step inserts a randomly selected later sentence as a "fake error" followed by [BACK]—produces substantial accuracy gains:

• On iGSM-med op = 23, retry_weak with retry_rate = 0.5 achieves approximately 93–94% accuracy (compared to 78% for error-free and 94–95% for perfect retry). The gap between retry_weak and perfect retry is small (~1–2 percentage points).
• On iGSM-hard op = 32, retry_weak with retry_rate = 0.5 achieves approximately 94% (compared to 78% for error-free and 95% for perfect retry). Again, the gap is small.
• The gains are monotonic with retry_rate: higher rates provide better accuracy across the tested range.
• Masking provides no consistent benefit for retry_weak, mirroring the finding for perfect retry data.

Retry_miss results: In stark contrast, retry_miss—which inserts unused problem parameters as errors—actually degrades accuracy:

• On iGSM-med op = 23, retry_miss with retry_rate = 0.5 achieves only approximately 55% accuracy—far below the 78% achieved by error-free training. At retry_rate = 0.2, accuracy is approximately 64%.
• The degradation is worst at higher retry rates, and it is consistent across test conditions. The paper doesn't provide a definitive mechanistic explanation but suggests the intuition that retry_miss "encourages the model not only 'not to skip steps' but also not to compute unnecessary parameters"—and this additional constraint may interfere with learning effective step-ordering.

Behavioral statistics for fake retry data: Figure 11 (Appendix C) reports retry counts and unnecessary operation/parameter counts for retry_weak and retry_miss models. The key observations:

• Retry_weak and retry_miss models retry more frequently than perfect retry models during generation (Figure 11a). For retry_rate = 0.5, retry_weak models average 4–6 retries on the hardest problems (compared to 2–4 for perfect retry), and retry_miss models average 5–9 retries. This makes sense: since the fake errors are not genuinely illegal steps, the model may learn a looser criterion for when to retry.
• Despite higher retry counts, both retry_weak and retry_miss models still produce shortest solutions with essentially zero unnecessary operations/parameters (Figure 11b), matching the efficiency of error-free and perfect retry models.

Practical significance: The retry_weak method is trivially implementable for any dataset with step-by-step solutions—it requires only the ability to identify sentences that appear later in the solution, not any semantic understanding of why the step ordering matters. The paper presents this as a concrete, immediately actionable finding: "our results suggest that it can be beneficial to, for instance, use auxiliary models to rewrite math data to include fake mistakes of this type."

Ablation Studies and Robustness Checks

• Label masking on error tokens in pretraining (Results 2-3, Figure 4b): Adding label masks to exclude erroneous tokens from the loss provides no consistent accuracy improvement across all retry_rate values (0.05 to 0.5) and all test conditions. Masked and unmasked rows are nearly identical in Figure 4(b). This is a non-obvious finding: the model does not learn to generate errors even when trained to predict them via standard autoregressive loss. For LoRA fine-tuning (Results 7, Figure 7), in contrast, label masking matters significantly—at low LoRA ranks, masking prevents accuracy degradation, suggesting that when adaptation capacity is constrained, learning to predict error tokens interferes with learning correction.

• Retry rate sweep (Result 2, Figure 4b): Within the tested range retry_rate ∈ {0.05, 0.1, 0.2, 0.4, 0.5}, higher retry rates monotonically improve accuracy on OOD problems. This holds despite the fact that higher retry rates mean fewer distinct problems seen during training (because solutions are longer for the same token budget). The paper does not test retry_rate > 0.5, noting that "retry rate cannot approach 1" and "exploring such extreme failure settings is not particularly interesting."

• Pretraining vs. full fine-tuning vs. pretrain double-time (Result 7, Figures 7 and 10): Comparing full fine-tuning from an error-free model (total tokens = 2T) against pretraining directly on retry data for twice the tokens (pretrain double-time, also 2T) shows no significant accuracy difference. This confirms that the two-stage process is simply continued pretraining—there's no special benefit or harm from the error-free pretraining initialization when full-model training is used.

• LoRA rank sweep (Result 7, Figures 7 and 10): The paper tests LoRA ranks from r = 4 to r = 256 on query/value matrices (with embedding rank 2r), spanning from ~0.1% to near-100% of model parameters. Even r = 256—which is essentially full fine-tuning in terms of parameter count for a 768-dimensional model—fails to match retry pretraining accuracy. This suggests the limitation is not parameter count but rather the nature of the optimization: LoRA's low-rank constraint restricts the types of representational changes possible, and error correction requires changes that violate this constraint.

• Retry_weak with and without masking (Result 8, Figure 8): Mirroring the perfect retry data findings, label masking provides no consistent benefit for retry_weak training. The with-mask and without-mask rows for retry_weak are nearly identical across all retry rates and test conditions.

• Retry_miss as a negative result (Result 8, Figure 8): Retry_miss data actively degrades accuracy across all test conditions and retry rates, with the worst degradation at retry_rate = 0.5 (accuracy drops to 55% vs. 78% for error-free on iGSM-med op = 23). This serves as an important negative result: not all "fake error" strategies are beneficial, and teaching the model to avoid unused parameters appears to interfere with learning correct step ordering.

• Behavioral statistics across data types (Figures 5, 6, 11): The paper measures retry counts and solution efficiency (unnecessary ops/params) for all data conditions, establishing that: (a) retry-trained models rarely retry on correct solutions (Figure 5a), (b) retry-trained models retry extensively on wrong solutions (Figure 5b), (c) all training configurations produce shortest-path solutions with near-zero unnecessary computation (Figures 6 and 11b). These behavioral statistics rule out alternative explanations for the accuracy gains (e.g., "the model is just trying more random parameters and getting lucky").

• Two random seeds and two beam configurations (all results): Every accuracy cell in Figures 4(b), 7, 8, and 10 reports the best result across 2 pretraining seeds and 2 beam choices (beam=1 greedy or beam=4 multinomial). This provides a consistent (if optimistic) comparison framework; the absolute numbers are upper bounds, but relative comparisons between conditions remain valid since the same convention applies throughout.

• Evaluation across multiple OOD difficulty levels and distribution shifts (all figures): Results are reported separately for each op value (op = 20, 21, 22, 23 for iGSM-med; op = 28, 29, 30, 31, 32 for iGSM-hard), plus reask variants, in both pq and qp formats. The consistency of findings across these independent test conditions—higher retry rates help, masking doesn't matter, LoRA fails, retry_weak works—provides strong evidence that the results are not artifacts of a particular difficulty level or question format.

Critical Assessment

The paper's central claims, as established in the Executive Summary, are: (1) pretraining on retry data improves reasoning accuracy compared to error-free data using equal tokens, (2) label masking on error tokens is unnecessary, (3) retry-trained models do not generate more errors during inference, (4) error correction cannot be acquired through LoRA fine-tuning from error-free models, and (5) fake retry data (retry_weak) captures most of the benefit. The experimental evidence supporting each claim varies in strength and scope.

Claim 1 (retry data improves accuracy) is solidly supported within the experimental scope but has important boundary conditions that are not fully explored. The accuracy gains in Figure 4(b) are large and consistent: on iGSM-med op = 23, accuracy rises from roughly 78% to 95% when going from error-free to retry_rate = 0.5 pretraining, a 17-percentage-point absolute improvement. These gains are replicated across four dataset families (med_pq, med_qp, hard_pq, hard_qp), multiple OOD difficulty levels, and both reask and non-reask test distributions. The equal-tokens comparison framework is correctly implemented, ruling out the confound that retry models simply see more training tokens.

However, the claim is demonstrated exclusively on the iGSM synthetic dataset with GPT2-12-12. The dataset's specific properties—particularly that errors are of exactly one type (skip errors) and are programmatically inserted with 100% reliability—may make the retry pattern easier to learn than it would be on natural data with diverse error types and noisy corrections. The model scale (GPT2-12-12, roughly 85M parameters) is two to three orders of magnitude smaller than contemporary production LLMs, and it is unknown whether the finding that "within a reasonable range, the more mistakes the better" would hold at larger scales where models may have stronger priors or different learning dynamics. The paper explicitly acknowledges these limitations, stating it "does not claim that the synthetic data used here can directly aid in building future LLMs," but the strength of the empirical pattern within the studied setting warrants testing on natural data before the claim can be considered general.

Claim 2 (label masking is unnecessary) is well-supported but the mechanism is not fully explained. The evidence in Figure 4(b) is unambiguous: masked and unmasked pretraining produce essentially identical accuracy across all conditions. The paper offers a statistical intuition—at each step, the probability of a correct continuation is still higher than any specific wrong parameter, so the model is naturally incentivized to prefer correct steps—but this is a post-hoc rationalization rather than a demonstrated mechanism. The probing or mechanistic analysis that would confirm this explanation (e.g., showing that the model's internal probability distribution places low mass on error tokens even when trained on them) is not performed. This is a genuine gap: we know that masking is unnecessary, but not why the model successfully avoids learning the error pattern. The finding is also specific to the pretraining setting; in LoRA fine-tuning, masking matters substantially (Figure 7), so the claim applies only to the pretraining regime.

Claim 3 (retry-trained models don't generate more errors) requires careful interpretation. The behavioral statistics in Figures 5 and 6 do show that retry-trained models generate few retries on correct solutions and still produce shortest-path solutions. However, the paper defines "retry count" as the number of [BACK] tokens generated—this captures only errors that the model both made and recognized (by following with [BACK]). If the model makes errors but doesn't mark them with [BACK], those errors would appear as incorrect solutions, not as retry counts. The retry count metric therefore likely underestimates the model's actual error rate; it measures only caught and corrected errors, not all errors. The paper partially addresses this through the wrong-solution retry counts in Figure 5(b), which show higher retry rates on wrong solutions, but a direct measurement of error frequency (e.g., through probing all generated steps for can_next violations, regardless of whether [BACK] follows) would provide a more complete picture of whether the model's underlying error tendency changes with retry training.

Claim 4 (LoRA cannot teach error correction) is very strongly supported but the interpretation deserves nuance. The LoRA sweep in Figures 7 and 10 is unusually thorough, testing ranks from 4 to 256 across multiple retry rates. The finding is robust: LoRA fine-tuning, regardless of configuration, cannot match retry pretraining accuracy. Even r = 256—which the paper correctly notes is "almost equivalent to full finetuning" in terms of parameter count—falls substantially short (e.g., ~85% vs. 92% on iGSM-med op = 23 with retry_rate = 0.2). This is the paper's strongest experimental result because of the thoroughness of the sweep and the size of the gap.

What requires nuance is the claim that this shows error correction "cannot be acquired during a LoRA finetune stage." The paper does not disentangle whether the limitation is due to LoRA's low-rank constraint per se, or due to the learning rate, optimization dynamics, or interaction between the frozen pretrained weights and the LoRA adapters. A full fine-tuning run with the same learning rate and schedule as the LoRA experiments would test whether the optimization procedure or the parameterization is the bottleneck. The paper does run full fine-tuning, but with pretraining-scale learning rates (0.001) rather than the LoRA learning rate (also 0.001 but with different effective optimization dynamics due to the low-rank constraint). The claim that error correction "requires major weight changes" is a reasonable interpretation, but alternative explanations (e.g., LoRA's optimization landscape makes it harder to escape the error-free model's local minimum) are not ruled out.

Claim 5 (retry_weak captures most of the benefit) is well-supported for the iGSM setting but the generalization claim is speculative. Figure 8 shows retry_weak with retry_rate = 0.5 achieving roughly 93–94% on iGSM-med op = 23 vs. 94–95% for perfect retry—a gap of only 1–2 percentage points. This is a striking result: inserting random future steps as errors is nearly as effective as inserting genuinely illegal steps. However, the claim's practical significance hinges on whether retry_weak would work on natural math problems, where solution steps may be longer, less formulaic, and have more complex dependency structures. The paper is appropriately cautious: "We cannot explore such follow-up directions due to GPU resource limitations" and "we aim to use controlled experiments to predict what could be the important data changes." This is a hypothesis-generating finding, not a demonstrated practical method, and the paper's framing reflects this.

Genuine weaknesses in the experimental design include:

• Single model architecture and scale. All experiments use GPT2-12-12. The finding that error correction requires pretraining-scale optimization (not LoRA) may interact with model scale—larger models might have more capacity to adapt to new reasoning patterns through lightweight fine-tuning. The paper cannot address this.

• Single error type. All retry data inserts exactly one kind of error: skip errors (parameters that cannot legally be computed next). Real LLM reasoning errors are far more diverse—arithmetic mistakes, logical fallacies, incorrect assumptions, missing cases. Whether retry training on skip errors would transfer to other error types is not tested and is far from obvious.

• No dynamic or adaptive retry rate experiments. The paper uses a fixed retry_rate throughout training. A curriculum where retry_rate increases over training, or where errors are selectively inserted on problems the model finds difficult, might be more effective or more token-efficient. This is not explored.

• The difficulty estimation from the prior analysis is entirely absent here. Unlike the paper summarized in the example (which used difficulty binning to allocate test-time compute), this paper's retry data is applied uniformly to all problems regardless of difficulty. The interaction between problem difficulty and optimal retry_rate—whether easy problems benefit as much as hard ones, whether different rates should be used for different difficulty levels—is not analyzed.

• No combination with beam search at inference time. The retry-trained models are evaluated with beam=1 or beam=4. Whether retry training's benefits are additive with or superseded by more aggressive beam search at inference is not tested. This is a relevant omission because the comparison with beam search in Section 3 shows beam search alone is ineffective on error-free models, but the combination could be synergistic.

Experiments that would have strengthened the paper:

• Probing analysis of retry-trained models. The paper uses probing only for error-free models (Result 0). Probing retry-trained models to understand how their internal representations differ—do they have stronger can_next signals? Do they represent correction alternatives before generating them?—would provide mechanistic insight into what retry training teaches.

• Testing on natural math data with LLM-generated errors. Even a small-scale experiment using GPT-4-generated errors and corrections on a subset of GSM8K would help bridge the synthetic-to-real gap and validate the paper's practical recommendations.

• Retry_rate beyond 0.5. The paper stops at 0.5, noting that exploring "extreme failure settings is not particularly interesting," but a sweep up to 0.8 or 0.9 could reveal whether there is a point where the "more mistakes the better" trend reverses, which would be theoretically informative even if practically irrelevant.

• Measurement of undetected errors in retry-trained model outputs. A direct comparison of error rates (not just retry counts) between error-free and retry-trained models, using the ground-truth can_next function to check every generated step, would clarify whether retry training actually reduces the model's tendency to make errors or just teaches it to catch and correct them more effectively.

Conditional nature of the claims:

• The claim that "retry data improves reasoning accuracy" holds for OOD problems where the error-free model's accuracy is substantially below ceiling (e.g., 78% on op = 23). On in-distribution problems (e.g., op = 15), both error-free and retry-trained models achieve 99–100%, so retry data provides no benefit—it is not harmful, but it doesn't help on problems the model already solves perfectly.

• The claim that "the more mistakes the better" holds within the tested range of retry_rate ∈ [0.01, 0.5]. The paper does not claim monotonicity extends beyond 0.5, and the authors explicitly note that retry_rate "cannot approach 1."

• The claim that "label masking is unnecessary" holds for pretraining with the iGSM data format. For LoRA fine-tuning, masking matters significantly, so the claim is specific to the pretraining regime.

• The claim that "error correction cannot be acquired through LoRA" holds for the LoRA configurations and training durations tested. Whether a different adaptation method (e.g., full fine-tuning with a smaller learning rate, or adapter-based approaches with different architectural placement) could succeed is not ruled out—only that standard LoRA cannot.

Overall, the experimental evidence for the paper's core claims is strong within the controlled iGSM setting, with the most robust findings being the accuracy gains from retry pretraining (Figure 4b), the ineffectiveness of LoRA (Figures 7, 10), and the effectiveness of retry_weak (Figure 8). The primary limitations are the single dataset/model/error-type scope and the absence of mechanistic analysis explaining why retry training works. The paper's value lies in the clarity and consistency of the empirical patterns it establishes, which provide concrete hypotheses for future work on natural data and larger models.

6. Limitations and Trade-offs

Single Synthetic Dataset with a Single Error Type

The assumption or constraint. All experiments are conducted exclusively on the iGSM dataset, a program-generated, synthetic math reasoning benchmark where errors are of exactly one type: "skip errors," defined as generating a parameter whose dependencies have not yet been computed. The paper's core intervention—retry data—inserts precisely this type of error into training solutions. The authors are transparent about the scope:

"By utilizing fully-controllable synthetic data (e.g., controlling error rates or label masking), conducting controlled experiments (e.g., beam search vs. retry vs. error-free; pretrain vs. finetune), and performing fair comparisons (e.g., same number of training tokens), the goal of this paper is to try to predict the needs of future LLMs. We do not claim that the synthetic data used here can directly aid in building future LLMs."

And in justifying the synthetic choice:

"While it is tempting to use language models such as GPT-4 to synthesize such data, there is no evidence that GPT-4 can robustly generate errors or make those corrections consistently."

The consequence. The paper's central finding—that pretraining on retry data teaches models to correct errors—is demonstrated for a single, highly-structured error type in a setting where the distinction between correct and incorrect steps is binary and perfectly verifiable. Real LLM reasoning errors are far more diverse: arithmetic mistakes (e.g., 12 + 13 = 25 when modulo arithmetic is not used), logical fallacies (affirming the consequent, circular reasoning), incorrect assumptions, missing edge cases, and semantic misinterpretations of problem statements. The iGSM dataset explicitly removes arithmetic difficulty (computations are modulo 23) and common-sense requirements precisely to isolate logical reasoning, but this also removes the types of errors that dominate real-world LLM failures on tasks like GSM8K, MATH, or code generation. A model that learns to correct skip errors may not transfer this capability to correcting arithmetic mistakes or logical fallacies, because the internal representational changes needed to detect and recover from those error types may be entirely different.

Furthermore, the retry data construction in this paper relies on perfect, programmatic error insertion: every "retry" involves a parameter that is definitively illegal to compute, followed by a correct step. In real training data constructed from LLM outputs or human annotations, errors will be noisy—some corrections may themselves be wrong, some errors may be ambiguous, and the boundary between "error" and "acceptable alternative reasoning path" will be fuzzy. Whether retry training remains beneficial under realistic noise levels is untested.

What evidence exists in the paper. The paper contains no experiments on natural math data, no experiments with error types other than skip errors, and no experiments with noisy or imperfect corrections. The entire empirical corpus—Figures 3 through 11, all accuracy tables, all behavioral statistics—is on iGSM data. The authors acknowledge this in the conclusion:

"While grade-school level math problems have many other difficulties (including arithmetic or common sense), following [29], we have focused on the (logic-following) reasoning aspect, which is one of the weakest aspects of GPT-4."

But the interaction between retry training and other error types is never tested.

Mitigation status. The paper does not attempt to address this limitation empirically. It frames the work as a "physics of language models" study aimed at discovering principles rather than building deployable systems. The retry_weak experiment (Section 6) provides a partial bridge: by showing that the benefit of retry data can be captured without semantically accurate error identification, it suggests a path toward real-world application where perfect error labeling is unavailable. But this does not test whether retry training on skip errors transfers to other error types, or whether models would learn to correct arithmetic mistakes from analogous retry data. The paper's prescriptive claims—"We strongly discourage using fine-tuning to teach a model to correct errors... We advocate for adding mistakes and corrections at the pretrain level"—are presented as general principles despite being validated on only one error type in one synthetic setting. Evaluating these claims on natural math benchmarks with diverse error types remains essential future work.

Single Model Architecture and Scale

The assumption or constraint. All experiments use GPT2-12-12, a roughly 85M-parameter model with 12 transformer layers, 12 attention heads, and a 768-dimensional hidden state. The authors note:

"We also played with the Llama architecture (especially with gated MLP layers) and did not see any benefit of using it. This GPT2 performs comparably to Llama/Mistral at least for knowledge tasks."

The only architectural variation tested is the choice between GPT2 and Llama—all results in the paper are from the GPT2 variant.

The consequence. The finding that error correction "cannot be acquired during a LoRA finetune stage" (Result 7) and requires pretraining-scale optimization (or equivalent continued pretraining) may be specific to this model scale. Larger models—with billions rather than millions of parameters—have qualitatively different learning dynamics. They exhibit stronger in-context learning, more robust generalization, and different fine-tuning behavior. A 70B-parameter model fine-tuned with LoRA on retry data might successfully acquire error correction in ways that an 85M-parameter model cannot, simply because the larger model's pretrained representations are richer and more amenable to low-rank adaptation for novel reasoning patterns. The paper's claim that error correction requires "major weight changes, potentially beyond what parameter-efficient fine-tuning (PEFT) can handle" may be a statement about the capacity of 85M-parameter models, not a fundamental principle about error correction.

Similarly, the finding that "within a reasonable range, the more mistakes the better" for retry_rate may not hold at larger scales. Larger models, with their stronger priors from more diverse pretraining data, might be more sensitive to the distribution of errors in training data, or might require lower error rates to avoid learning spurious error-generation patterns. The optimal retry_rate might decrease with model scale—or it might increase. The paper provides no evidence either way.

What evidence exists in the paper. The paper contains exactly one model scale: GPT2-12-12. There are no scaling experiments, no comparisons across model sizes, and no ablations testing whether the key claims (LoRA failure, optimal retry_rate, masking necessity) change with model capacity. The authors tested one architectural alternative (Llama) and report no benefit, but this is an architecture comparison at fixed scale, not a scaling study.

Mitigation status. Not addressed. The paper does not claim its findings are scale-invariant, but it also does not qualify its prescriptive conclusions by model size. The recommendation to add retry data "at the pretraining stage, rather than in the finetuning (alignment) stage" is stated as a general principle. Testing whether this recommendation holds for models in the 1B–70B parameter range—or whether LoRA becomes effective for error correction at sufficient scale—is critical future work that the paper does not attempt.

The Difficulty Estimation Problem Is Avoided Entirely

The assumption or constraint. The paper's retry data construction requires knowing, at each solution step, which parameters are legal to compute next. This information is available for the synthetic iGSM dataset because the dependency graph is programmatically generated alongside each problem. The paper acknowledges this implicitly when describing retry data construction:

"Since we use a controllable, synthetic math dataset, we can, at the beginning of each solution sentence, with probability retry rate ∈ [0, 1), insert a wrong parameter that cannot be computed next."

The retry_weak method (Section 6) partially addresses this by removing the need for semantic understanding—it inserts future sentences as fake errors without checking legality. But retry_weak still requires knowing the correct solution order (i.e., which sentences appear later), and more fundamentally, it does not solve the problem of how to identify genuine reasoning errors in real solutions.

The consequence. The paper's headline finding—that pretraining on retry data improves accuracy—cannot be directly applied to natural math data without solving a prerequisite problem: how to generate retry data from natural solutions. This requires either (a) a reliable method for identifying errors in LLM-generated solutions, or (b) a reliable method for generating realistic errors and corrections from correct solutions. The paper demonstrates that method (b) works in the synthetic setting (retry_weak), but this does not establish that it works on natural data.

For real-world math solutions, the challenges are substantial:

• Correct solutions from textbooks or training corpora may not have clean, sentence-level step boundaries that correspond to dependency graph nodes. A single sentence might combine multiple reasoning steps, or a single reasoning step might span multiple sentences.
• The "correct order" of steps may not be unique—there may be multiple valid topological sorts of the reasoning dependencies, and what constitutes "skipping ahead" is less clear.
• Errors in real solutions are diverse: skip errors exist but are joined by arithmetic mistakes, incorrect formula applications, misinterpreted problem constraints, and logical leaps. Inserting random future steps as "errors" (retry_weak) would not capture these error types.

The paper also does not address the cost of generating retry data at scale. In the synthetic setting, retry data is generated on-the-fly with negligible computational overhead. For real data, whether the retry data is constructed by prompting LLMs, using auxiliary verifier models, or through human annotation, the data preparation cost is non-trivial and would need to be amortized against the accuracy gains.

What evidence exists in the paper. The retry_weak results (Figure 8) provide partial evidence that perfect error identification is not necessary, but these are exclusively on iGSM data. There are no experiments with natural math data, no ablations testing sensitivity to error identification quality, and no analysis of the cost of retry data generation relative to the benefit. The authors explicitly flag this gap:

"We cannot explore such follow-up directions due to GPU resource limitations."

Mitigation status. The paper acknowledges the gap and proposes a research direction: "use auxiliary models to rewrite math data to include fake mistakes of this type." But it provides no implementation or evaluation of this approach. The retry_weak method is a proof of concept within the synthetic setting, not a validated method for real data. A practitioner wanting to apply retry training to their own math reasoning dataset would need to solve the data generation problem from scratch—the paper provides inspiration but not a solution.

Retry Upon Regret Is the Only Inference-Time Baseline, and the Comparison Is Weak in Several Dimensions

The assumption or constraint. The paper compares retry data pretraining against two alternative approaches: (1) standard autoregressive generation with varying beam sizes (beam=1, 4, 16, 32), and (2) "retry upon regret," which uses a trained error detector to trigger regeneration when an error is detected. The paper argues that neither matches retry pretraining accuracy, concluding that error correction learned during pretraining is fundamentally different from what can be achieved at inference time.

The consequence. The baseline comparison has several weaknesses that limit the strength of this conclusion:

• The retry upon regret detector is not jointly optimized with the generation strategy. The can_next probe is trained on the error-free model's hidden states, not on the model as it is actively generating under the retry-upon-regret regime. A detector jointly trained or continually adapted during the retry process might achieve higher accuracy, especially on the distribution of states produced after multiple regeneration attempts.

• The beam search comparison never uses the retry model as the proposal distribution. The paper's strongest inference-time approach—beam search—is evaluated only on error-free pretrained models, not on retry-pretrained models. It is possible that beam search applied to a retry-pretrained model would be synergistic: the retry model generates better individual candidates (because it can self-correct), and beam search amplifies this by exploring multiple paths. The paper never tests this combination.

• The retry upon regret procedure does not use the [BACK] token mechanism that retry pretraining teaches. Retry upon regret operates by truncating the generation and re-sampling from the previous sentence boundary—it does not use the [BACK] token at all. Yet the retry-pretrained model has learned to use [BACK] as its correction mechanism. The comparison is therefore between two different correction strategies (external truncation vs. internal [BACK] generation), not between two models trying to perform the same task. A fairer comparison would evaluate whether retry-pretrained models, when forced to use the retry-upon-regret generation procedure (i.e., external error detection + truncation), outperform error-free models under the same procedure.

• The comparison does not include approaches like training a separate verifier model (as in Cobbe et al., 2021) to score and select among multiple complete solutions. This is a widely-used alternative for improving reasoning accuracy at inference time, and the paper does not evaluate whether verifier-based selection could close the gap with retry pretraining.

What evidence exists in the paper. Figure 3(b) provides the beam search and retry-upon-regret results for error-free models. Figure 4(b) provides the retry pretraining results. The comparison between them is the basis for Result 6:

"Error correction is a skill that can be fundamentally different from beam search or retry based on the model's randomness."

But this comparison does not control for the fact that beam search and retry upon regret are evaluated on error-free models while retry pretraining involves a different training procedure. The paper does not report beam search accuracy for retry-pretrained models, nor retry-upon-regret accuracy for retry-pretrained models, nor any combination of retry pretraining with inference-time search strategies.

Mitigation status. Not addressed. The paper treats inference-time approaches and pretraining-time approaches as separate categories and compares them only at the aggregate level. The claim that error correction is "fundamentally different" from beam search is supported in the sense that beam search on error-free models doesn't help, but the paper does not rule out that beam search on retry-pretrained models would help more, or that a hybrid approach combining retry pretraining with inference-time search could outperform either alone. This is a missed opportunity to characterize the relationship between learned error correction and inference-time computation strategies.

The Claim That Error Correction Cannot Be Learned Through LoRA Is Not Fully Characterized—It May Be a Limitation of Optimization, Not Representation

The assumption or constraint. Section 5 demonstrates that LoRA fine-tuning of an error-free pretrained model on retry data fails to match the accuracy of models pretrained directly on retry data. The LoRA sweep ranges from rank 4 to rank 256 on query/value matrices (with embedding rank 2r), where r = 256 is "almost equivalent to full finetuning, given that the hidden dimension is 768." The paper concludes:

"Error correction is a skill that can be very different from the original (error-free) reasoning and cannot be acquired during a LoRA finetune stage from language models pretrained only using error-free data."

The consequence. The paper interprets the LoRA failure as evidence that error correction requires "major weight changes, potentially beyond what parameter-efficient fine-tuning (PEFT) can handle." However, the experiments do not disentangle several possible explanations for the LoRA failure:

• Optimization difficulty vs. representational capacity. LoRA fine-tuning uses the same learning rate (0.001) as full fine-tuning, but the effective optimization landscape for low-rank adapters differs substantially from full-model training. The low-rank constraint may create a more challenging optimization problem—narrower valleys, more local minima, or slower convergence—that additional training steps or a different learning rate schedule could overcome. The paper trains LoRA for 200,000 steps and states the loss plateaus, but this does not rule out that a different optimizer configuration (learning rate warmup, different decay schedule, different optimizer entirely) would succeed. The failure might be about the optimization procedure reaching a suboptimal point in the low-rank parameter space, not about the fundamental expressivity of low-rank adaptations for the error correction task.

• The interaction between frozen pretrained weights and LoRA adapters. Even with rank 256 LoRA, the pretrained weights from error-free training remain frozen and constitute the majority of the model's computation. It is possible that these frozen weights actively interfere with the LoRA adapters' attempts to implement error correction—for instance, the frozen attention patterns might attend to the wrong context for identifying correction alternatives, and the LoRA adapters cannot override these attention patterns strongly enough to redirect computation. The limitation would then be about the interaction between fixed and adapted components, not about the capacity of the adaptation per se.

• The nature of the data, not the nature of the skill. The retry data used for LoRA fine-tuning is identical to the retry data used for pretraining. But a model pretrained on error-free data has already developed specific internal representations and generation policies optimized for that data. Presenting it with retry data during fine-tuning creates a distribution shift that may require "unlearning" certain patterns (e.g., the policy of always generating the first legal parameter) before "learning" new ones (e.g., the policy of detecting and correcting skip errors). Full fine-tuning can unlearn and relearn; LoRA, with its frozen base weights, may be unable to sufficiently suppress the original generation policy.

The paper's claim that error correction "cannot be acquired during a LoRA finetune stage" is empirically true for the tested configurations, but the mechanistic interpretation—that this reveals something fundamental about error correction as a skill—is a stronger claim than the experiments directly support.

What evidence exists in the paper. Figure 7 and Figure 10 provide the LoRA results. The sweep across ranks is thorough, and the failure is consistent. The paper does not provide:

• Experiments varying the LoRA learning rate, optimizer, or schedule.
• Experiments testing whether longer LoRA training (beyond plateau) helps.
• Experiments with alternative PEFT methods (adapters, prefix tuning, prompt tuning) to test whether the failure is specific to LoRA's low-rank formulation.
• Probing analysis of LoRA-fine-tuned models to understand what they learn vs. what they fail to learn. For instance, does LoRA fine-tuning improve the model's internal error detection representations even if it doesn't improve generation accuracy? This would distinguish between "LoRA can't learn correction at all" and "LoRA learns correction internally but can't express it during generation due to frozen base weights."
• Experiments where the base model is partially unfrozen (e.g., unfreezing only the last few layers while keeping LoRA on the rest) to test whether the bottleneck is the frozen early layers.

Mitigation status. The paper acknowledges one dimension of this limitation: full fine-tuning can succeed, but requires "sufficiently many retry data... though this resembles continued pretraining." This correctly identifies that the capability can be learned with full-model training. But the paper does not explore the intermediate space between LoRA and full fine-tuning—partially unfreezing layers, using higher learning rates for LoRA, or testing whether the LoRA failure is robust to optimization hyperparameters. The claim that error correction "may not be acquired during a LoRA finetune stage" is stated as a general principle without characterizing the boundary conditions under which it holds.

The Paper Measures Retries but Not Underlying Error Rates—The Claim That Retry Training Doesn't Encourage Mistakes Is Incompletely Supported

The assumption or constraint. A central concern the paper addresses is whether training on error-containing data teaches models to generate more errors. The paper's response relies on the retry count metric: the average number of [BACK] tokens generated per solution. Result 4 states:

"Models pretrained on retry data hardly retry (unless retry rate is very high). For instance, Figure 5(a) shows if retry rate = 0.2, even when pretrained without label masking, the model retries an average of < 0.3 times even for math problems with large op."

The consequence. The retry count metric captures only errors that the model both makes and catches (by following with [BACK]). It does not measure errors that the model makes but does not catch—for instance, if the model generates "Define Central High's Film Studio as B; so B = p + W = 17 + 13 = 7" before p and W have been defined, and then continues without generating [BACK], this is an error that would manifest as an incorrect solution, not as a retry count. The retry count metric could remain low even if the model's underlying error rate increased, as long as the model learned to not use [BACK] when it made errors—i.e., if retry training taught the model to generate errors without self-correcting them.

The paper partially addresses this concern through wrong-solution retry counts in Figure 5(b), which shows that on wrong solutions, models do use [BACK] extensively. But this still doesn't measure the underlying error rate—the probability that any given generated step is a skip error—separately from the correction rate—the probability that the model marks an error with [BACK] and corrects it. An ideal analysis would use the ground-truth can_next function to label every generated step in every evaluation run as correct or incorrect, then report both the error rate (fraction of steps that are wrong, regardless of whether [BACK] follows) and the correction rate (fraction of wrong steps that are followed by [BACK] and a correct revision). This would reveal whether retry training reduces the underlying error rate (the model makes fewer mistakes), increases the correction rate (the model catches more of its mistakes), or both.

Without this decomposition, it is not possible to fully validate the paper's claim that retry training "teaches models how to correct errors if needed, rather than encouraging mistakes." The low retry count on correct solutions is consistent with this claim but does not prove it, because the retry count conflates error generation with error detection.

What evidence exists in the paper. The paper provides retry counts (Figures 5 and 11a) and solution-level accuracy (Figure 4b and others). It does not provide step-level error analysis using ground-truth labels. The accuracy improvements in Figure 4(b) show that retry-trained models produce more correct final answers, which implies that the net effect of whatever changes occur in error rate and correction rate is positive. But the claim that retry training "does not encourage mistakes" is not directly measured—it is inferred from the combination of low retry counts and high accuracy. A model could have a higher underlying error rate than the error-free model but compensate through more effective correction, which would contradict the claim that retry training doesn't encourage mistakes while still being consistent with the observed accuracy improvement.

Mitigation status. The paper does not acknowledge this measurement limitation. The retry count is treated as a proxy for error tendency, but the relationship between retry count and actual error rate is never validated. The probing infrastructure from Section 3 (which can label steps as correct or incorrect using the can_next function) could be applied to generated solutions to directly measure error rates, but this analysis is not performed. This is a notable omission given that probing is already used extensively in Sections 3.1-3.2 for the error-free model and could be applied with minimal modification to retry-trained models.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new architecture, a new training algorithm, or even a new dataset. It proposes something more fundamental: a reframing of when and how reasoning capabilities must be acquired by language models. The central empirical finding—that error correction cannot be learned through post-hoc fine-tuning from a model pretrained on error-free data, but can be learned during pretraining—shifts the conversation around self-correction from "how do we prompt models to fix their mistakes?" to "what data must be present during pretraining for models to develop correction abilities at all?"

This is a conceptual shift of moderate magnitude. It is not a paradigm overthrow on the scale of the transformer architecture or the Chinchilla scaling laws—the paper works within the standard autoregressive pretraining framework and does not challenge any foundational assumptions about how language models learn. But it does challenge a specific and widely-held assumption: that verification and self-correction capabilities can be "unlocked" from pretrained models through clever prompting or lightweight adaptation. The evidence from Sections 3 through 5 paints a clear picture: error detection is already present in models pretrained on error-free data (99% probing accuracy from a rank-8 linear readout), but this detection capability cannot be converted into effective correction behavior through prompting, beam search, or LoRA fine-tuning—even with perfect detection and 50 retry attempts. The capability gap between detection and correction is not one of elicitation but of representation: the model simply does not have the internal circuitry to execute the correction behavior, and building that circuitry requires exposure to correction patterns during the optimization process that forms the model's core reasoning structures.

The paper resolves a specific tension in the self-correction literature. Prior work has been divided between optimism (Madaan et al., 2024; Weng et al., 2022: models can self-correct when prompted) and pessimism (Huang et al., 2023: "large language models cannot self-correct reasoning yet"). The paper's decomposition of self-correction into detection (easy, present by default) and correction (hard, requires pretraining data) offers a reconciliation: both camps are right, but about different things. Models can detect their errors, which enables prompted self-correction in some cases, but they cannot reliably generate corrections unless correction patterns were in their training data. The mixed results in the literature reflect the fact that different studies tested different models on different tasks with different amounts of implicit correction data in their pretraining corpora. This paper provides the controlled evidence needed to move from "sometimes it works" to a predictive understanding of when it works.

The most significant change this paper introduces is methodological rather than technical. The systematic comparison of pretraining, LoRA fine-tuning, and full fine-tuning for the same capability—what might be called a learnability-stage taxonomy—provides a template for how the field should evaluate whether a reasoning skill belongs in pretraining or can be deferred to alignment. The paper shows that this question can be answered empirically with controlled data, matched compute budgets, and thorough sweeps over adaptation methods. This methodology is more informative than the common practice of testing a capability on a pretrained model, finding it lacking, and concluding the model "can't do it"—because the paper demonstrates that for error correction, the capability is learnable but only during pretraining. Future work on reasoning capabilities should adopt this pretrain-vs-finetune comparison as a standard diagnostic rather than treating pretrained model performance as a ceiling.

Research directions that become more attractive:

• Pretraining data composition for reasoning skills. The paper makes a strong case that certain reasoning behaviors must be present in pretraining data, not just in fine-tuning data. This elevates the importance of pretraining corpus curation and synthetic data generation as research areas. If error correction requires pretraining-scale exposure, then what other reasoning capabilities—formal proof verification, counterfactual reasoning, multi-step planning—have the same requirement? The paper's methodology provides a way to test this systematically.

• Understanding the representational basis of learned correction. The paper shows that retry training works and that LoRA cannot substitute for it, but does not investigate what changes in the model's internal representations. Probing retry-trained models using the same can_next technique from Section 3—or developing new probes for correction-specific computations—would open a line of investigation into the mechanistic basis of self-correction.

• Data augmentation for error-correction patterns in real pretraining corpora. The retry_weak result (Section 6) suggests that even superficial error-correction patterns can be beneficial. This motivates research into how to automatically annotate or generate correction data at the scale needed for pretraining—not just for math but for code, logical reasoning, and other structured generation tasks.

Research directions that become less attractive:

• Prompting-based self-correction as a primary remedy for reasoning errors. If error correction requires pretraining data, then prompting can at best elicit a capability the model already has—it cannot create one from nothing. Research effort is better directed at understanding what data produces the capability during pretraining, and then verifying that prompting can surface it, rather than trying to engineer prompts that produce correction behavior in models that lack the underlying representational structures.

• Verifier-guided search that does not modify the proposal distribution. The comparison with beam search (Section 3) and retry upon regret (Result 1) suggests that simply exploring more trajectories from an error-free model is fundamentally limited. The model's probability distribution places insufficient mass on correct alternatives when it is in an error state. Search alone cannot compensate for a proposal distribution that does not represent correction alternatives. Research on improving verifiers should be paired with research on improving the generator's native correction capabilities, as this paper demonstrates for retry pretraining.

Follow-Up Research This Work Enables

Testing retry training on natural math reasoning data with LLM-generated errors. The paper's central claim—that pretraining on retry data improves reasoning accuracy—is validated exclusively on synthetic iGSM data with programmatically inserted skip errors. The most urgent follow-up is to test whether the same pattern holds on natural data (GSM8K, MATH) with errors generated by language models rather than by a dependency-graph oracle. A concrete experiment: take correct GSM8K solutions from a training split, prompt GPT-4 to generate realistic skip errors and corrections at controlled rates, use this as retry data for continued pretraining of a base model (e.g., Llama-3-8B), and compare accuracy against continued pretraining on the same number of error-free solution tokens. The paper's retry_weak result suggests that the errors need not be perfectly realistic—randomly reordering steps and marking them with a [BACK]-like token might suffice. The key measurement is whether the accuracy gains observed on iGSM (e.g., 78% → 95% on the hardest OOD problems) translate to even modest improvements on natural benchmarks, and whether the optimal retry_rate differs for natural data with noisier, more diverse error types.

Scaling retry training to larger models to test the LoRA failure hypothesis. The paper's finding that LoRA cannot teach error correction is demonstrated on GPT2-12-12 (~85M parameters). Larger models have stronger pretrained representations and different fine-tuning dynamics—a 7B-parameter model fine-tuned with LoRA on retry data might succeed where an 85M-parameter model fails. A critical scaling experiment: repeat the LoRA sweep from Section 5 (ranks from 4 to 256) on models at 1B, 7B, and 70B parameter scales, all pretrained on error-free data and fine-tuned on retry data with matched compute budgets. If the LoRA gap closes at larger scales, the paper's claim about error correction requiring "major weight changes" would be revised to "major weight changes at small scale, but learnable through PEFT at scale." If the gap persists, it would strengthen the claim's status as a general principle. A corollary experiment: test whether the gap between LoRA and full fine-tuning can be bridged by unfreezing specific layers (e.g., the last 25% of transformer blocks) while keeping LoRA on the rest—this would identify whether the bottleneck is in the frozen early layers' representations or in the overall capacity constraint.

Mechanistic interpretability of retry-trained models to understand what correction looks like internally. The paper establishes that retry training teaches error correction, but offers no analysis of how the model implements this behavior. The probing infrastructure from Section 3 provides a natural starting point: the can_next probe measures whether the model's hidden states encode the legality of the next parameter. Applied to retry-trained models at every generation step, this would reveal whether retry training changes the dynamics of the can_next signal—does it become stronger? Does it activate earlier (before the wrong parameter is fully generated)? Does the model represent correction alternatives in its hidden states before generating the [BACK] token? A more ambitious experiment: train probes on retry-trained models to decode which correction the model will generate after [BACK], tested before the [BACK] token is generated. If such probes achieve above-chance accuracy, it would demonstrate that the model plans its correction in advance, supporting the paper's claim that correction involves active strategy rather than random resampling. If they fail, correction might be a more reactive process triggered by the [BACK] token itself.

Curriculum learning with adaptive retry rates based on problem difficulty. The paper uses a fixed retry_rate throughout training. An open question is whether a curriculum approach—where retry_rate varies during training or is conditioned on problem difficulty—would be more token-efficient. A concrete experiment: during pretraining, estimate each problem's difficulty (e.g., by its op value in iGSM, or by the model's current loss on that problem) and apply higher retry_rates to harder problems. This tests whether the model benefits more from seeing correction patterns on problems at the edge of its current capability, where errors are most informative. A related experiment: sweep over schedules where retry_rate increases from 0 to some maximum over the course of training, testing whether early exposure to error-free solutions establishes correct reasoning patterns that later retry data builds upon. The paper's finding that higher retry rates monotonically improve accuracy suggests the current fixed-rate approach is not optimal—a curriculum might achieve the same final accuracy with fewer total tokens or higher final accuracy for the same token budget.

Retry data for error types beyond skip errors. The iGSM dataset isolates exactly one error type: generating a parameter whose dependencies are unsatisfied. Real reasoning errors are far more diverse. A diagnostic experiment: construct iGSM-like datasets with different error types—arithmetic mistakes (the computation is correct but uses a wrong operand), dependency-order errors (computing parameters in a valid but suboptimal order that leads to unnecessary computation), and semantic misinterpretations (using a parameter from the wrong entity type, e.g., "Film Studio's Backpack" vs. "Dance Studio's Backpack"). Train separate retry models for each error type and test cross-type generalization: does a model trained on skip-error retry data also correct arithmetic mistakes, or is the correction skill error-type-specific? This would clarify whether retry training teaches a general "be careful and correct mistakes" meta-skill or a narrow "check dependency legality and retry if violated" behavior. The paper's retry_weak result—where fake errors that aren't genuinely illegal still produce benefits—hints at some generality, but the experiment would make this precise.

Stress-testing retry training with noisy or imperfect corrections. All retry data in the paper contains perfectly correct corrections: the step following [BACK] is always the legally correct next parameter. Real training data would contain imperfect corrections—some corrections might themselves be wrong, some might fix one error but introduce another, and some might correct a step that wasn't actually wrong. A robustness experiment: corrupt a fraction of the corrections in retry data so that with probability q, the "correction" after [BACK] is itself an illegal parameter. Sweep q from 0 to 0.5 and measure how accuracy degrades. This would establish the noise tolerance of retry training and inform how carefully correction data needs to be curated in practice. If retry training remains beneficial even with 20-30% corrupted corrections, the practical barriers to adoption are much lower than if it requires near-perfect correction quality.

Practical Applications and Downstream Use Cases

Synthetic data generation for pretraining corpora of future LLMs. The paper's most directly actionable finding for large-scale LLM training teams is that error-correction patterns in pretraining data improve reasoning accuracy, and that these patterns can be constructed without perfect error identification (retry_weak). For organizations pretraining models on massive web corpora, this suggests a concrete data augmentation strategy: identify structured reasoning content in the corpus (math solutions, code explanations, step-by-step tutorials), programmatically reorder steps to create "fake error" sequences with explicit correction markers, and include this augmented data in the pretraining mix. The paper's numbers provide a rough calibration: on the hardest OOD problems, perfect retry data with retry_rate=0.5 improves accuracy from 78% to 95%, and retry_weak achieves 93-94%—a 15-16 percentage point absolute gain. Even if natural data yields smaller gains (say, 3-5 percentage points on benchmarks like GSM8K or MATH), the improvement from a purely data-side intervention with no architectural changes or inference-time overhead would be highly significant for production systems. The key implementation detail is that label masking is unnecessary for pretraining (Result 3), so retry-augmented data can be mixed into standard autoregressive training pipelines with zero modification to the training objective.

Self-improvement pipelines where models generate their own correction training data. The paper's findings directly inform the design of self-improvement or iterative refinement systems (such as STaR, ReST^EM, or rejection sampling fine-tuning). In these pipelines, a model generates solutions, some are identified as correct and used for further training, and the goal is to maximize the model's improvement per iteration. The paper suggests a specific augmentation: when a model generates a correct solution, construct retry data from it by inserting errors (either via retry_weak or by using a separate error-detection model to identify plausible mistakes) and include this in the next iteration's training data. This leverages the paper's finding that retry training on perfectly correct solutions with inserted errors is nearly as effective as training on naturally occurring errors, meaning the data generation step does not require the model to actually make mistakes—it only needs access to correct solutions, which the self-improvement pipeline already produces. The expected benefit is that each iteration not only teaches the model to produce more correct solutions but also teaches it to recover from the errors it still makes, potentially accelerating the self-improvement curve compared to training on error-free correct solutions alone.

Cost-efficient deployment of smaller models for structured reasoning tasks. The paper's comparison between retry pretraining and inference-time approaches has a direct economic implication: spending compute on retry data during pretraining may be more cost-effective than spending compute on beam search or multi-round prompting at inference time. A model pretrained on retry data achieves 95% accuracy on the hardest iGSM problems using greedy decoding (beam=1), while an error-free model achieves only 78% even with beam=32 (which costs 32x more inference compute per problem) or with retry-upon-regret using a perfect error detector and 50 retries. For applications where a model serves many queries, the pretraining cost is amortized and the per-query inference cost dominates. In such settings, investing in retry data during a one-time pretraining phase to enable cheap greedy decoding at inference time is economically preferable to deploying an error-free model that requires expensive multi-beam or multi-round generation to approach the same accuracy. This is particularly relevant for on-device deployment, edge computing, or high-throughput API services where per-token inference cost is the binding constraint.

When to Prefer This Method

The paper explicitly positions retry pretraining against two alternatives: (1) training on error-free data with inference-time error correction strategies (beam search, retry upon regret), and (2) post-hoc fine-tuning on retry data from an error-free pretrained model (LoRA, full fine-tuning). The tradeoffs are clear from the experimental results:

Prefer pretraining on retry data (with retry_rate in the 0.1–0.5 range, no label masking) when:

• You have control over the pretraining data composition and can generate or augment data with correction patterns. The paper shows that even fake corrections (retry_weak) are effective, so perfect error identification is not required.
• The target task involves structured, step-by-step reasoning where skip-type errors are common—mathematical problem solving, multi-step logical deduction, code generation with dependency constraints.
• Inference-time compute is constrained (e.g., greedy decoding preferred), since retry pretraining achieves its gains without requiring beam search or multi-round prompting at test time.
• The model will be trained from scratch or via continued pretraining with sufficient budget for deep weight changes, as LoRA fine-tuning cannot substitute for pretraining-scale optimization for this capability (Result 7).

Prefer error-free pretraining with inference-time correction strategies (beam search, retry upon regret, or multi-round prompting) when:

• You cannot modify the pretraining data (e.g., you are working with an existing pretrained model and can only control inference-time behavior).
• The error types in your task differ substantially from the skip-error pattern studied here, and you have not validated that retry data generalizes to your error distribution.
• The inference volume is low enough that per-query costs of beam search or multi-round prompting are acceptable, and the accuracy gains from retry pretraining (which are large on hard OOD problems but negligible on easy in-distribution problems in the paper) would not justify the pretraining data engineering effort.

Prefer full fine-tuning (continued pretraining) on retry data over LoRA fine-tuning when:

• You have an error-free pretrained model and sufficient computational budget for full-model training on retry data. The paper shows this can achieve accuracy comparable to direct retry pretraining, but requires twice the total training tokens (error-free pretraining + retry continued pretraining).
• The LoRA results (Figures 7, 10) demonstrate that even high-rank LoRA configurations fail to close the gap with full-model training for the error correction capability, so full fine-tuning or continued pretraining is the only viable post-hoc path.

The paper does not position retry pretraining against other methods like training a separate verifier model (Cobbe et al., 2021) or using reinforcement learning from human feedback for reasoning—these comparisons remain open questions.