STaR: Bootstrapping Reasoning With Reasoning

ArXiv: 2203.14465

🎯 Pitch

A 6B-parameter language model can teach itself to reason step-by-step, eventually matching a 30× larger model on commonsense QA—without any human-written rationales. It does this by using its own successful reasoning chains as training data and, crucially, also learning from its failures by generating explanations for the correct answer when it gets one wrong.

---

1. Executive Summary

This paper introduces the Self-Taught Reasoner (STaR), a bootstrapping method that iteratively improves a language model's ability to generate step-by-step rationales for complex reasoning tasks by learning from its own correct reasoning. Applied to GPT-J (6B parameters) on arithmetic, CommonsenseQA, and GSM8K, STaR combines rationale generation (few-shot prompting the model to produce reasoning chains, then fine-tuning on those that yield correct answers) with rationalization (providing the correct answer as a hint to generate rationales for problems the model initially fails). On CommonsenseQA, STaR achieves 72.5% accuracy—comparable to a 30× larger fine-tuned GPT-3 model (73.0%)—while training on only 86.7% of the dataset, establishing that a model can bootstrap its own reasoning capabilities from a small set of rationale examples without requiring massive human-annotated datasets, though only when its initial few-shot performance is above chance.

2. Context and Motivation

The Core Problem: Rationales Are Powerful But Expensive to Obtain

The central tension this paper addresses is deceptively simple: step-by-step reasoning dramatically improves language model performance, but the two main methods for getting models to produce such reasoning both have crippling limitations. The paper does not dispute the value of rationales — it accepts the growing consensus from works like Nye et al. (2021), Wei et al. (2022), and Rajani et al. (2019) that generating intermediate reasoning steps before a final answer consistently boosts accuracy across mathematical, commonsense, and symbolic reasoning tasks. The gap STaR fills is not whether rationales help, but how to obtain them at scale without prohibitive cost.

The practical stakes are significant. If rationale generation could be made cheap and automatic, it would unlock the reasoning benefits demonstrated by scratchpad methods (Nye et al., 2021) and chain-of-thought prompting (Wei et al., 2022) across an arbitrary number of domains without requiring teams of human annotators or hand-crafted templates for each new task. This matters for any application where logical deduction from world knowledge is required — math tutoring, scientific QA, legal reasoning, medical diagnosis support — and where constructing rationale datasets manually would be economically infeasible.

The theoretical motivation is equally compelling. The paper argues that language models pre-trained on vast corpora already contain latent reasoning capabilities, as evidenced by their non-trivial few-shot chain-of-thought performance. The problem is that these capabilities are difficult to elicit reliably without explicit fine-tuning on high-quality rationales. STaR proposes a mechanism to extract and amplify this latent reasoning without external supervision, effectively letting the model teach itself. This connects to broader questions in machine learning about bootstrapping: can a system improve its own capabilities by learning from its own successful outputs, and under what conditions does this self-reinforcing loop converge to better performance rather than stagnate or degrade?

Why Prior Approaches Fall Short

The paper identifies two dominant paradigms for rationale generation, each with a fundamental weakness that STaR is designed to circumvent:

1. Human-annotated rationale datasets (Rajani et al., 2019; Cobbe et al., 2021). The most direct approach is to pay humans to write out their reasoning for thousands of training examples, then fine-tune a model on those human-produced rationales. This demonstrably works: Rajani et al. showed that fine-tuning on human explanations improved commonsense reasoning, and Cobbe et al. provided human-written step-by-step solutions for grade-school math problems. The problem is cost. Each new domain requires a new dataset. Each dataset requires thousands of examples. Each example requires a human to articulate their reasoning process clearly and correctly. For the 12,247-question CommonsenseQA dataset alone, producing human rationales would be a substantial annotation effort; scaling this across dozens or hundreds of reasoning tasks is impractical. The paper characterizes this limitation bluntly in Section 1: "Manual methods are expensive, and it is infeasible to construct such a dataset for each interesting problem."

2. Few-shot chain-of-thought prompting (Nye et al., 2021; Wei et al., 2022). Rather than fine-tuning, one can simply include a handful of rationale examples in the model's prompt and rely on in-context learning to elicit reasoning behavior. This is essentially free — it requires only a small number of exemplar rationales, which the paper's authors themselves provide by lightly modifying 10 examples from Wei et al.'s prompts. The problem here is performance. While chain-of-thought prompting consistently outperforms direct prompting (asking for the answer without reasoning), it substantially underperforms models that have been fine-tuned on large datasets, even when those fine-tuned models only learn to predict answers directly without any reasoning. Table 1 quantifies this gap: on CommonsenseQA, few-shot CoT with GPT-J (6B parameters) achieves 36.6%, while GPT-J fine-tuned to directly predict answers — without any reasoning training — achieves 60.0%. Fine-tuned GPT-3 (175B parameters, ~30× larger) reaches 73.0%. The few-shot approach leaves the vast majority of the model's potential reasoning capability untapped.

These two approaches represent opposite ends of a cost-quality tradeoff curve. Human annotation is high-quality but expensive. Few-shot prompting is cheap but low-quality. The paper's goal is to establish a third point on this curve: high-quality rationale generation with negligible human annotation cost, achieved by letting the model generate its own training data.

Conflicting Evidence on Self-Improvement and Explanation Quality

The paper enters a landscape where the relationship between explanations and model performance is not straightforward. Prior work on explainable AI (Camburu et al., 2018; Chen et al., 2021) focused on producing natural language justifications for model decisions, but primarily from the perspective of interpretability rather than performance improvement. Rajani et al. (2019) explicitly note that "post-hoc explanations" — generating a justification after selecting an answer — generally do not improve model accuracy. This creates a puzzle: if explanations don't help when generated after the fact, why would training on them help at all?

The resolution, which STaR implicitly relies on, is that rationales generated before the answer constrain the answer generation process itself. When a model produces a chain of reasoning first and then derives an answer from that reasoning, the intermediate steps shape the probability distribution over final answers. This is fundamentally different from generating an answer and then explaining it. The paper frames this through a latent variable lens (Section 3.1): the model's output distribution can be decomposed as $p(y | x) = \sum_r p(r | x)p(y | x, r)$, where the rationale $r$ is a latent variable that mediates the mapping from input $x$ to output $y$. Training on correct rationales improves the model's ability to sample $r$ values that lead to correct $y$, which directly improves overall accuracy.

This framing also explains why the filtering step in STaR is critical. The model generates many rationales during each iteration; only those that result in the correct final answer are kept for training. Discarding incorrect rationales prevents the model from learning reasoning patterns that, while fluent and plausible, systematically lead to wrong conclusions. This is a form of weak supervision: the ground-truth answer serves as a binary correctness signal that labels the rationale as useful or not, without requiring any human to evaluate the reasoning quality directly.

Where Existing Bootstrapping Approaches Leave Gaps

STaR did not emerge from a vacuum of self-improvement techniques. The paper explicitly connects to a lineage of iterated learning and expert iteration methods:

Expert Iteration (ExIt, Anthony et al., 2017) is a reinforcement learning framework where an "apprentice" improves through self-play and imitation learning from a slower "expert," with the expert being periodically replaced by the now-improved apprentice. STaR can be interpreted as a simplified ExIt where filtering generated examples based on answer correctness serves as "expert feedback." The critical difference is that STaR has a fixed expert — the ground-truth answer checker — and does not require training a separate value function. This simplification is both a strength (it makes STaR easy to implement with standard fine-tuning machinery) and a limitation (it cannot improve on problems where the model never stumbles upon the correct answer, since those examples provide no training signal).

Formal mathematics curriculum learning (Polu et al., 2022) applied ExIt-style bootstrapping to theorem proving in Lean, but only worked because (1) proofs could be mechanically verified for correctness, (2) numerous candidate solutions could be sampled for each problem, (3) a separate value function model was trained to guide search, and (4) the base model (GPT-f) was already fine-tuned on a large mathematical dataset. The STaR authors explicitly note that "there are many domains where these conditions do not all apply." Their goal is a bootstrapping method that works with ordinary natural language reasoning tasks, a standard pretrained language model, and no value function or formal verifier.

Iterated learning for visual question answering (Vani et al., 2021) used compositional modular networks that could be combined in structured ways. This approach was domain-specific and architecture-specific; STaR aims for generality across reasoning domains and compatibility with any standard autoregressive language model.

The Specific Gap STaR Fills

The gap that STaR occupies is therefore: a general-purpose bootstrapping method that requires (a) a small set of exemplar rationales, (b) a dataset of questions with ground-truth answers but no rationales, and (c) a pretrained language model with non-trivial few-shot reasoning capability, and that produces (d) a fine-tuned model that generates high-quality rationales and achieves accuracy competitive with models trained on human-annotated reasoning data. No prior method simultaneously satisfies all four desiderata.

The paper's key insight about why this works comes from the observation that few-shot chain-of-thought performance, while low in absolute terms compared to fine-tuned models, is above random for tasks where the model possesses relevant knowledge. On arithmetic, few-shot accuracy is near zero for multi-digit addition — but on CommonsenseQA, few-shot CoT achieves 36.6% versus 20% random chance. This non-trivial baseline provides enough correct reasoning examples to seed the first iteration of fine-tuning. Each subsequent iteration generates slightly better rationales, which in turn provide better training data, creating a virtuous cycle. The process can be understood as gradually shifting the model's rationale distribution toward higher-quality regions of the output space, using only the binary feedback of whether the final answer matches the ground truth.

How the Paper Positions Itself

The paper explicitly frames STaR not as a new reasoning architecture or a better prompting technique, but as a meta-method for dataset construction. Section 1 states the contribution directly: "We propose a bootstrapping mechanism to iteratively generate a rationale dataset from a few initial examples with rationales — without needing to check new rationales' correctness." The key phrase is "without needing to check new rationales' correctness" — the method never requires a human or an automated verifier to evaluate whether a generated reasoning chain is logically valid. It only uses the ground-truth answer, which is assumed to be available as part of the original question dataset. This is what makes the approach scalable: acquiring questions with answers is far cheaper than acquiring questions with detailed step-by-step solutions.

The paper also positions rationalization — providing the correct answer as a hint and asking the model to generate a backwards justification — as a novel contribution that addresses a specific failure mode of the naive bootstrapping loop. Without rationalization, the model only trains on problems it can already solve, and "improvement ends when the model fails to solve new problems in the training set" (Section 3.2). Rationalization breaks this deadlock by forcing the model to reason about problems it initially gets wrong, using the answer as a scaffold. The paper's ablation studies (Table 1) show that rationalization adds 3.7 percentage points on CommonsenseQA (68.8% → 72.5%) and provides qualitatively different training examples that expose the model to harder problems.

A subtle but important positioning choice: the paper explicitly avoids claiming that STaR produces faithful rationales — ones that accurately represent the model's internal reasoning process. The discussion of limitations in Appendix G acknowledges that "it is difficult, if not impossible, to ensure that the rationales reflect the model's internal processing." The model might select an answer immediately based on shallow patterns and then generate a post-hoc justification. However, the empirical results (improved accuracy over direct answer prediction, and the ablation studies from Wei et al. showing that rationale generation non-trivially improves answer quality) demonstrate that the rationales are doing some useful computational work, even if their relationship to the model's internal reasoning is complex. This is an honest boundary to draw: STaR improves task performance through learned reasoning patterns, but makes no strong claims about interpretability.

The Computational Constraints Shaping the Design

The paper's design choices reflect practical computational constraints that are worth highlighting because they influence the method's scope. All experiments use GPT-J (6B parameters), chosen explicitly because "the checkpoint and fine-tuning code are publicly available" and the model is "large enough to generate rationales of non-trivial quality to be bootstrapped from" (Section 4.1). The authors note they were limited to a single TPU-v3 node for both training and sampling. This constraint motivated several efficiency-oriented design decisions: training from the original pretrained model at each iteration rather than continuing to train one model (to avoid overfitting), using packing to combine shorter examples into sequences up to 1024 tokens (to improve TPU utilization), and limiting hyperparameter search. The method was therefore designed to work within academic-scale compute budgets, making it more accessible to researchers without industrial resources.

A notable limitation acknowledged in the paper is that for the first iteration to succeed, "few-shot performance must be above chance, implying that the initial model must be big enough to have some reasoning capabilities." The paper reports that GPT-2 was unable to bootstrap from few-shot reasoning even in arithmetic, establishing a minimum capability threshold for the base model. This places STaR in a regime where the model already has some reasoning ability — the method amplifies existing capability rather than creating it from nothing — which parallels a key theme in more recent test-time compute scaling research.

3. Technical Approach

3.1 Reader Orientation

The Self-Taught Reasoner (STaR) is a training loop that takes a standard pretrained language model, a dataset of questions with ground-truth answers (but no reasoning traces), and a handful of example rationales, and produces a fine-tuned model that generates step-by-step reasoning chains to solve problems. The problem it solves is the bottleneck of rationale data acquisition: instead of paying humans to write thousands of reasoning traces or accepting the low accuracy of few-shot prompting, STaR lets the model generate its own training data by attempting to solve problems, keeping only the reasoning that leads to correct answers, and then using that reasoning to improve itself through fine-tuning—repeating this process until the model stops getting better.

3.2 Big-Picture Architecture (Diagram in Words)

The STaR system has four major components arranged in an outer loop:

1. A pretrained language model $M$ (GPT-J, 6B parameters) that can be both few-shot prompted and fine-tuned. This model is the thing being improved; it starts with some latent reasoning ability from pretraining.
2. A small prompt set $P$ of 10 examples, each containing a question, a human-written (or lightly edited) step-by-step rationale, and the correct answer. These examples teach the model the format and style of rationale generation through in-context learning.
3. A large dataset $D$ of questions with ground-truth answers but no rationales (e.g., 50,000 arithmetic problems, 9,741 CommonsenseQA training questions, 7,473 GSM8K problems). The model attempts to solve every question in this dataset; only the successful attempts become training data.
4. A fine-tuning mechanism that takes the collected correct rationales and retrains the original pretrained model on them, producing an improved model for the next iteration.

Information flows in a loop: the current model attempts to solve all questions in $D$ via few-shot prompting (rationale generation) → correct solutions are collected into a training set → the model is reset to its original pretrained weights and fine-tuned on the collected rationales → the fine-tuned model becomes the new model for the next iteration → the loop repeats until performance plateaus. A parallel rationalization path also runs: for questions the model gets wrong, the correct answer is injected as a hint into the prompt, and the model generates a backwards justification; these rationalized solutions are also added to the training set.

3.3 Roadmap for the Deep Dive

• First, the policy gradient framing (Equation 1) connects STaR to a formal reinforcement learning objective, showing why filtering by answer correctness approximates a valid gradient.
• Second, the rationale generation loop (Algorithm 1 without rationalization) walks through the core bootstrapping mechanism, including how rationales are generated, filtered, and used for fine-tuning.
• Third, rationalization (the blue parts of Algorithm 1) explains how the model generates backward justifications for problems it initially fails, and why this breaks the stagnation problem.
• Fourth, the restart-from-scratch training protocol details the critical design choice to retrain from the original pretrained model each iteration rather than continuing to train one model.
• Fifth, the specific implementations per domain cover how these abstract mechanisms are instantiated for arithmetic, CommonsenseQA, and GSM8K, including task-specific prompt designs and hyperparameter choices.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a bootstrapping method paper whose core idea is that a language model with non-trivial few-shot reasoning ability can generate its own rationale training data, and the quality of that data—and therefore the model's reasoning ability—improves with each iteration of self-training, as long as the model is periodically exposed to harder problems through rationalization.

---

The Policy Gradient Framing

The paper provides a formal justification for why STaR works by connecting it to a reinforcement learning objective. This framing is not used algorithmically (no actual policy gradient is computed), but it explains the conceptual foundation for the filtering step.

The language model $M$ is viewed as a latent variable model:

$p_M(y | x) = \sum_r p_M(r | x) \, p_M(y | x, r)$

where $x$ is a question, $y$ is an answer, and $r$ is a rationale—a sequence of intermediate reasoning tokens. The model first samples a rationale $r$ from its distribution over reasoning chains conditioned on the question, then samples an answer $y$ conditioned on both the question and the rationale.

What it computes: This decomposition expresses the answer probability as a marginalization over all possible rationales. In practice, the model generates these jointly (the rationale tokens precede the answer tokens autoregressively), but the decomposition separates the two stages conceptually.

Why this form: This latent-variable view matters because it shows that improving rationale quality (making $p_M(r | x)$ concentrate on rationales that lead to correct answers) directly improves answer accuracy. It also clarifies why filtering is legitimate: we cannot marginalize over all rationales (that would be prohibitively expensive), so we approximate by sampling.

Now, define a reward function—the indicator that the generated answer matches the ground truth:

$\mathbf{1}(\hat{y} = y)$

where $\hat{y}$ is the model's predicted answer and $y$ is the ground-truth correct answer from the dataset. The total expected reward across the dataset $D = \{(x_i, y_i)\}_{i=1}^D$ is:

$J(M, X, Y) = \sum_i \mathbb{E}_{\hat{r}_i, \hat{y}_i \sim p_M(\cdot | x_i)} \left[ \mathbf{1}(\hat{y}_i = y_i) \right]$

where $\mathbb{E}_{\hat{r}_i, \hat{y}_i \sim p_M(\cdot | x_i)}$ denotes the expectation over rationales and answers sampled from the model conditioned on $x_i$.

What it computes: This is the expected number of correct answers across the dataset when we sample from the model's rationale-then-answer distribution. It sums over all questions, and for each question, averages the binary correctness indicator over the model's sampling distribution.

Why this form: This objective directly captures what we care about—getting correct answers—without requiring any assessment of rationale quality beyond whether the final answer is right. The indicator function is a natural choice because ground-truth answers are binary correct/incorrect for the tasks we study (arithmetic, multiple-choice QA, math problems with numeric answers).

Taking the gradient of this objective yields:

$\nabla J(M, X, Y) = \sum_i \mathbb{E}_{\hat{r}_i, \hat{y}_i \sim p_M(\cdot | x_i)} \left[ \mathbf{1}(\hat{y}_i = y_i) \cdot \nabla \log p_M(\hat{y}_i, \hat{r}_i \mid x_i) \right]$

What it computes: This is the standard policy gradient (REINFORCE) gradient for the expected reward. For each sampled rationale-answer pair, the gradient contribution is the log-probability gradient multiplied by the binary reward. When the reward is 1 (correct answer), the model receives a positive gradient that increases the probability of that sampled rationale-answer sequence. When the reward is 0 (incorrect answer), the gradient contribution is zeroed out.

Why this form: The log-derivative trick transforms the problem of differentiating through a sampling operation into an expectation over log-probability gradients weighted by rewards. Crucially, the indicator function acts as a filter: it discards the gradient for all sampled rationales that do not lead to the correct answer. This is exactly the filtering step in STaR (Algorithm 1, Line 5).

STaR approximates this policy gradient in two ways:

1. Greedy decoding instead of sampling: Rather than sampling multiple rationales and computing a stochastic gradient estimate, STaR generates one rationale per question (using greedy decoding or low-temperature sampling) and accepts it if it yields the correct answer. This reduces variance at the cost of potentially biased exploration—the model only explores rationales it is already confident about.

2. Multiple gradient steps on the same batch: Instead of taking one gradient step and re-sampling, STaR collects a batch of correct rationales and fine-tunes on that batch for multiple epochs (40 steps in the first iteration, increasing by 20% each subsequent iteration). This is similar to how some policy gradient algorithms (like PPO) perform multiple update steps on the same data.

The paper explicitly notes that these approximations "make STaR a simple and broadly applicable method that can be implemented with standard LLM training machinery" and that "future work should more closely investigate the link between STaR and the RL objective above."

---

Rationale Generation Bootstrapping (STaR Without Rationalization)

The core loop of STaR—what the paper calls "rationale generation bootstrapping"—is a simple iterative process that alternates between generating rationales and fine-tuning on the successful ones. Here is the precise sequence of operations, corresponding to Algorithm 1 with the rationalization parts removed (and reproduced explicitly as Algorithm 2 in Appendix E):

Step 1: Initialize. Start with the pretrained model $M$ and a dataset $D = \{(x_i, y_i)\}_{i=1}^D$ consisting of questions $x_i$ and their ground-truth answers $y_i$. Also prepare a small prompt set $P$ of $P$ examples with rationales: $P = \{(x_i^p, r_i^p, y_i^p)\}_{i=1}^P$ where $P \ll D$. For CommonsenseQA and GSM8K, $P = 10$; for arithmetic, 10 examples per digit length (1 through 5 digits).

Step 2: Construct prompts. For each question $x_i$ in the dataset, concatenate the entire prompt set in front of it, producing an input sequence like:

$x_i^{\text{prompted}} = (x_1^p, r_1^p, y_1^p, \ldots, x_P^p, r_P^p, y_P^p, x_i)$

where the parentheses denote concatenation of token sequences. This is standard few-shot in-context learning: the model sees several examples of the format "Question → Rationale → Answer" and is then asked to continue the pattern for the new question $x_i$.

Step 3: Generate rationales. The current model $M_{n-1}$ (where $n$ is the outer loop iteration, and $M_0 = M$ is the original pretrained model) is prompted with each $x_i^{\text{prompted}}$. The model autoregressively generates a rationale $\hat{r}_i$ followed by an answer $\hat{y}_i$. In practice, the model is set to produce the rationale tokens first and then derive the answer, following the structure demonstrated in the few-shot prompts. The paper notes that sampling is done with low temperature (greedy decoding or near-greedy; the default temperature in the GPT-J fine-tuning script is used, which is effectively greedy) because higher temperatures were found to increase the likelihood of correct answers despite incorrect reasoning, which would corrupt the training data.

Step 4: Filter by answer correctness. For each generated $(\hat{r}_i, \hat{y}_i)$ pair, compare the predicted answer $\hat{y}_i$ to the ground-truth answer $y_i$. Keep only the examples where the answers match:

$D_n = \{(x_i, \hat{r}_i, y_i) \mid i \in [1, D] \text{ and } \hat{y}_i = y_i\}$

The rationale $\hat{r}_i$ is paired with the ground-truth answer $y_i$ (not the predicted answer—though they match by construction since these are the filtered examples). The key assumption, stated explicitly in Section 3.1, is that "rationales that lead to correct answers are of better quality than those that lead to incorrect answers." This is not always true—the model might arrive at the correct answer through faulty reasoning—but is a reasonable heuristic that works on average.

Step 5: Fine-tune from scratch. Take the original pretrained model $M$ (not $M_{n-1}$) and fine-tune it on $D_n$ using standard language model training (next-token prediction on the concatenated sequence $[x_i, \hat{r}_i, y_i]$). The model is trained to predict the rationale and answer tokens conditioned on the question tokens and the preceding rationale tokens. The paper uses the GPT-J fine-tuning script with a learning rate of $1 \times 10^{-6}$ (selected via hyperparameter search over $10^{-7}$ to $10^{-4}$ using Adam), batch size of 8 sequences (with packing to fill sequences up to 1024 tokens), no weight decay, and the Adam optimizer with default GPT-J settings. Training starts with 40 steps in the first outer loop iteration, and the number of training steps increases by 20% with each subsequent outer loop iteration. A 100-step learning rate warmup is used, after which the learning rate remains constant.

Step 6: Repeat. Set $M_n$ to the newly fine-tuned model. Go back to Step 3 and use $M_n$ to generate new rationales. The loop continues until performance on the training set plateaus—specifically, until the model stops solving new problems it previously failed to solve.

Why restart from the original pretrained model each iteration (Step 5): This is a critical design choice. Instead of continually fine-tuning one model across iterations (which would compound distribution shift as the model's outputs drift from its training data), the paper resets to the original pretrained checkpoint $M$ at the start of each iteration's fine-tuning. This prevents overfitting to the model's own generated rationales, which the paper found could cause degradation. The training data $D_n$ changes each iteration (because the improved model generates different—presumably better—rationales), but the base weights are always the same pretrained parameters. This means each iteration is comparing the newly fine-tuned model against a fixed baseline, making the effect of the updated training data cleanly measurable.

What overfitting means in this context: The paper notes that this restart protocol avoids "overfitting," but this is a specific type of overfitting: the model overfitting to its own generation artifacts. If you continually fine-tune one model on its own outputs, errors in the model's reasoning style can become amplified—the model learns to produce rationales that look like training examples but don't actually improve answer accuracy. By restarting from the original pretrained model, you get a fresh initialization that can learn from the improved data without accumulating generation artifacts.

The stagnation problem: Even with these mechanics, the loop described so far has a fundamental limitation. Since $D_n$ only contains questions the model answered correctly, and the model only improves by training on $D_n$, the set of solvable problems can never grow beyond what the model already solves. The paper states this explicitly: "Since the model is only trained on the examples which it answers correctly, improvement ends when the model fails to solve new problems in the training set." At that point, each iteration generates the same set of correct answers, produces the same training set, and fine-tunes to the same model—no further improvement is possible. This is exactly the deadlock that rationalization is designed to break.

---

Rationalization: Generating Rationales Backwards From Answers

Rationalization is the mechanism that allows STaR to extract training signal from problems the model initially gets wrong. The key insight is that generating a rationale given the correct answer (reasoning backwards) is often easier than generating both the correct rationale and the correct answer (reasoning forwards). The paper frames this in terms of conditional distributions: rationale generation samples from $p(r \mid x)$, while rationalization samples from $p(r \mid x, y)$. The latter distribution may be "a better search space for rationales" because conditioning on the answer constrains the rationale to be consistent with a known correct conclusion.

Step R1: Identify failures. After the rationale generation step (Step 3 in the main loop), identify the subset of questions where the model's predicted answer $\hat{y}_i$ does not match the ground truth $y_i$: $D^{\text{failed}} = \{i \mid \hat{y}_i \neq y_i\}$.

Step R2: Inject hints. For each failed question, construct a new prompt that includes the correct answer as a hint. The paper's approach is simple: they append the correct answer to the question in the prompt context. For arithmetic, this means including the correct numeric answer after the "Target:" line and asking the model to produce the scratchpad (intermediate computation steps) and then reproduce the answer. For CommonsenseQA, this means stating which multiple-choice option is correct (e.g., "(b) grocery cart" is the correct answer) and asking the model to generate a rationale. Figure 2 in the paper provides an example where the prompt explicitly says "Answer Choices: ... (b) grocery cart (CORRECT)" and then asks the model to generate the rationale justifying why that answer is correct.

This hinting approach is simple but not always straightforward. The paper acknowledges in Section 5 that "the method to add the 'hint' does not follow immediately from the question and answer and in some contexts providing it may be nontrivial." For example, on GSM8K, the hint format must match the expected input structure—the model needs to see the correct numeric answer in context and be prompted to generate the steps leading to it. The exact format is included in the problem-specific sections described below.

Step R3: Generate rationalized rationales. The current model $M_{n-1}$ is prompted with the hint-augmented questions and generates rationales $\hat{r}_i^{\text{rat}}$ followed by answers $\hat{y}_i^{\text{rat}}$. The model is asked to produce a rationale in the same style as the normal rationale generation examples—the hint is present only during generation, not in the final training example.

Step R4: Filter by correctness again. Keep only the rationalized examples where the model's answer again matches the ground truth:

$D_n^{\text{rat}} = \{(x_i, \hat{r}_i^{\text{rat}}, y_i) \mid i \in D^{\text{failed}} \text{ and } \hat{y}_i^{\text{rat}} = y_i\}$

Note that the hint in the prompt makes it more likely the model will output the correct answer, but it is not guaranteed—the model might ignore the hint or produce an answer that contradicts it. Filtering ensures only consistent examples enter the training set.

Step R5: Strip the hints for training. When adding these rationalized examples to the training set, the hint is not included in the prompt. The training example looks identical to a naturally generated rationale example: $(x_i, \hat{r}_i^{\text{rat}}, y_i)$. This is crucial because it means the fine-tuned model learns to produce the rationale as if it had reasoned forward to the answer without any hint. The paper describes this as training "as if the model had come up with the rationale without the hint" (Section 3.2).

Step R6: Combine and fine-tune. The final training set for iteration $n$ is the union of the naturally generated correct rationales and the rationalized correct rationales:

$D_n^{\text{combined}} = D_n \cup D_n^{\text{rat}}$

The model is fine-tuned on this combined set, still restarting from the original pretrained model $M$.

Why rationalization has two distinct benefits (Section 3.2):

The paper identifies a primary benefit of exposure to harder problems: rationalization "exposes the model to difficult problems which otherwise would not have appeared in its fine-tuning dataset." Without rationalization, the model never trains on problems it cannot solve, so its capabilities are capped at whatever it could do few-shot. Rationalization forces the model to learn from—and therefore eventually learn to solve—problems that were initially beyond its reach.

The paper identifies a secondary benefit of increased dataset size: rationalization adds more training examples to each iteration. For CommonsenseQA, rationalization adds examples covering 8.5% of the training set that rationale generation missed, bringing total coverage from 69.7% to 86.7% (Table 1).

How rationalization connects to off-policy RL (Section 5): The paper sketches a more formal interpretation: "rationalization could be framed as an off-policy estimate of the objective in Equation 1, sampling from the hint-augmented model as a proposal distribution." In standard policy gradient, the gradient estimate uses samples from the current policy. In off-policy policy gradient, you can use samples from a different (proposal) distribution and correct with importance weights. Here, the hint-augmented model $p_M(r \mid x, y)$ is the proposal distribution—it generates rationales that are more likely to lead to correct answers than $p_M(r \mid x)$. The paper does not actually compute importance weights (it simply filters and trains), making this a heuristic approximation, but the conceptual connection explains why rationalization can be effective even though it changes the sampling distribution.

The low-temperature choice and its relationship to rationalization (Section 5): The paper tested an alternative to rationalization: generating multiple high-temperature samples per question and keeping any that yield correct answers. This seems like a natural way to expand the training set without rationalization. However, the paper found this was "counterproductive" because higher temperatures "substantially increase the likelihood of a correct answer despite incorrect reasoning, and training on bad or irrelevant reasoning prevents generalization." In structured tasks like arithmetic, high-temperature scratchpads "diverge into meaninglessness and cause the model to stagnate." Rationalization avoids this problem because conditioning on the correct answer constrains the rationale to be at least superficially consistent with the correct conclusion, reducing the rate of nonsensical reasoning that happens to match the answer by chance.

The paper also notes that high-temperature sampling is computationally wasteful: "generating 10 sample outputs is approximately 10 times slower than generating one sample output." Rationalization generates one rationale per failed question (at low temperature), making it far more efficient per training example obtained.

---

The Restart-from-Scratch Training Protocol

The training protocol at each fine-tuning iteration contains several important details beyond the high-level loop structure:

Training from the original pretrained model: At each outer loop iteration $n$, the model is initialized from the same pretrained GPT-J checkpoint $M$, not from $M_{n-1}$. The paper describes this as "train from the original pre-trained model M instead of continually training one model to avoid overfitting" (Section 3.1). This means the only thing that changes across iterations is the training dataset $D_n^{\text{combined}}$, not the model initialization.

Progressive increase in training steps: The number of fine-tuning steps increases with each outer loop iteration. The paper states: "we start with 40 training steps at the first outer loop, and increase the number of fine-tuning training steps by 20% with each outer loop." So iteration 1 uses 40 steps, iteration 2 uses $40 \times 1.2 = 48$ steps, iteration 3 uses $48 \times 1.2 \approx 58$ steps, and so on. The motivation is that "training more slowly at the beginning ultimately benefits model performance"—early iterations have lower-quality training data (rationales generated by the weaker model), so overtraining on them would be harmful.

Constant learning rate with warmup: The learning rate is constant at $1 \times 10^{-6}$ after a 100-step warmup period. The warmup linearly increases the learning rate from 0 to $1 \times 10^{-6}$ over the first 100 steps. For iterations with fewer than 100 training steps (the early iterations), this means the effective learning rate is lower than $1 \times 10^{-6}$ for most of training.

Packing for efficiency: The paper uses "packing, namely, packing the shorter examples to form longer sequences (up to length 1024) to improve TPU utilization." When a training example (question + rationale + answer) is shorter than 1024 tokens, multiple examples are concatenated into one training sequence separated by appropriate delimiters. This maximizes the number of tokens processed per training step on TPU hardware.

Batch size: 8 sequences of length 1024 each per training step. With packing, a single training step processes many more individual questions than 8.

No weight decay: The Adam optimizer is used without weight decay, a departure from many standard fine-tuning recipes that add L2 regularization. The paper does not discuss this choice explicitly, but it likely reflects the small number of training steps—with only 40-100 steps per iteration, regularization is less critical than with longer training runs.

Maximum iterations: The loop runs until performance plateaus. For arithmetic, this was 16 iterations without rationalization and approximately 36 iterations with rationalization. For CommonsenseQA, 36 iterations for STaR without rationalization and an additional 10 iterations with rationalization. For GSM8K, 36 iterations without rationalization and 10 more with rationalization. The paper reports results at the iteration with the best validation performance.

---

Arithmetic: Task-Specific Implementation

The arithmetic task is to compute the sum of two $n$-digit integers, for $n \in \{1, 2, 3, 4, 5\}$ (and later extended to 9 and 10 digits in an auxiliary experiment). The paper follows the scratchpad format introduced by Nye et al. (2021).

Data generation: A dataset of 50,000 randomly sampled addition problems is generated, with digit lengths sampled uniformly. For each outer loop iteration, 10,000 problems are randomly sampled from this dataset. The problems are in the format "6 2 4 + 2 5 9" with spaces between digits, and the target answer is a number like "883."

Scratchpad format: The model is asked to produce a structured scratchpad enclosed in <scratch> and </scratch> tags, followed by the final answer. Each line of the scratchpad corresponds to summing one pair of digits from right to left. For the example "6 2 4 + 2 5 9":

• Line 1: "6 2 4 + 2 5 9 , C: 0" — initial setup with carry 0
• Line 2: "2 + 5 , 3" — sum the rightmost digits (4+9=13), write down 3
• Line 3: "C: 1" — carry the 1
• Line 4: "6 + 2 , 8 3" — sum the next digits with carry (2+5+1=8), write 83 so far
• Line 5: "C: 0" — no carry
• Line 6: " , 8 8 3" — sum the leftmost digits (6+2=8)
• Line 7: "C: 0" — no carry
• Line 8: "0 8 8 3" — pad to match digit count

The final answer is "8 8 3" (without leading zeros).

Few-shot prompts: For each digit length (1 through 5), 10 random few-shot examples are included in the prompt. The few-shot examples follow the scratchpad format exactly, providing the model with the expected input-output structure.

Rationalization on arithmetic: When rationalizing, the correct answer is provided after "Target:" in the prompt, and the model is asked to generate the scratchpad that leads to that answer, and then reproduce the answer after the scratchpad. Because arithmetic scratchpads are fully determined by the operands and the answer (the carry digits follow deterministically), rationalization on arithmetic essentially asks the model to reconstruct the intermediate computation that must have produced the given sum.

Accuracy measurement: Accuracy is measured per-digit-length, meaning a 3-digit addition problem is correct only if all digits of the answer are correct. The paper reports accuracy curves for each digit length across iterations (Figure 4).

Baseline: A model trained on 10,000 arithmetic examples without scratchpads (direct answer prediction) for 5,000 steps achieves 76.3% accuracy. STaR after 16 iterations achieves 89.5%.

---

CommonsenseQA: Task-Specific Implementation

CommonsenseQA (CQA) is a 5-way multiple-choice commonsense reasoning dataset. Each question presents a short scenario and five possible answers; the model must select the single best answer. The dataset has 9,741 training questions, 1,221 dev questions, and 1,285 test questions (the test set labels are withheld, so evaluation is on the dev set).

Few-shot prompts: The paper uses 10 few-shot examples, adapted from the chain-of-thought prompts in Wei et al. (2022). The authors "modified [the prompts] slightly to fix an incorrect answer and to more explicitly reference relevant knowledge." The modifications include fixing a typo in one answer and making the rationales more explicit about the knowledge being applied. The full set of 10 modified prompts is reproduced in Appendix B.

Rationale format: The model is expected to produce a rationale in the format established by the few-shot examples: "The answer must be [property that the correct answer should have]. [Reasoning applying world knowledge]. Therefore, the answer is [answer text] ([answer letter])." For example: "The answer must be something that can be used to carry a small dog. Baskets are designed to hold things. Therefore, the answer is basket (b)."

Rationalization on CQA: When the model answers a question incorrectly, the rationalization prompt includes the correct answer as a hint. Figure 2 shows the format: the answer choices list the correct option with "(CORRECT)" appended, and the model is prompted to generate a rationale. The paper uses the same 10 few-shot prompts during rationalization as during rationale generation, with the hint added to the question portion of the prompt (not to the few-shot examples). The hint is present during generation but stripped from the training example.

Few-shot prompt inclusion during training: The paper makes a nuanced choice about whether to include the few-shot prompts during fine-tuning. Including them means the training examples look exactly like the generation prompts—the few-shot examples are prepended. Excluding them means the model is fine-tuned on just the question-rationale-answer triples, without the few-shot prefix. The paper reports that including few-shot prompts has "a meaningful performance benefit" (60.9% to 68.8% without rationalization, 69.9% to 72.5% with rationalization), and they "generally suggest its use for at least some portion of the training." The benefit likely comes from reducing distribution shift: if the model is fine-tuned with the few-shot prefix in context, it learns to generate rationales in the same context it will see at test time.

Evaluation: The model's predicted answer is extracted from the generated text by parsing the final answer letter or answer text. Accuracy is the fraction of questions on the CQA dev set (1,221 questions) where the extracted answer matches the ground truth. The main results (Table 1) report dev set accuracy.

Human evaluation of rationale quality (Section 4.4, Appendix C): To assess whether STaR improves rationale quality beyond just answer accuracy, the authors conducted a small human evaluation. They randomly selected 50 questions that both few-shot CoT and STaR (without rationalization) answered correctly. For each question, they showed crowdworkers on Prolific three rationales in random order: the few-shot CoT rationale, the STaR-generated rationale, and a human-written rationale from the dataset of Rajani et al. (2019). Each of 20 crowdworkers ranked 10 random question-rationale triples. The participants were 30% more likely to rank the STaR rationales higher than the few-shot rationales (p = 0.039) and 74% more likely to prefer STaR rationales over human-written rationales (p < 0.001). The paper is careful to note the limitations: the human-written rationales from Rajani et al. sometimes contained uninformative text (e.g., "The only answer that makes sense," "This word was most relevant," or restatements of the answer), making them a weak baseline for human reasoning quality. The paper clarifies that this result "speaks to the difficulty of eliciting high-quality rationales" rather than demonstrating superhuman reasoning.

Few-shot CoT baseline discrepancy: The paper reports a few-shot CoT accuracy of 36.6% on the CQA dev set, while Wei et al. (2022) reported 55.6% for a 137B LaMDA model—a much larger model. The GPT-J few-shot CoT accuracy is significantly higher than random (20%), which is sufficient for STaR to bootstrap from, but the gap to the larger model illustrates how much headroom exists for improvement through self-training.

---

Grade School Math (GSM8K): Task-Specific Implementation

GSM8K consists of grade-school-level math word problems expressed in natural language, requiring 2-8 calculation steps. The dataset has 7,473 training examples and 1,319 test examples.

Few-shot prompts: The paper uses 6 few-shot examples adapted from Cobbe et al. (2021), reproduced in Appendix I. Each example shows a problem statement, a step-by-step solution with intermediate calculations in <<calculations>> format, and a final answer marked with ####.

The <<...>> notation is a format convention from the original GSM8K dataset where intermediate calculator operations are enclosed in double angle brackets: for example, <<48/2=24>> indicates the computation "48 divided by 2 equals 24." This makes the arithmetic steps explicit and machine-parseable, though the model learns to produce these purely through text generation—no actual calculator is invoked.

Rationale format: The model generates a free-text rationale that walks through the problem, performing calculations in <<...>> format and ending with #### [answer]. The few-shot examples demonstrate variable-length reasoning chains, from 1-2 steps to 7+ steps.

Training details: The paper caps the number of total training steps at 7,912 (reached by the 30th iteration) "to prevent the training process from becoming prohibitively long." Results for GSM8K are reported after 36 iterations without rationalization, and an additional 10 iterations with rationalization. The test accuracy metrics use the full 1,319-question test set.

Rationalization on GSM8K: The paper found that rationalization did not substantially improve GSM8K performance compared to rationale generation alone. Table 2 shows 10.1% without rationalization and 10.7% with rationalization—only a 0.6 percentage point gain, compared to the 3.7 point gain on CQA. The paper suggests this may be because providing the answer as a hint on GSM8K is less helpful: the reasoning chain is the main challenge, and seeing the final number doesn't necessarily make it easier to reconstruct the multi-step derivation that produced it. In contrast, for CQA, seeing that "(b) grocery cart" is the correct answer immediately narrows the rationale to arguing why grocery carts are used for checkout, which is a more constrained reasoning task.

Analysis of solution steps (Figure 6): The paper compares the number of calculation steps (occurrences of <<...>>) in the model-generated solutions versus the ground-truth solutions. They find that "most often, the number of calculation steps generated by the model matches the number of steps taken by humans (generally between 53% and 57% agreement across all iterations)." When the model uses fewer steps, it is "typically because the model skips steps, but occasionally it finds different solutions." An example in Appendix J (Figure 8) shows a case where the ground truth uses 6 calculation steps to solve a problem about distributing bottles, while the model produces a single calculation step (180/2=90) and gets the correct answer—it recognized that the breakdown by drink type was irrelevant and the problem simplified to dividing by 2.

---

Key Hyperparameter Summary

Consolidating the hyperparameters mentioned throughout the method:

---

Design Choices and Their Justifications

Why filter by answer correctness rather than evaluating rationale quality directly? Because evaluating whether a reasoning chain is logically sound is, in the general case, as hard as solving the problem—if you had an automated way to verify reasoning quality, you could use it to solve the original task directly. Filtering by answer correctness uses the ground-truth answer as a weak proxy for rationale quality, which is available as part of the original dataset without additional annotation. This is the same insight that makes reinforcement learning from outcome rewards possible: you don't need to evaluate the quality of each action, only the final outcome.

Why use few-shot prompts during fine-tuning? The paper reports that including few-shot prompts during fine-tuning dramatically reduces "drift"—the phenomenon where later rationales become increasingly dissimilar from the initial few-shot set. Without few-shot prompts in the training data, the model's rationale style can diverge from the intended format over successive iterations, producing rationales that may be less structured or less useful. Including the few-shot prompts in the training data anchors the model to the original rationale format. The downside is longer sequence lengths (the prompt set adds tokens to every training example) and potentially constraining the model to the quality ceiling of the initial few-shot examples.

Why start with fewer training steps and increase gradually? The paper found that "training more slowly at the beginning ultimately benefits model performance." Early iterations have rationales generated by the weakest version of the model (the original pretrained GPT-J with only few-shot prompting), so the training data is of the lowest quality. Overtraining on this weak data would cause the model to overfit to suboptimal reasoning patterns. By gradually increasing training steps, later iterations—which have higher-quality rationales from the improved model—receive more training time, putting more weight on better data.

Why not use higher temperatures to increase training data diversity? The paper explicitly tested this and found it harmful (Section 5). At higher temperatures (0.5-0.7), the model more frequently produces correct answers despite incorrect reasoning, because on multiple-choice tasks like CQA (with 20% random baseline), a wrong reasoning chain can accidentally end with the right answer. Training on these false-positive rationales teaches the model to produce reasoning that is fluent but logically unsound, which degrades generalization. Low-temperature (near-greedy) generation produces rationales the model is confident about, which are more likely to be genuinely correct when the answer is right.

Why train from the original pretrained model each iteration rather than continuing to fine-tune? The paper explicitly states this is "to avoid overfitting." In a continual fine-tuning setup, the model's weights drift further from the pretrained distribution with each iteration. The rationales generated by the drifted model may still be correct, but they are generated from a different distribution, and continuing to fine-tune on them can cause further drift—a compounding effect. Restarting from the original pretrained checkpoint provides a stable foundation: the model learns from improved data each iteration, but the learning process starts from the same initialization, making each iteration's training trajectory independent of previous iterations' fine-tuning artifacts. This also makes it possible to compare iterations directly, since the only variable changing is the training data quality, not the cumulative training duration.

Why does rationalization use the same prompt style as rationale generation? The paper strips the hint from rationalized examples before adding them to the training set. This means the training data for rationalized examples looks identical to the training data for naturally generated rationales: question → rationale → answer, with no hint. The model therefore learns to produce these rationales in the forward direction, as if it had generated them without assistance. If the hint were included in the training data, the model might learn to expect hints at test time, which would defeat the purpose of training on them.

Why does rationalization work but high-temperature sampling doesn't? This is a subtle point that rewards careful attention. Both techniques attempt to expand the training set beyond what the model naturally generates correctly at low temperature. The difference is that rationalization uses the ground-truth answer to constrain the rationale generation, making the rationales more likely to represent genuine reasoning (even if constructed backwards), while high-temperature sampling introduces randomness without constraint, making correct answers more likely to arise from coincidental rather than causal relationships between the reasoning and the answer. The paper's evidence for this is that high-temperature training causes the model to "diverge into meaninglessness" on arithmetic, while rationalization consistently improves performance.

---

4. Key Insights and Innovations

Innovation 1: Bootstrapping Rationale Generation from Outcome-Only Feedback

The defining intellectual move in STaR is the recognition that a language model can generate its own rationale training data using nothing more than answer correctness as a learning signal. Prior to this work, the dominant assumption was that obtaining rationales required either human annotation (Rajani et al., 2019; Cobbe et al., 2021) or hand-crafted templates that encode task-specific solution strategies (Nye et al., 2021; Shwartz et al., 2020). Both approaches embed external knowledge about how to reason into the training pipeline. STaR inverts this: the model already possesses latent reasoning capability from pretraining, and the challenge is eliciting and amplifying it, not injecting it from outside.

This reframes rationale acquisition from a data annotation problem to a self-supervised extraction problem. The core mechanism—generate rationales, filter by answer correctness, fine-tune, repeat—is simple enough to describe in a paragraph, but the conceptual leap is treating the ground-truth answer as a weak binary reward that labels the entire rationale as useful or not, without ever evaluating the intermediate steps. The paper formalizes this through the policy gradient lens (Equation 1), showing that filtering by outcome correctness approximates a valid gradient for improving rationale quality. But the practical significance is that this works at all: the model's own correct reasoning, even when generated by simple few-shot prompting, contains enough signal to bootstrap substantial improvement.

The evidence is most striking on CommonsenseQA, where STaR without rationalization trains on only 69.7% of the dataset (the fraction the model initially answers correctly) yet achieves 68.8% accuracy—already surpassing GPT-J fine-tuned to directly predict answers on 100% of the data (60.0%). This means the model extracts more value from partial, self-generated reasoning data than from complete, answer-only data. The rationales are doing real computational work beyond what surface-level answer patterns provide.

This is a fundamental shift in how to think about rationale acquisition, but it is not a complete solution: the bootstrapping loop inherently plateaus when the model stops solving new problems, a limitation the paper diagnoses and partially addresses with rationalization.

Innovation 2: Rationalization as Backward Reasoning to Break the Bootstrapping Deadlock

The paper's second conceptual contribution is the diagnosis of a specific failure mode in self-training loops—stagnation from training only on problems the model already solves—and the introduction of rationalization as a targeted remedy. This is not merely an algorithmic trick; it identifies a structural limitation of any bootstrapping method that filters by outcome: the training distribution is always a subset of the model's current capability frontier, so the frontier can never expand.

Rationalization solves this by providing the correct answer as a hint and asking the model to generate a rationale backwards. The paper frames this as sampling from $p(r \mid x, y)$ rather than $p(r \mid x)$—conditioning on the answer to access rationales that would be unlikely under forward generation. This is conceptually distinct from prior hint-based approaches (e.g., Rajani et al.'s human-written explanations) because the hint is used only during generation, not during training: the rationalized examples are stripped of hints before fine-tuning, so the model learns to produce the reasoning as if it had generated it forward without assistance.

The empirical impact on CommonsenseQA is clear but moderate: rationalization adds 3.7 percentage points (68.8% → 72.5%) and expands training data coverage from 69.7% to 86.7%. On GSM8K, the gain is negligible (10.1% → 10.7%). This inconsistency is itself informative: rationalization helps most on tasks where conditioning on the answer substantially constrains the space of valid rationales. On multiple-choice QA, knowing "(b) grocery cart" is correct immediately narrows the justification to arguing about grocery carts. On multi-step math word problems, knowing the final number doesn't obviously help reconstruct the chain of operations that produced it—the backward distribution $p(r \mid x, y)$ may not be much easier to sample from than the forward distribution.

This innovation is incremental in mechanism but diagnostically important: it establishes that bootstrapping methods need a mechanism to push beyond the capability frontier, and it provides a concrete (if partial) solution whose effectiveness varies by task structure.

Innovation 3: The Policy Gradient Connection as a Unifying Framework for Self-Training

The paper's formal framing of STaR as an approximation to a policy gradient objective (Equation 1 and surrounding discussion in Section 3.1) provides an intellectual bridge between self-supervised fine-tuning and reinforcement learning that was not obvious in prior rationale-generation work. Before STaR, rationale-based training was largely viewed through the lens of supervised learning—you need target rationales to compute a loss, and those targets must come from somewhere (humans, templates, or a stronger model). The policy gradient framing shows that outcome-level feedback (answer correct/incorrect) is sufficient to define a valid gradient for improving rationale quality, and that STaR's filtering step is equivalent to discarding zero-reward samples in REINFORCE.

This connection is not merely theoretical decoration. It explains why the method works despite never evaluating rationale quality directly: the indicator reward $\mathbf{1}(\hat{y} = y)$ provides an unbiased (if high-variance) signal for which rationales are worth imitating. It also suggests natural extensions that the paper flags but doesn't explore—importance weighting for rationalized samples, using the model's answer confidence as a continuous reward, or proper off-policy corrections for multi-iteration training.

The significance of this framing is that it positions self-training for reasoning within the mature optimization framework of RL, opening the door to techniques from that literature (variance reduction, exploration bonuses, critic networks) being applied to rationale generation. Subsequent work on RLHF for reasoning (e.g., Qu et al., 2024, which uses outcome-level rewards to train revision models) is a direct intellectual descendant of this insight.

This is a theoretical contribution—a reframing rather than a new algorithm—but it is important because it converts STaR from an ad-hoc bootstrapping recipe into a principled approximation of a well-understood optimization objective.

Innovation 4: The Diagnostic Finding That Rationale Quality Improves Even When Answers Are Already Correct

A subtle but empirically grounded finding emerges from the human evaluation in Section 4.4 and the case study in Figure 7: STaR improves rationale quality on problems the model already answers correctly via few-shot prompting. This is not obvious from the method's design, which only adds new examples to the training set when answers are correct—existing correct examples are already in the training data, so why would their quality improve?

The human evaluation provides evidence: crowdworkers were 30% more likely to prefer STaR-generated rationales over few-shot CoT rationales on questions both methods answered correctly (p = 0.039). The case study in Figure 7(b) illustrates this concretely: for a question about attending a university, the few-shot rationale simply asserts "The answer is university (b)," while the STaR rationale explains "Universities are places where people go to learn about things." The model has learned to produce more informative, explanatory rationales through iterative self-training, even on questions it could already answer.

This finding matters because it demonstrates that STaR is not merely improving accuracy (which could come from better answer selection without better reasoning) but genuinely improving the coherence and explanatory quality of the reasoning process. The paper is careful not to overclaim about faithfulness—the rationales may still be post-hoc justifications rather than faithful traces of internal computation—but the improvement in perceived quality suggests the model is learning something about what constitutes good reasoning beyond surface-level answer patterns.

This is an empirical diagnostic finding rather than a methodological contribution, but it addresses a key concern about self-training: that models trained on their own outputs might simply reinforce existing patterns without genuine improvement. The evidence suggests otherwise, at least for reasoning tasks where the quality floor is low enough that iterative refinement can produce measurably better outputs.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three distinct benchmarks. Arithmetic: a synthetic dataset of 50,000 randomly generated n-digit addition problems (n ∈ {1,…,5}, later extended to 9–10), sampled 10,000 per outer loop iteration, formatted with the scratchpad convention from Nye et al. (2021). CommonsenseQA (CQA) (Talmor et al., 2019): 12,247 five-way multiple-choice commonsense reasoning questions (9,741 train, 1,221 dev, 1,285 test), constructed from ConceptNet, with human performance at 89%. The paper evaluates on the dev set since test labels are withheld. GSM8K (Cobbe et al., 2021): 7,473 train and 1,319 test examples of grade-school math word problems requiring 2–8 calculation steps. All datasets provide ground-truth answers but (initially) no rationales.

• Base model(s). All experiments use GPT-J (6B parameters), a 28-layer decoder-only transformer with embedding dimension 4096, 16 attention heads of dimension 256, FFN hidden layer of size 16384, and vocabulary size 50.4K, pretrained on The Pile (Gao et al., 2020). The paper chose GPT-J because its checkpoint and fine-tuning code are publicly available, and because it is "large enough to generate rationales of non-trivial quality to be bootstrapped from" (Section 4.1). For the CQA comparison, the paper also reports the fine-tuned GPT-3 result (175B, ~30× larger) from Xu et al. (2021) and the few-shot LaMDA 137B result from Wei et al. (2022).

• Metrics. The primary metric is accuracy: for arithmetic, whether all digits of the sum are correct (per-digit-length); for CQA, exact match of the predicted answer choice to the ground truth (dev set); for GSM8K, exact match of the final numeric answer (test set). Rationale quality is assessed indirectly via a human preference study (Section 4.4, Appendix C): 20 crowdworkers each ranked 10 randomly selected question-rationale triples (few-shot CoT, STaR, human-written) on "how well they justify the correct answer."

• Baselines. Several baselines are compared in Table 1 and Table 2: (1) Few-shot Direct — standard few-shot prompting without rationales, asking the model to directly output the answer. (2) Few-shot CoT (Chain-of-Thought) — few-shot prompting with 10 rationale examples (same prompts used to seed STaR), following Wei et al. (2022). (3) GPT-J Direct Finetuned — GPT-J fine-tuned on 100% of the training data to directly predict the final answer without generating rationales. (4) GPT-3 Direct Finetuned (Xu et al., 2021) — a 175B GPT-3 model fine-tuned on CQA, reported at 73.0%. (5) Few-shot CoT LaMDA 137B (Wei et al., 2022) — a 137B LaMDA model few-shot prompted with chain-of-thought rationales, reported at 55.6% on CQA. For arithmetic, the paper also reports a baseline trained on 10,000 examples without scratchpads for 5,000 steps (76.3% accuracy). The paper also ablates STaR with vs. without rationalization, which acts as an internal comparison.

• Generation budget / compute accounting. There is no standardized compute budget metric across methods in this paper, because the primary comparison is between training paradigms (bootstrapped rationale fine-tuning vs. direct answer fine-tuning vs. few-shot prompting), not between inference-time strategies. The relevant "budget" is the number of training examples used, reported as a percentage of the dataset for CQA and GSM8K. For arithmetic, the budget is the number of samples per outer loop iteration (10,000). For inference, all methods generate one rationale per question at low (near-greedy) temperature — there is no Best-of-N or search component in STaR. The paper notes that higher-temperature sampling was tested as an alternative to rationalization (Section 5) but found counterproductive, and that "generating 10 sample outputs is approximately 10 times slower than generating one sample output," but this is not used as a formal budget axis.

• Cross-validation / statistical protocol. The paper does not use cross-validation for model selection. The outer loop runs until performance plateaus (16–36 iterations depending on task), and the paper reports results at the best-performing iteration. For CQA, evaluation is on the standard dev set. For GSM8K, evaluation is on the standard test set. For arithmetic, evaluation is on held-out examples from the 50,000-question pool. No statistical significance testing is reported for the main accuracy comparisons. The human evaluation reports p-values from a binomial test (p = 0.039 for STaR vs. few-shot, p < 0.001 for STaR vs. human-written rationales), based on 20 participants each providing 10 rankings.

---

Main Quantitative Results

Arithmetic: STaR Learns Multi-Digit Addition From Near-Zero Few-Shot Performance

The headline result for arithmetic (Figure 4) is that STaR with rationalization achieves 89.5% overall accuracy after 16 iterations, compared to a direct-prediction baseline of 76.3% trained on the same number of examples (10,000). This is striking because few-shot accuracy on multi-digit addition is near zero: "accuracy on 2-digit addition is less than 1%, and accuracy on more digits close to zero" (Section 4.3).

Figure 4a (STaR without rationalization) and Figure 4b (with rationalization) trace accuracy per-digit-length across iterations. Without rationalization, performance improvement is stage-wise: "the model generally has poor performance on the n-digit sum until it has good performance on the (n − 1)-digit sum." The model must master shorter-digit addition before longer-digit addition becomes learnable. With rationalization, "the model can learn many lengths at once, though not with equal accuracy." Rationalization allows the model to train on harder problems earlier, compressing the learning curve.

The specific trajectory with rationalization (Figure 4b): after one fine-tuning iteration, 2-digit addition improves from <1% to 32%. After 16 iterations, per-digit accuracies (read approximately from Figure 4b) reach roughly: 1-digit near 100%, 2-digit around 98%, 3-digit around 95%, 4-digit around 90%, 5-digit around 80%. The gap between digit lengths persists, suggesting the model learns digit-specific patterns rather than a fully general addition algorithm.

Out-of-distribution generalization: When the paper introduces additional digits (6–10 digits) at the 20th iteration while keeping the training set size fixed, the model successfully solves many of these unseen-length problems (Figure 5). Exact numbers are not given in the text, but the figure shows non-zero accuracy on 9- and 10-digit addition, which the model was never trained on. This suggests some degree of algorithmic generalization. However, the introduction of new digits "appears to make the training less stable" (Figure 5 shows oscillating accuracy), and the paper states "the exact cause is unclear."

CommonsenseQA: STaR Approaches GPT-3 Performance While Training on Less Data

The headline result for CQA (Table 1) is that STaR with rationalization achieves 72.5% accuracy on the dev set, compared to 73.0% for a 30× larger fine-tuned GPT-3 model (Xu et al., 2021). The key comparison points in Table 1:

Several aspects of this result deserve attention:

STaR without rationalization already strongly outperforms direct fine-tuning (68.8% vs. 60.0%) despite using only 69.7% of the training data. This means that training on model-generated rationales for the subset of questions the model can already solve is more effective than training on all questions to predict answers directly. The rationales provide a richer learning signal than the answers alone, even when the rationales are imperfect.

Rationalization adds 3.7 percentage points (68.8% → 72.5%) and expands data coverage from 69.7% to 86.7%. The data coverage expansion (an additional 17.0% of the dataset) is the secondary benefit the paper claims for rationalization. Note that rationalization adds 8.5% of the training data despite being applied to 30.3% of the questions — many rationalization attempts fail (the model doesn't produce the correct answer even with the hint), so only a subset of rationalized examples pass the filtering step.

The gap to the 30× larger GPT-3 model is 0.5 percentage points (72.5% vs. 73.0%). This is the paper's most dramatic comparison, and it suggests that self-generated reasoning can compensate for massive differences in model scale. However, this comparison carries important caveats: (1) GPT-3 was fine-tuned to directly predict answers, not to generate rationales — a fairer comparison would fine-tune GPT-3 on STaR-generated rationales or human rationales. (2) GPT-3 is a different model family with different pretraining data, so scale is confounded with other factors.

Few-shot CoT with GPT-J (36.6%) versus with LaMDA 137B (55.6%): The 19-point gap between GPT-J and a ~23× larger model on the same prompting task illustrates how much few-shot reasoning capability varies with model scale. STaR recovers most of this gap (72.5% vs. 55.6%) without increasing model size.

Human Evaluation: STaR Rationales Preferred Over Few-Shot and Human-Written Rationales

The human evaluation (Section 4.4, Appendix C) provides evidence about rationale quality independent of answer accuracy:

• STaR vs. few-shot CoT on questions both answer correctly: Crowdworkers were 30% more likely to rank STaR-generated rationales higher than few-shot rationales (p = 0.039).
• STaR vs. human-written rationales from Rajani et al. (2019): Crowdworkers were 74% more likely to prefer STaR-generated rationales (p < 0.001).

The paper explicitly tempers the human-written comparison: the Rajani et al. dataset contains many uninformative rationales (e.g., "The only answer that makes sense," "This word was most relevant," restatements of the answer, or even nonsensical text), making it a weak baseline for human reasoning quality. The paper states: "We do not believe that this indicates human-level rationale-generation performance. Instead, we feel that it speaks to the difficulty of eliciting high-quality rationales" (Section 4.4). The crowdworker instructions and the full set of evaluated examples are reproduced in Appendix C.

GSM8K: Modest Gains, Rationalization Provides Minimal Benefit

Table 2 reports GSM8K test accuracy:

The key observations:

STaR improves over both few-shot and direct fine-tuning baselines, but absolute accuracy remains low (10.7%). This is substantially below the CQA results and reflects the greater difficulty of multi-step mathematical reasoning for a 6B-parameter model. The few-shot CoT baseline is essentially at chance level (3.1% vs. 3.0% for direct), consistent with GPT-J's limited pretraining on math.

Rationalization adds only 0.6 percentage points (10.1% → 10.7%), compared to 3.7 points on CQA. The paper suggests this is because "providing the answer as a hint on GSM8K is less helpful: the reasoning chain is the main challenge, and seeing the final number doesn't necessarily make it easier to reconstruct the multi-step derivation that produced it." The backward distribution p(r | x, y) is not substantially easier to sample from than p(r | x) for these problems.

Data coverage is low: only 25.0% of training examples (without rationalization) and 28.7% (with rationalization) are solved. This means the model only trains on the easiest quarter of the dataset. However, even this subset provides better training signal than the full answer-only dataset (10.1% vs. 5.8%).

Analysis of solution steps (Figure 6): The model-generated solutions match the number of human calculation steps 53–57% of the time across iterations. When they disagree, the model typically uses fewer steps — sometimes because it skips steps, sometimes because it finds simpler solutions. The paper provides an example (Appendix J, Figure 8) where the model solves a 7-step ground-truth problem in a single step by recognizing that the breakdown by drink type is irrelevant and the problem reduces to dividing by 2.

---

Ablation Studies and Robustness Checks

Rationalization on vs. off (Table 1, Table 2, Figure 4): Adding rationalization consistently improves final accuracy and accelerates learning. On CQA: 68.8% → 72.5% (+3.7 points). On GSM8K: 10.1% → 10.7% (+0.6 points, minimal). On arithmetic: without rationalization, learning is stage-wise and plateaus earlier; with rationalization, multiple digit lengths are learned concurrently and convergence is faster (Figure 4a vs. 4b). The paper explicitly notes that rationalization provides two distinct benefits: exposure to harder problems (allowing the model to expand its capability frontier) and increased dataset size.

Few-shot prompts included vs. excluded during fine-tuning (Section 5): Including few-shot prompts in the training data has "a meaningful performance benefit." On CQA without rationalization, including prompts raises accuracy from 60.9% to 68.8% (+7.9 points). With rationalization, from 69.9% to 72.5% (+2.6 points). The paper attributes this to reduced "drift" — the model's generated rationales remain stylistically consistent with the intended format. A tradeoff is noted: including prompts increases sequence length (the few-shot examples add tokens) and may constrain the model to the quality ceiling of the initial prompts. The paper also observes that without few-shot prompts during training, the model performs "gradually worse at rationalization as it trains for longer periods of time."

Higher-temperature sampling as an alternative to rationalization (Section 5): Tested as a way to expand the training set by generating multiple diverse samples and keeping correct ones. The paper reports this is consistently counterproductive: "higher temperatures (e.g., 0.5 or 0.7) consistently led to models worse than models with reasoning alone." On arithmetic, "scratchpads that the model learns to produce with a higher-temperature sampling approach diverge into meaninglessness and cause the model to stagnate." The mechanism is that higher temperature increases the rate of correct answers that occur despite incorrect reasoning, and "training on bad or irrelevant reasoning prevents generalization." This is a negative result that justifies the low-temperature, single-sample-per-question design of STaR and motivates rationalization as a targeted alternative for expanding training data.

Iteration count and training step schedule (Section 4.1, Figure 4): The paper uses progressive increases in training steps (starting at 40 steps, increasing 20% per iteration) because "training more slowly at the beginning ultimately benefits model performance." The specific per-iteration step counts are not ablated systematically, but the paper notes this was chosen based on the intuition that early iterations have lower-quality rationales and should not be over-trained on. The outer loop runs until performance plateaus: 16 iterations for arithmetic without rationalization, ~36 with rationalization; 36 iterations for CQA without rationalization plus 10 more with rationalization; 36 iterations for GSM8K without rationalization plus 10 with rationalization. The paper does not report the sensitivity of final accuracy to the stopping point.

Training from scratch vs. continual fine-tuning (Section 3.1): This is a design choice, not an ablation — the paper always retrains from the original pretrained GPT-J at each iteration. The stated motivation is to "avoid overfitting." No direct comparison to continual fine-tuning is reported, so the magnitude of this benefit is unknown. However, the paper's framing suggests that continual fine-tuning was attempted and found to cause degradation (the overfitting concern is stated as a justification for the design, not as a hypothesis to test).

Out-of-distribution digit generalization on arithmetic (Figure 5): When additional digits (6–10) are introduced at iteration 20, the model solves many unseen-length problems, demonstrating some generalization. However, training stability decreases, with oscillating accuracy for the new digits. The paper does not quantify the OOD accuracy or compare it to a model trained from scratch on the extended digit range.

Model scale dependence (implicit, Section 6): The paper states that GPT-2 was not able to bootstrap from few-shot reasoning in arithmetic, establishing a minimum capability threshold: "the initial model must be big enough to have some reasoning capabilities." The exact scale at which STaR becomes viable between GPT-2 and GPT-J (6B) is not explored. This is a limitation of the study's scope rather than a formal ablation.

CQA few-shot prompt quality (Section 4.1, Appendix B): The paper modified the Wei et al. (2022) chain-of-thought prompts "slightly to fix an incorrect answer and to more explicitly reference relevant knowledge." The impact of these modifications is not ablated against the original prompts. The paper cites Min et al. (2022) to argue this is "unlikely to meaningfully affect few-shot performance," but provides no direct evidence.

---

Critical Assessment

Claim: STaR achieves accuracy comparable to a 30× larger fine-tuned model on CommonsenseQA.

Assessment: This claim is factually correct (72.5% STaR vs. 73.0% GPT-3) but overstates what was demonstrated. The comparison is not between equivalent training paradigms: STaR uses rationale-based fine-tuning, while GPT-3 was fine-tuned to directly predict answers (Xu et al., 2021). A properly matched comparison would require fine-tuning GPT-3 on the same STaR-generated rationale dataset, or comparing both models under direct answer prediction. The paper implicitly acknowledges this asymmetry by including GPT-J Direct Finetuned (60.0%) as a same-paradigm baseline. The 12.5-point gap between STaR and GPT-J Direct Finetuned is the cleaner demonstration of STaR's benefit within a fixed model scale. The 30× comparison is a headline result that combines the effect of rationale training with the effect of model scale differences. A more precise version of the claim would be: "STaR on GPT-J (6B) approaches the accuracy of a much larger model (175B GPT-3) that was fine-tuned without rationales, primarily by leveraging self-generated reasoning."

The paper also does not report confidence intervals for CQA dev set accuracy. The dev set has 1,221 questions, so a 0.5 percentage point difference corresponds to approximately 6 questions — well within the plausible range of sampling variance.

Claim: STaR significantly improves performance over few-shot baselines.

Assessment: Strongly supported. On CQA: 36.6% (few-shot CoT) → 72.5% (STaR with rationalization), a 35.9 percentage point gain. On GSM8K: 3.1% → 10.7%, a 7.6 point gain. On arithmetic: from near-zero on multi-digit addition to 89.5% overall. The improvement is large and consistent across tasks. The few-shot baselines use the same prompts that seed STaR, so the comparison isolates the effect of the bootstrapping process.

However, the few-shot CoT baselines for GPT-J are notably low (36.6% on CQA, 3.1% on GSM8K), which creates a low performance floor. The 36.6% on CQA is above random (20%) but far below the 55.6% that Wei et al. (2022) achieved with a 137B LaMDA model. STaR's improvements might be partially attributable to the large headroom available for improvement — it's unclear whether the method would produce similar gains starting from a model with higher few-shot performance, because the training data would contain fewer errors to correct and the gain from rationale refinement might be smaller.

Claim: Rationalization accelerates and improves the bootstrapping process.

Assessment: Partially supported, with important task-dependent caveats. On CQA, rationalization adds 3.7 points (68.8% → 72.5%) and expands data coverage from 69.7% to 86.7%. On GSM8K, the gain is negligible (10.1% → 10.7%). On arithmetic, rationalization qualitatively changes the learning dynamics (stage-wise → concurrent across digit lengths) and accelerates convergence, but the paper does not report a final accuracy comparison at matched iterations with vs. without rationalization — Figure 4 shows different iteration counts, making direct comparison difficult.

The paper's explanation for the GSM8K failure — that conditioning on the answer doesn't substantially simplify generating the rationale for multi-step word problems — is plausible but not empirically demonstrated. A diagnostic experiment measuring rationalization success rate (fraction of failed questions where rationalization produces a correct answer) across tasks would have clarified this. The paper only reports the final accuracy impact, not the intermediate rationalization yield.

The claim that rationalization "exposes the model to difficult problems which otherwise would not have appeared in its fine-tuning dataset" is mechanistically true (the rationalized examples come from previously unsolved questions), but the extent to which this exposure translates to improved capability on unseen test problems of similar difficulty is not directly quantified beyond the aggregate accuracy numbers.

Claim: STaR allows a pretrained language model to iteratively improve itself.

Assessment: Supported with a critical scope limitation noted by the paper itself: the initial model must have above-chance few-shot performance. GPT-2 could not bootstrap on arithmetic (Section 6). This means STaR is an amplification method, not a capability creation method. It improves reasoning that already exists in some nascent form, but cannot create reasoning ability where none exists. This limitation is not a weakness of the paper (it is honestly acknowledged) but it bounds the generality of the claim.

The "iteratively improve itself" framing also deserves nuance: STaR does not produce a single model that continuously improves. Instead, it produces a sequence of models, each trained from scratch on progressively better data. The improvement is in the data, not in the model's intrinsic self-modification capacity. This is a form of iterative dataset refinement, not lifelong learning.

Missing experiments that would strengthen the paper:

1. Ablation on dataset size: How does STaR accuracy scale with the number of training questions? The paper uses the full CQA and GSM8K training sets. An experiment reducing the dataset size by 2×, 4×, etc. would characterize STaR's data efficiency relative to direct fine-tuning.

2. Ablation on few-shot example count: The paper uses 10 few-shot examples for all tasks. How sensitive is STaR to this number? Can it bootstrap from 3 examples? From 1? This matters for domains where high-quality exemplar rationales are scarce.

3. Applying STaR to a larger base model: The paper's most dramatic result (matching GPT-3 at 30× scale) compares GPT-J + STaR to GPT-3 without STaR. What would STaR + GPT-3 achieve? This experiment would determine whether STaR's benefits are complementary to scale or whether they diminish as few-shot performance increases.

4. Stricter baselines for CQA: The paper compares to GPT-3 fine-tuned on answers (Xu et al., 2021) but not to GPT-3 with chain-of-thought prompting or GPT-3 fine-tuned on human-provided rationales. The 73.0% baseline is a moving target — subsequent work might achieve higher accuracy through better fine-tuning recipes — so the absolute gap is less informative than a matched-paradigm comparison at scale.

5. Quantifying the computational cost of STaR: The paper does not report the total FLOPs or GPU-hours consumed by the full bootstrapping loop (multiple outer loop iterations, each requiring inference on the full dataset plus fine-tuning). This makes it difficult to assess whether STaR's accuracy gains are compute-efficient compared to simply fine-tuning a larger model. The paper's framing emphasizes data annotation cost, but the computational cost is also relevant for practitioners.

6. Statistical significance for main results: The paper reports p-values only for the human evaluation. No confidence intervals or significance tests are reported for the accuracy numbers in Tables 1 and 2. Given the 1,221-question CQA dev set, the standard error for 72.5% accuracy is approximately 1.3 percentage points — the 0.5 point gap to GPT-3 is within one standard error.

Conditional scope of claims:

• STaR works when the base model has non-trivial few-shot reasoning ability (above-chance performance). This imposes a minimum model scale.
• STaR works best when the task involves reasoning from world knowledge (CQA) or systematic computation (arithmetic), and when rationales strongly constrain answers.
• Rationalization helps when conditioning on the answer meaningfully constrains the rationale search space — true for multiple-choice QA (where the answer narrows the justification), less true for open-ended math word problems (where the final number doesn't obviously simplify reconstructing the derivation).
• STaR does not eliminate the need for ground-truth answers — it requires a dataset of question-answer pairs, just not question-rationale pairs.
• STaR does not guarantee faithful rationales — the paper explicitly acknowledges that generated rationales may not reflect the model's internal processing, and the method provides no mechanism to verify faithfulness beyond answer correctness.

6. Limitations and Trade-offs

The Base Model Must Already Possess Non-Trivial Reasoning Capability

STaR is fundamentally an amplification method, not a capability creation method. The entire bootstrapping loop depends on the first iteration generating enough correct rationales to serve as training data, which requires that the base model's few-shot chain-of-thought performance be above chance. The paper acknowledges this explicitly in Section 6:

"In order for the first iteration of STaR to succeed, few-shot performance must be above chance, implying that the initial model must be big enough to have some reasoning capabilities."

The paper reports that GPT-2 "was not able to bootstrap from few-shot reasoning in even the arithmetic domain" (Section 6), establishing a hard failure case. On arithmetic, few-shot accuracy for multi-digit addition with GPT-J is near zero (less than 1% for 2-digit addition, Section 4.3), yet STaR still succeeds because the model can solve 1-digit addition and gradually generalizes to longer sequences. This suggests the success threshold is not merely "above random" but depends on the existence of a subset of the problem space where few-shot reasoning is reliable enough to seed the first iteration. For tasks where no such subset exists—where the model's few-shot reasoning is uniformly poor across all problem difficulties—STaR would fail completely.

The consequence is that STaR is not a general solution for teaching reasoning to arbitrary language models. It is restricted to models that have already acquired some reasoning ability through pretraining, and the quality ceiling is constrained by what the base model can eventually learn from its own outputs. The policy gradient framing (Equation 1 in Section 3.1) makes this clear: the gradient is only non-zero for sampled rationales that yield correct answers. If the model never samples a correct answer for a class of problems, it receives zero training signal for those problems, and they remain unsolvable forever. This is exactly what happens on the hardest GSM8K problems, where the model trains on only 25–28.7% of the dataset and achieves only 10.7% test accuracy (Table 2)—the other ~75% of training problems are simply never solved, so the model never learns from them.

The paper does not systematically characterize the minimum capability threshold. It reports the GPT-2 failure and the GPT-J success but does not test intermediate model sizes or measure how few-shot performance correlates with STaR's final accuracy. A practitioner with a model of unknown reasoning capability has no diagnostic for whether STaR will work beyond trial and error. This limitation is partially mitigated by the paper's transparency—the requirement is stated clearly—but it narrows the method's applicability to models that are already "large enough," a threshold the paper does not quantify.

---

The Ground-Truth Answer Requirement Makes STaR Inapplicable to Datasets Without Labeled Answers

STaR's filtering mechanism depends entirely on comparing the model's predicted answer $\hat{y}_i$ to the ground-truth answer $y_i$ (Algorithm 1, Line 5). This means the method requires a fully labeled dataset of question-answer pairs for training. The paper is explicit about this assumption—the dataset is defined as $D = \{(x_i, y_i)\}_{i=1}^D$ (Section 3.1)—but does not treat it as a limitation in the discussion. Yet it fundamentally restricts STaR's scope compared to few-shot prompting, which can be applied to any new task with just a handful of examples and no labeled training set.

The consequence is that STaR does not eliminate the data annotation bottleneck—it shifts it from rationale annotation (expensive, requires human reasoning) to answer annotation (cheaper, but still requires labeling). For tasks where answers are naturally available (math problems with numeric solutions, multiple-choice QA with answer keys), this shift is practical. For open-ended generation tasks, subjective reasoning, or domains where correctness is ambiguous or multi-dimensional, STaR cannot be applied in its current form. The paper acknowledges this implicitly by restricting experiments to tasks with unambiguous ground-truth answers (arithmetic, multiple-choice QA, math word problems with numeric answers). Section 5 mentions a potential extension using majority voting from Wang et al. (2022) to handle datasets "of only questions, without answers," by treating the majority-vote result as pseudo-ground-truth, but this is suggested as future work and not evaluated.

The mitigation status is that no solution is provided within the paper. The majority-voting extension is sketched but untested, and it would introduce its own limitations: majority voting requires generating multiple samples per question, increasing inference cost, and the pseudo-ground-truth would be noisy (majority vote is not always correct), potentially corrupting the training signal in ways the paper does not analyze.

---

Rationalization Is Task-Dependent and Fails to Help on Multi-Step Mathematical Reasoning

Rationalization is presented as a key innovation that "accelerates and improves the bootstrapping process" (Section 1, contribution 2). However, the empirical results reveal a sharp task dependence that the paper diagnoses but does not resolve. On CommonsenseQA, rationalization adds 3.7 percentage points (68.8% → 72.5%) and expands data coverage from 69.7% to 86.7% (Table 1). On GSM8K, the gain is negligible: 10.1% → 10.7%, a difference of 0.6 percentage points on a 1,319-question test set (Table 2). On arithmetic, rationalization qualitatively changes the learning dynamics—enabling concurrent learning across digit lengths rather than stage-wise progression (Figure 4)—but a direct final-accuracy comparison at matched iterations is not reported.

The paper's explanation for the GSM8K failure is that "providing the answer as a hint on GSM8K is less helpful: the reasoning chain is the main challenge, and seeing the final number doesn't necessarily make it easier to reconstruct the multi-step derivation that produced it" (interpreted from Section 5 discussion). This is plausible but diagnostically incomplete—it identifies why rationalization might fail without specifying when it will fail. A practitioner considering STaR for a new task has no predictive criterion for whether rationalization will help. The paper frames rationalization in terms of conditional distributions: rationale generation samples from $p(r \mid x)$, while rationalization samples from $p(r \mid x, y)$ (Section 5). The effectiveness of rationalization depends on whether $p(r \mid x, y)$ is easier to sample high-quality rationales from than $p(r \mid x)$. The paper does not provide a way to estimate this without running the full STaR loop.

The consequence is that rationalization cannot be relied upon as a general-purpose mechanism for breaking the bootstrapping deadlock. On tasks where the reasoning chain is the primary challenge and the final answer provides little constraint on the intermediate steps—multi-step math, complex logical deduction, multi-hop reasoning—rationalization may provide minimal benefit. The paper's framing of rationalization as a core contribution (title, abstract, Section 3.2) is therefore stronger than the empirical evidence supports across tasks.

Mitigation status: The paper acknowledges the task dependence implicitly through the contrasting GSM8K and CQA results but does not propose alternative mechanisms for the case where rationalization fails. The deadlock problem—"improvement ends when the model fails to solve new problems in the training set" (Section 3.2)—remains fundamentally unsolved for tasks where backward reasoning is as difficult as forward reasoning.

---

Computational Cost Is Not Characterized, Making Efficiency Comparisons Impossible

STaR is an iterative method that requires, at each outer loop iteration: (1) running full-dataset inference with the current model to generate rationales, (2) running a second inference pass for rationalization on failed examples, (3) fine-tuning from scratch on the collected dataset, and (4) repeating this process 16–36+ times until convergence. The paper reports no total FLOPs, GPU-hours, or wall-clock time for any experiment. The only computational detail provided is that experiments run on "a single TPU-v3 node" (Appendix H).

This omission has several consequences for interpreting the paper's claims:

The 30× model scale comparison on CQA is incomplete. The paper compares STaR on GPT-J (6B parameters) to a fine-tuned GPT-3 (175B parameters) and reports comparable accuracy (72.5% vs. 73.0%). This is framed as a favorable tradeoff: a much smaller model achieves similar accuracy. But the total computational cost of achieving that accuracy is unknown. If running 36 iterations of STaR on GPT-J consumes, say, the equivalent of training a 30B-parameter model once, then the efficiency claim is weaker than it appears. The paper cannot answer the question: for a fixed FLOPs budget, is it better to run STaR on a 6B model or to simply fine-tune a larger model on answer-only data?

The cost of rationalization is unaccounted for. Rationalization requires a second inference pass over all failed questions at each iteration. The paper reports that rationalization expands data coverage from 69.7% to 86.7% on CQA (Table 1)—an increase of 17 percentage points—but does not report the computational cost of achieving this expansion. On GSM8K, rationalization adds only 0.6 points for presumably similar overhead, making its cost-effectiveness questionable.

The progressive training step increase compounds cost. The paper starts with 40 training steps at iteration 1 and increases by 20% each iteration. By iteration 16 (arithmetic without rationalization), this is $40 \times 1.2^{15} \approx 690$ steps per iteration. By iteration 36 (CQA without rationalization), it is $40 \times 1.2^{35} \approx 11,700$ steps. The total training steps across all iterations is substantial, and the inference cost (generating rationales for the full dataset at each iteration) likely dominates. Without cost reporting, a practitioner cannot estimate whether STaR is practical for their compute budget.

Mitigation status: The paper does not acknowledge this as a limitation. The focus is entirely on annotation cost (human effort to write rationales) rather than computational cost. This is a framing choice—the paper positions STaR as solving a data bottleneck, not a compute bottleneck—but it leaves a critical practical question unanswered for anyone considering deploying the method.

---

The CommonsenseQA Results Overstate the Fairness of the 30× Scale Comparison

The paper's most dramatic result—STaR on GPT-J (72.5%) approaching GPT-3 fine-tuned (73.0%)—is used to argue that self-generated reasoning can compensate for a 30× difference in model scale. This comparison is factually accurate but methodologically asymmetric in ways that inflate its apparent significance.

The asymmetry has two dimensions:

Training paradigm mismatch. STaR trains GPT-J to generate rationales and then answer; the GPT-3 baseline (from Xu et al., 2021) was fine-tuned to directly predict answers without generating intermediate reasoning. The appropriate same-paradigm comparison is STaR vs. GPT-J Direct Finetuned, where the gap is 12.5 points (72.5% vs. 60.0%)—substantial, but entirely attributable to the value of rationale-based training at a fixed model scale. To claim that STaR compensates for model scale, one would need to compare STaR on GPT-J to the same training paradigm on GPT-3 (e.g., GPT-3 fine-tuned on the STaR-generated rationale dataset, or GPT-3 fine-tuned on human-provided rationales). The paper does not provide this comparison.

Model family and pretraining confounds. GPT-J and GPT-3 are different models with different architectures, pretraining data, and tokenizers. GPT-3 was trained primarily on CommonCrawl, WebText2, Books, and Wikipedia (Brown et al., 2020); GPT-J was trained on The Pile (Gao et al., 2020), which has a different composition. The 30× parameter ratio is confounded with these differences. A cleaner comparison would use models from the same family at different scales—for instance, running STaR on GPT-3 Small (125M) vs. GPT-3 Large (760M) vs. GPT-3 6.7B vs. GPT-3 175B—to isolate the effect of scale from pretraining and architecture.

The consequence is that the headline efficiency claim overstates what was demonstrated. The paper has shown that rationale-based training on a 6B model substantially outperforms answer-only training on a 175B model—an important result about the value of rationales, not about scale substitution. The actual scale-substitution effect (how much larger a model must be to match STaR's performance under equivalent training) is not measured.

The paper partially mitigates this by including the GPT-J Direct Finetuned baseline (60.0%), which provides the within-model-scale comparison. But the 30× framing in the abstract and introduction—"performs comparably to fine-tuning a 30× larger state-of-the-art language model"—does not carry the caveats needed for accurate interpretation. A precise claim would be: "STaR on a 6B model with rationale training achieves accuracy within 0.5 points of a 175B model with answer-only training, primarily due to the benefit of reasoning-based training rather than scale substitution per se."

---

Rationale Faithfulness Is Neither Guaranteed Nor Measurable Within the Framework

STaR trains models to generate rationales that lead to correct answers, but provides no mechanism to ensure that these rationales faithfully represent the model's internal reasoning process. The paper acknowledges this limitation explicitly in Appendix G:

"While STaR encourages the use of reasoning in rationales which leads the model to correct answers, it is difficult, if not impossible, to ensure that the rationales reflect the model's internal processing. For example, it is straightforward to imagine the model implicitly selecting a particular answer immediately and then generating a rationale to justify that selected answer."

This is not merely a philosophical concern—it has concrete practical implications:

Bias amplification. If the model selects answers based on shallow patterns or biases (e.g., gender stereotypes, spurious word associations) and then generates post-hoc rationales that appear reasonable but are disconnected from the actual decision process, the rationales provide a false sense of interpretability. The paper acknowledges this risk in Appendix G: "STaR is designed to amplify the reasoning that leads to correct solutions on a given dataset. The implication of this with a dataset like CommonsenseQA is that if biases are 'useful' in solving the dataset then they will be amplified." The paper provides a few encouraging examples where the model appears to disregard irrelevant gender information, but notes that "this question warrants a much more comprehensive study."

Error mode obfuscation. The failure cases documented in Appendix A include rationales that "beg the question" (the model's reasoning implicitly assumes the answer), provide "red herrings" (true but irrelevant statements), or assert that "the question implies the answer" without explanation. These rationales are fluent and superficially plausible, but they do not represent valid logical reasoning. Training on such rationales may improve answer accuracy (because the answer is correct) without improving genuine reasoning capability. The model learns to produce rationales that look like good reasoning to the outcome-based filter, not rationales that are good reasoning.

The human evaluation addresses quality, not faithfulness. The human evaluation (Section 4.4) shows that crowdworkers prefer STaR rationales over few-shot rationales on dimensions like "how well they justify the answer." This measures perceived quality, not faithfulness. A post-hoc justification can be highly persuasive without being causally connected to the model's decision process. The paper is appropriately cautious about this distinction: the human evaluation results are not claimed as evidence of faithfulness.

Mitigation status: The paper acknowledges the faithfulness problem honestly but provides no solution. It notes that "ablation studies from papers such as [Wei et al., 2022] make it clear that the generation of a rationale before producing an answer non-trivially improves the model's answer quality," which establishes that rationales have some causal role in answer selection (since generating them changes the answer distribution). But this is a weak guarantee—it shows rationales are not entirely epiphenomenal, not that they faithfully trace the computation. The faithfulness limitation is fundamental to any method that trains on self-generated explanations without an independent mechanism for verifying reasoning quality.

7. Implications and Future Directions

How This Work Changes the Landscape

STaR represents a methodological reframing rather than a paradigm shift. It does not introduce a new model architecture, a new training objective, or a new form of supervision. What it changes is the default assumption about where rationale training data must come from. Before STaR, the tacit consensus was that high-quality rationales required external sources: human annotators (Rajani et al., 2019), template-driven generation (Nye et al., 2021), or at minimum a stronger teacher model. STaR demonstrates that a model can generate its own rationale training data using nothing more than answer correctness as a filter—and that doing so yields accuracy improvements comparable to increasing model scale by orders of magnitude on certain tasks.

The landscape shift is therefore from "rationales must be provided" to "rationales can be bootstrapped." This is not a universal claim—the paper is explicit that the base model must already possess above-chance few-shot reasoning, and that STaR fails entirely on models below a capability threshold (GPT-2 could not bootstrap even on arithmetic). But for models that clear this bar, the bottleneck moves from annotation cost to answer availability. Questions with answers are far cheaper to obtain than questions with detailed step-by-step solutions. This reframing makes rationale-based training practical for any domain where answer-labeled datasets exist or can be constructed cheaply.

STaR also provides a unifying diagnostic framework for understanding why prior work on self-improvement and explanation quality reached contradictory conclusions. The paper's RL framing (Section 3.1, Equation 1) explains why filtering by outcome works: the indicator reward $\mathbf{1}(\hat{y} = y)$ provides an unbiased gradient signal for which rationales to imitate, even though it never evaluates reasoning quality directly. This reconciles the tension between works showing that explanations help (Rajani et al., 2019; Wei et al., 2022) and works showing that post-hoc explanations don't improve accuracy (Camburu et al., 2018): the key is whether the rationale is generated before the answer (constraining the answer distribution) or after (decorrelated from the decision). STaR's mechanism—generate first, filter by outcome, retrain—enforces the former causal structure.

The paper also resolves an apparent contradiction between the success of chain-of-thought prompting (which improves accuracy with just a few exemplar rationales) and its limitations (which underperform fine-tuned models by large margins). The few-shot CoT baseline achieves only 36.6% on CQA with GPT-J, while STaR reaches 72.5%. The gap is not because few-shot CoT is fundamentally limited—it's because the model's latent reasoning capability is underexploited by surface-level prompting alone. STaR extracts this latent capability through iterative self-training, demonstrating that the few-shot baseline is a lower bound on what the model can do, not an upper bound.

Several research directions become more attractive in light of this work:

• Self-supervised rationale generation for domains without answer labels becomes a natural next step. The paper mentions using majority voting (Wang et al., 2022) to create pseudo-ground-truth from unlabeled questions, but does not test it. This direction is now more plausible because STaR provides a working template for the bootstrapping loop—the open question is how noisy the pseudo-labels can be before the loop degrades.

• Investigating the gap between perceived rationale quality and actual faithfulness becomes more urgent. The human evaluation shows STaR rationales are preferred by crowdworkers, but the paper explicitly acknowledges they may not reflect the model's internal processing (Appendix G). As STaR-style methods are deployed in higher-stakes settings, the distinction between persuasive rationales and faithful rationales becomes critical.

• Combining STaR with process-based verification—training a separate model to evaluate reasoning steps rather than just final answers—becomes a natural extension. The paper's RL framing suggests that a learned value function (critic) could reduce the variance of the gradient estimate and allow training on partial reasoning chains, not just complete correct solutions.

Directions that become less attractive after this work:

• Human annotation of large rationale datasets for standard reasoning benchmarks looks increasingly unnecessary. STaR achieves 72.5% on CQA—within 0.5 points of a 30× larger fine-tuned model—using only 10 human-provided exemplar rationales. For tasks within a model's capability range, the marginal value of additional human-written rationales is likely small compared to the cost.

• Purely few-shot approaches without any fine-tuning remain viable for zero-shot deployment but cannot approach the accuracy of STaR-trained models on the same base architecture (36.6% vs. 72.5% on CQA). The few-shot paradigm is not obsolete, but its accuracy ceiling on reasoning tasks is now better characterized: it leaves substantial latent capability untapped.

---

Follow-Up Research This Work Enables

1. Quantifying the minimum capability threshold for successful bootstrapping. The paper reports a binary result: GPT-2 fails, GPT-J succeeds. A systematic study would evaluate STaR across a range of model scales (e.g., GPT-2 1.5B, GPT-J 6B, GPT-NeoX 20B, and several GPT-3 sizes) on a fixed reasoning task, measuring the relationship between few-shot CoT accuracy and final STaR accuracy. The key question: is there a smooth phase transition (gradual improvement in STaR's effectiveness as few-shot accuracy increases), or a sharp threshold (STaR either works or fails completely)? If the latter, what few-shot accuracy is needed? On CQA, few-shot CoT with GPT-J achieves 36.6% and STaR reaches 72.5%—does STaR still work at 25% few-shot? At 30%? Characterizing this threshold would tell practitioners when STaR is worth attempting, and would inform model scaling decisions: if you need to train a model to reach the threshold, how much pretraining compute does that require?

2. STaR with process-supervised verifiers replacing outcome-only filtering. The paper filters rationales by final answer correctness—a binary outcome reward. This discards rationales that contain mostly correct reasoning but make a single arithmetic error at the end, and it accepts rationales that reach the correct answer through flawed reasoning. A natural extension is to train a process reward model (PRM) on the STaR-generated dataset itself, then use it to score individual reasoning steps. In a subsequent STaR iteration, the PRM could provide denser training signal: keep rationales where all steps score above a threshold, or use the step scores as importance weights. This directly addresses the over-optimization problem the paper documents in Section 5 (high-temperature sampling producing correct answers through incorrect reasoning) by replacing the binary filter with a step-level quality estimate. A strong follow-up would compare STaR + PRM filtering against vanilla STaR on GSM8K, where the outcome-only filter is weakest (only 25–28.7% of training data retained) and step-level filtering might salvage partially correct solutions.

3. Rationalization diagnosis: when does $p(r \mid x, y)$ help, and why? The paper's rationalization results are sharply task-dependent: +3.7 points on CQA, +0.6 points on GSM8K. The paper hypothesizes this is because conditioning on the answer constrains the rationale space more for multiple-choice QA than for open-ended math. A diagnostic study would directly measure this: on a fixed set of problems the model fails, compare the diversity and quality of rationales sampled from $p(r \mid x, y)$ vs. $p(r \mid x)$ using multiple samples per problem and human evaluation of reasoning quality. If $p(r \mid x, y)$ produces higher-quality rationales (not just more answers that match $y$), then rationalization is teaching genuine backward reasoning. If it only produces rationales that are syntactically consistent with $y$ but logically shallow, then the mechanism is closer to template-matching. This experiment would determine whether rationalization is limited to tasks with small answer spaces or can be made generally useful.

4. STaR on a larger model: does the benefit diminish or compound? The paper's headline result compares STaR on GPT-J (6B) to GPT-3 (175B) without STaR. The missing experiment is STaR applied to GPT-3 itself. If STaR on GPT-3 achieves, say, 85% on CQA (vs. 73% for GPT-3 answer-only), then self-generated reasoning provides value independent of scale. If it achieves only 75%, then STaR primarily compensates for limited model capacity and the benefit saturates. This matters because the paper's framing—"STaR lets a model improve itself"—implies generality across scales, but the evidence only covers the 6B-parameter regime. A follow-up using a model family with publicly available checkpoints at multiple scales (e.g., LLaMA 7B, 13B, 33B, 65B) would characterize how STaR's gains scale with base model capability. The hypothesis: STaR's absolute accuracy improvement may shrink as few-shot baselines rise, but the relative efficiency gain (accuracy per unit of human annotation effort) should remain large or even grow if larger models generate higher-quality self-training data.

5. STaR for non-reasoning tasks: probing the boundary conditions. The paper evaluates on arithmetic, commonsense QA, and grade-school math—all tasks with clear correctness criteria where the model must apply knowledge through multi-step inference. What happens on tasks that require factual recall rather than reasoning (e.g., closed-book QA like TriviaQA), or tasks where correctness is subjective (e.g., summarization quality), or tasks where the reasoning is implicit in generating structured output (e.g., code generation with unit tests)? A boundary-condition study would apply STaR across a taxonomy of task types, measuring whether the bootstrapping loop provides value beyond standard fine-tuning. For factual recall, STaR might generate rationales that are post-hoc justifications disconnected from retrieval, providing no accuracy benefit. For code generation, unit tests provide a natural outcome filter, and STaR could bootstrap from a few example programs with comments to generate commented solutions—a direct extension that tests the method's generality.

6. The overfitting diagnosis: what happens if you don't restart from scratch each iteration? The paper's design choice to retrain from the original pretrained GPT-J at each iteration is motivated by avoiding overfitting, but the alternative—continual fine-tuning of a single model—is never evaluated. A negative-result study would compare the two protocols on arithmetic: restart-from-scratch vs. continual fine-tuning. The hypothesis from the paper's discussion is that continual fine-tuning causes the model's output distribution to drift, compounding generation artifacts across iterations and eventually degrading accuracy. If this is confirmed, it establishes an important boundary condition for self-training loops: the data distribution and the model distribution must remain close enough for bootstrapping to be stable. If it is disconfirmed—if continual fine-tuning works equally well or better—then the restart protocol is unnecessary computational overhead, and STaR can be simplified. Either outcome is informative.

---

Practical Applications and Downstream Use Cases

1. Low-cost rationale dataset construction for domain-specific reasoning tasks. The most direct application is using STaR to create rationale-augmented training sets for specialized domains where human annotators are expensive or scarce. For example, a medical question-answering dataset with doctor-verified answers but no reasoning traces could be augmented with STaR-generated rationales, using a handful of exemplar reasoning chains from a medical textbook. The STaR-trained model would then generate step-by-step diagnostic reasoning, which is valuable both for accuracy (as the CQA results demonstrate, rationale-based training outperforms answer-only training by 12.5 points on GPT-J) and for interpretability in high-stakes settings. The cost is one-time: a few exemplar rationales and a fine-tuning budget, after which the model generates rationales for all training examples. On CQA, STaR produced useful training data for 86.7% of questions starting from 10 exemplars—a nearly 1000× leverage ratio in terms of human-written vs. model-generated rationales.

2. Bootstrapping reasoning in small on-device models. GPT-J at 6B parameters is modest by contemporary standards, and STaR improved its CQA accuracy from 60.0% (answer-only fine-tuning) to 72.5%. If similar gains hold for smaller models that clear the few-shot reasoning threshold, this enables on-device deployment of models that can perform multi-step reasoning without cloud connectivity. A 1–3B parameter model trained with STaR on a domain-specific task could provide step-by-step explanations for its answers, which is valuable for educational applications (showing students how to solve a math problem, not just the answer) or assistive technology (explaining a recommendation rather than just providing it). The paper's arithmetic results suggest this is plausible: STaR on GPT-J learned multi-digit addition from near-zero few-shot performance to 89.5% accuracy, demonstrating that systematic reasoning can be bootstrapped even when the initial capability is minimal.

3. Data augmentation for self-improving AI pipelines. STaR's core loop—generate solutions, filter by correctness, retrain—is a template for self-improving systems that can be applied whenever a correctness oracle exists. In code generation, unit tests provide the correctness filter; a model could iteratively generate solutions to programming problems, keep those that pass tests, and retrain on the successful implementations. In formal theorem proving, a proof checker provides the filter; a model could bootstrap from a few example proofs to generate training data for new theorems. The STaR paper demonstrates the viability of this pattern: the loop works as long as the initial model has non-trivial success on a subset of problems. The efficiency depends on how many iterations are needed and what fraction of problems remain unsolved—on GSM8K, the fraction was 71–75%, suggesting that domains with low initial success rates may not benefit. But for domains where the model can solve a small but non-trivial fraction of problems with few-shot prompting, STaR provides a template for turning that fraction into a larger fraction through iterative self-training.

When to Prefer This Method

The paper does not explicitly position STaR against named alternative methods with a clear decision rule. However, it implicitly defines a tradeoff space along two axes: annotation cost (human effort to produce training data) and base model capability (whether the model has above-chance few-shot reasoning). From the paper's results and stated limitations, a practitioner can derive the following guidance:

• Prefer STaR over human rationale annotation when a dataset of question-answer pairs already exists, the task requires multi-step reasoning (math, commonsense QA, logical deduction), and a handful of high-quality exemplar rationales (10–20) can be written once. The CQA results show that 10 exemplar rationales plus STaR produces accuracy within 0.5 points of a 30× larger answer-only model, without the cost of annotating thousands of reasoning traces.

• Prefer STaR over answer-only fine-tuning when interpretability of model outputs matters, or when rationale generation is expected to improve generalization (as the arithmetic out-of-distribution digit results suggest). The CQA gain is 12.5 points over answer-only fine-tuning at the same model scale, and the rationales provide human-readable justifications.

• Do not prefer STaR when the base model's few-shot chain-of-thought performance is near chance across all problem difficulties (no subset of problems provides seed training data), when ground-truth answers are unavailable, or when the reasoning task involves open-ended generation without a clear correctness criterion. In these cases, STaR's filtering mechanism has no signal to operate on, and the loop cannot begin.