Democratizing Reasoning Ability: Tailored Learning from Large Language Model

ArXiv: 2310.13332

🎯 Pitch

Even a 175B teacher model fails to improve a 6B student's reasoning if it can't see the student's mistakes—but when the student feeds back its wrong answers in a multi-round dialogue, the teacher generates targeted training data that nearly doubles math accuracy (from 15.6% to 33.1% on GSM8K).

1. Executive Summary

This paper proposes a tailored learning approach to distill reasoning ability from large black-box LLMs into smaller open-source LMs, aiming to democratize the emergent reasoning capability typically reserved for models exceeding 100B parameters. Using GPT-J (6B) as the student and ChatGPT (175B) as the teacher on mathematical (GSM8K, MultiArith, SVAMP) and commonsense (CSQA, StrategyQA) benchmarks, the method introduces a multi-round interactive learning paradigm (where the student exposes its mistakes as feedback to the teacher, who then generates customized training data targeting those specific deficiencies) and self-reflection learning (a contrastive triplet loss that pushes the student to distinguish correct rationales from its own wrong ones). The combined approach achieves substantial gains over one-round distillation—improving GSM8K accuracy from 15.6% to 33.1% (+17.5 points) and SVAMP from 47.7% to 55.0%—while outperforming concurrent distillation methods that treat the LLM merely as a data annotator, establishing that student-aware feedback produces measurably better reasoning transfer than static one-shot rationale generation.

2. Context and Motivation

The Core Problem: Reasoning Ability Is Locked Inside Massive Models

The fundamental issue this paper confronts is a deep inequity in the current landscape of language model capabilities. Wei et al. (2022a,b) established that emergent abilities—particularly complex reasoning—only manifest in models exceeding roughly 100 billion parameters. Below this threshold, smaller language models can follow instructions reasonably well (as demonstrated by Vicuna and Alpaca; Chiang et al., 2023; Taori et al., 2023) but consistently fail at tasks requiring multi-step logical deduction, mathematical problem-solving, or implicit commonsense inference.

This creates a practical crisis for the open-source and research communities. Models like GPT-3 (175B), PaLM (540B), and ChatGPT are closed-source, expensive to serve, and impossible to inspect or modify. A researcher wanting to build a math tutoring system, verify reasoning chains, or deploy a lightweight reasoning engine on-device faces an impossible choice: use a small, open model that can't actually reason, or pay per-query to access a proprietary black box with no transparency, reproducibility, or data privacy guarantees.

The problem is not merely one of convenience—it's a structural barrier to scientific progress. If only a handful of organizations can access reasoning-capable models, then research on improving reasoning, auditing reasoning, or applying reasoning in sensitive domains remains gatekept. The paper frames this explicitly as a democratization problem: "the democratization of the exclusive reasoning ability" (Section 1). The ambition is to bridge the gap between the haves and have-nots in reasoning capability.

Why This Problem Is Difficult: The Rationale-Generation Bottleneck

The difficulty of teaching reasoning to smaller models stems from a chicken-and-egg problem specific to chain-of-thought (CoT) reasoning. CoT works by prompting a model to generate intermediate reasoning steps (rationales) before arriving at a final answer, but this only succeeds when the base model already possesses the capacity to produce coherent rationales. Wei et al. (2022a) showed that the CoT performance curve exhibits a phase transition: below roughly 100B parameters, asking a model to "think step by step" produces nonsensical or circular reasoning that doesn't improve answer accuracy. The model can't self-generate the very training signal it needs to learn.

This means you cannot simply prompt a small LM to improve itself through introspection the way you might with ChatGPT. The reasoning capability must be imported from somewhere else—specifically, from a larger model that already possesses it. But how exactly should that importation work? The paper argues that the existing approaches to this knowledge transfer problem are fundamentally incomplete.

Prior Approaches and Their Limitations

Approach 1: Training on human-annotated rationales. The most direct solution is to collect datasets where human experts write out step-by-step reasoning for each question, then fine-tune a small model on those examples. This is what Chung et al. (2022) demonstrated—smaller LMs can partially master CoT when trained on rationale-augmented data. However, this approach runs headlong into a scalability wall. Most existing reasoning datasets (like GSM8K, SVAMP, or CSQA) provide only question-answer pairs, not intermediate reasoning steps. Manually annotating thousands of problems with high-quality rationales is prohibitively expensive, especially for specialized domains. The paper notes this directly: "most existing reasoning datasets lack high-quality rationale annotations, and manual labeling them can be costly" (Section 1). Even when annotations exist (e.g., the GSM8K training set has some), they represent a fixed, static teaching signal that cannot adapt to the learning needs of a particular student model.

Approach 2: The STaR self-improvement loop. Zelikman et al. (2022) proposed STaR (Self-Taught Reasoner), an iterative bootstrapping method: (1) prompt the model to generate rationales for training questions, (2) filter to keep only those leading to correct answers, (2) fine-tune on those filtered rationales, and (4) repeat. This avoids the cost of human annotation by letting the model teach itself. However, as the paper's Table 1 shows, STaR performs poorly on multi-step mathematical reasoning tasks when applied to smaller LMs: 10.7% on GSM8K and 26.7% on SVAMP with GPT-J 6B. The reason is straightforward: the quality ceiling of self-generated rationales is bounded by the model's current capability. If the base model cannot produce coherent reasoning chains, filtering for correct answers doesn't help—the few chains that accidentally reach correct answers do so through flawed or nonsensical reasoning that provides poor training signal. STaR works well when the model already has some reasoning competence (e.g., on simple commonsense tasks requiring single-step inference), but collapses on tasks requiring genuine multi-step deduction. The paper highlights this contrast: "Though STaR performs well in CSQA which often only involves single-step reasoning, the self-generated rationales encounter difficulties when applied to other multi-step reasoning tasks" (Section 4.4).

Approach 3: One-shot distillation from LLMs. Several concurrent works (Ho et al., 2022; Fu et al., 2023b; Magister et al., 2023; Shridhar et al., 2023) recognized that LLMs could serve as rationale generators, avoiding both human annotation costs and the self-training quality ceiling. The recipe is simple: (1) feed training questions to a large LLM (typically GPT-3 or ChatGPT), (2) collect the generated rationales, and (3) fine-tune a smaller LM on those rationale-augmented examples. This is effectively black-box knowledge distillation applied to reasoning.

This approach yields meaningful improvements—the paper's own "One-Round Distillation" baseline achieves 15.6% on GSM8K and 47.7% on SVAMP, well above STaR's 10.7% and 26.7%—but the paper identifies a critical conceptual limitation. In all these methods, the LLM acts as a data annotator, not a teacher. The distinction matters. An annotator produces rationales for a dataset independent of any particular student. A teacher produces rationales for a student, tailored to that student's specific misconceptions and learning state. The paper argues (Section 1) that "the LLMs are not aware of the weaknesses of the smaller LMs, thereby hindering their powerful ability to analyze and provide targeted feedback, which undermines the effectiveness of the reasoning distillation."

Consider the analogy: if a student makes consistent errors in subtracting before dividing in multi-step arithmetic, a teacher who sees those errors can generate examples that specifically highlight the correct order of operations. An annotator who never sees the student's work can only generate generic correct solutions. The student might learn something, but they won't learn what they specifically need. This is not just a philosophical distinction—the paper's ablation results in Table 3 demonstrate that providing student feedback to the LLM increases both the number of successfully generated correct rationales and the downstream accuracy of the student, with a +1.7% accuracy gain on GSM8K in the second round alone.

Approach 4: Self-correction / self-reflection in LLMs. A separate line of work (Huang et al., 2022; Madaan et al., 2023; Shinn et al., 2023; Pan et al., 2023) showed that large models can improve their outputs by reflecting on their own mistakes—essentially, generating a critique of their initial answer and then producing a revised version. However, this ability is itself an emergent property that smaller LMs lack. The paper draws a direct analogy to human learning: "Another crucial paradigm for human learning lies in self-reflection on self-made mistakes" (Section 1). Humans do not only learn from teachers; they also learn by analyzing where they went wrong and why. The question the paper poses is whether a small LM can be trained to benefit from its own mistakes, even if it can't spontaneously self-correct the way ChatGPT can.

The limitation of prior work here is two-fold. First, no prior method had combined mistake-driven learning with teacher-guided learning in a single framework for small LMs. Second, even when mistakes are available, it's not obvious how to operationalize "learning from mistakes" for a language model—should the model be trained to produce corrections? To recognize flawed reasoning? To score its own outputs? The paper's contrastive triplet loss approach is one concrete instantiation of this idea, but the broader point is that the field lacked any systematic method for making small LMs benefit from their own errors.

How This Paper Positions Itself

The paper positions its contribution at the intersection of two previously disconnected research threads: knowledge distillation from black-box LLMs and iterative self-improvement through reflection. The central thesis is that neither approach alone is sufficient for democratizing reasoning, but that combining them—and crucially, connecting them through a multi-round feedback loop—yields gains beyond what either achieves independently.

The key intellectual move is reframing the LLM's role from "annotator" to "teacher." A teacher, unlike an annotator, needs to see the student's work. The paper's "multi-round interactive learning paradigm" (Section 3) formalizes this: in each round, the student LM takes an exam on the training set, collects its mistakes, provides those mistakes as feedback to the teacher LLM, and receives customized correct rationales in return. The teacher's output is explicitly conditioned on the student's wrong reasoning (via the prompt template in Figure 3), which shows the teacher both that the student failed and how the student failed. The paper's case study (Table 2) illustrates this concretely: for a StrategyQA question about whether mail carriers need multiple uniforms, the student argues "they have one uniform," and the teacher (armed with this feedback) produces a rationale that directly addresses this misconception by enumerating different uniform types for different conditions—a much richer response than the generic "they work outdoors" answer produced without feedback.

The paper also positions self-reflection not as an alternative to teacher guidance but as a complementary learning signal. The contrastive loss in Equation 2 pushes the model to produce different internal representations for correct and incorrect reasoning chains, regardless of where the correct chains came from (teacher or otherwise). This is a form of representation learning that operates orthogonally to the language modeling objective: the language modeling loss teaches the model what to generate, while the contrastive loss teaches it to distinguish quality in what it generates. The paper hypothesizes—and the t-SNE visualizations in Figure 4 support—that this dual supervision produces more sharply separated representations of good and bad reasoning, which in turn makes the model less likely to generate flawed chains.

A final aspect of the paper's positioning is its pragmatic stance on compute and data efficiency. The authors explicitly choose GPT-J (6B) as their primary student model and conduct a feasibility study with models from 760M to 2.7B parameters (Section 5.4), demonstrating that the approach works even with "individual affordable computation resources" (Table 7). This is not a method that requires a 70B-parameter student to work. The methods outperform "Specializing" (Fu et al., 2023b), which used an 11B model and 130k training rationales, while using only a 6B model and 54k rationales. The paper frames this as democratization in both directions—not just making reasoning available, but making the process of acquiring reasoning accessible to researchers without massive GPU clusters.

Unresolved Tensions the Paper Inherits

Several tensions from prior work carry into this paper, some of which it addresses and some of which it leaves as open challenges:

The fidelity gap. The teacher (ChatGPT, 175B) and student (GPT-J, 6B) differ by roughly 30× in parameters. The student will simply never match the teacher's capabilities regardless of training data quality. Where is the ceiling? The paper shows substantial gains but acknowledges that "a substantial gap still remains" (Limitations, point 1), leaving open the question of how much reasoning can actually be transferred versus how much is fundamentally tied to model scale.

Rationale quality evaluation. The paper—like nearly all work in this area—evaluates rationale quality solely by whether the final answer is correct. A rationale could be logically incoherent but accidentally reach the right answer, or be perfectly reasoned but make an arithmetic slip. The paper acknowledges this limitation explicitly (Limitations, point 4) and calls for more trustworthy evaluation criteria such as process reward models or GPT-4-based scoring, but does not implement these. This means some fraction of the "correct rationale" training data likely contains flawed reasoning, and some "wrong" rationales may contain valuable reasoning steps marred by a single error.

The upfront cost of teacher queries. Every round of learning requires querying ChatGPT thousands of times to generate customized rationales. The paper reports requesting 5,701 LLM calls for GSM8K in round 2 alone (Table 3), with a success rate of 92% (5,250 correct rationales generated). At the time of writing (early 2023), this would have incurred non-trivial API costs. The paper doesn't discuss the economic tradeoff, focusing instead on capability gains, but this is a practical consideration for democratization—the method democratizes the result (a reasoning-capable small model) but not necessarily the process (still depends on a proprietary LLM).

The negative transfer problem. The generalization results in Table 11 reveal a subtle issue: while in-domain performance improves, out-of-domain performance sometimes degrades. For example, a student trained on GSM8K (round 1) drops from 34.5% to 28.4% on CSQA, and from 47.2% to 38.3% on StrategyQA. This is consistent with fine-tuning on domain-specific reasoning data causing catastrophic forgetting of general capabilities. The multi-round paradigm partially mitigates this—note the CSQA-trained model improves on StrategyQA from 48.0% (round 1) to 51.1% (last round)—but the paper doesn't deeply analyze why some cross-task transfers improve while others degrade. This hints at a complex interaction between the reasoning patterns learned from the teacher and the pre-existing knowledge in the base model.

3. Technical Approach

3.1 Reader Orientation

This paper builds a multi-round training system where a 6B-parameter student language model learns to reason by alternating between taking exams, sending its mistakes to a 175B teacher LLM for targeted feedback, and training on both the teacher's customized rationales and a contrastive signal from its own errors. The system solves the problem of how to transfer complex reasoning ability from a black-box large model to an open-source small model when the small model cannot spontaneously generate coherent reasoning chains—it does so by making the knowledge transfer bidirectional, with the student's failures actively shaping what the teacher teaches next, rather than treating the LLM as a static rationale factory.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five interconnected components operating in a cyclic loop:

1. Student LM (GPT-J 6B) — a publicly available decoder-only language model that we want to endow with chain-of-thought reasoning ability. It starts from pre-trained weights with effectively zero mathematical reasoning capability (2.7% on GSM8K).

2. Teacher LLM (ChatGPT 175B) — a closed-source black-box model accessed via API that already possesses strong reasoning ability and can generate high-quality step-by-step rationales when prompted. It serves as the source of reasoning knowledge.

3. Exam Module — a procedure that runs the current student LM on the training set, collects generations where the final answer is incorrect, and packages those (question, wrong rationale, wrong answer) triples as the student's "feedback" to the teacher. This is the student→teacher communication channel.

4. Rationale Generation Pipeline — a prompt template (Figure 3) that wraps the question, the student's wrong solution, and the ground-truth answer into a request for the teacher LLM to produce corrected reasoning. The teacher generates multiple diverse correct rationales per error. This is the teacher→student communication channel.

5. Training Module — a joint optimization procedure that fine-tunes the student on two complementary losses: a standard language modeling loss on the teacher's correct rationales (learning what to generate) and a contrastive triplet loss that pushes the student's internal representations of correct vs. incorrect reasoning chains apart (learning how to distinguish quality).

Information flows cyclically: the student takes an exam → mistakes are collected → mistakes are sent to the teacher via the prompt template → the teacher generates customized correct rationales → these rationales populate the training set → the student trains on the combined language modeling + contrastive objective → the improved student takes the next exam → repeat until performance plateaus. The first round is a cold-start: since the untrained student produces incoherent noise, the initial rationales are generated without student feedback to bootstrap the process.

3.3 Roadmap for the Deep Dive

The technical breakdown follows the chronological order of a single learning round, since each component's output feeds directly into the next:

• First, the Exam mechanism (Section 3.1): how the student's mistakes are collected, filtered, and prepared as feedback. This defines the student→teacher signal.
• Second, the Prompt Template and Teacher Rationale Generation (Section 3.2): the exact prompt structure that conditions ChatGPT on the student's errors, why it includes a hint with the ground-truth answer, and how multiple diverse correct rationales are generated per mistake.
• Third, the Self-Reflection Contrastive Loss (Section 3.3, first part): the triplet objective that operates on pairs of correct and incorrect reasoning chains for the same question, why cosine similarity in the last-token hidden state is the chosen representation, and what the margin $\rho$ controls.
• Fourth, the Language Modeling Loss on Customized Rationales (Section 3.3, second part): how the teacher's output enters standard autoregressive fine-tuning, the role of fixed demonstrations prepended to each example, and why training on diverse reasoning paths matters.
• Fifth, the Joint Training Objective (Section 3.3, third part): how the two losses are combined via a single hyperparameter $\lambda$, the tradeoffs this creates, and the empirical finding that $\lambda = 0.5$ works across tasks without heavy tuning.
• Sixth, the Multi-Round Loop and Cold-Start Bootstrap (Section 3.4): the outer algorithm that iterates exams and training, why the first round skips student feedback, and the convergence criterion based on error rate plateau.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that reasoning distillation becomes more effective when the teacher LLM receives the student's mistakes as conditioning context, and when the student is simultaneously trained to discriminate correct from incorrect reasoning via a contrastive objective.

The Exam: Collecting Student Mistakes

Before the teacher can provide targeted feedback, the system must identify what the student gets wrong and how. This is treated as a structured data collection step rather than a real evaluation, though the error rate serves double duty as a progress metric.

Given a training dataset $\mathcal{D}_{\text{train}} = \{(x, y)\}$ where $x$ is a natural language question and $y$ is the ground-truth answer (usually a number, a multiple-choice label, or a yes/no token), the student LM $f$ generates an output $f(x) = [\hat{r}, \hat{y}]$ consisting of a generated rationale $\hat{r}$ followed by a generated answer $\hat{y}$. The rationale is the chain-of-thought reasoning steps; the answer appears at the end, typically after a delimiter like "Answer:".

The paper adopts the same pragmatic evaluation criterion used throughout the CoT distillation literature: a rationale is classified as correct if and only if its final answer matches the ground-truth $y$. This is acknowledged as a crude proxy—a rationale could be logically flawed but accidentally reach the right answer, or be perfectly reasoned but contain an arithmetic slip—but it is computationally free and avoids the need for human rationale grading. The authors explicitly flag this as a limitation (Section "Limitations", point 4), calling for future work on "more trustworthy criteria to evaluate the quality of rationales."

The mistake set $\mathcal{D}_{\text{neg}}$ is formally defined as:

$\mathcal{D}_{\text{neg}} = \{(x, \hat{r}, \hat{y}) \mid \hat{y} \neq y, (x, y) \in \mathcal{D}_{\text{train}}\}$

where $x$ is the original question, $\hat{r}$ is the generated rationale that led to the wrong answer, and $\hat{y}$ is the wrong answer itself. The condition $\hat{y} \neq y$ filters out any generation that accidentally produced the correct answer, regardless of rationale quality.

What it computes: for each question in the training set, the student generates up to 4 candidate solutions (the paper uses sampling-based decoding with temperature 1.0, top-k 50, and maximum generation length 128 tokens, returning 4 sequences per input; see Table 9). Any generation whose extracted final answer does not match the ground truth is retained as a $(x, \hat{r}, \hat{y})$ triple. Generations with correct answers are discarded from $\mathcal{D}_{\text{neg}}$ (though they may inform the training set construction in other ways). The resulting set $\mathcal{D}_{\text{neg}}$ serves two distinct downstream purposes: (1) as the feedback signal sent to the teacher LLM, and (2) as the negative contrastive samples in self-reflection learning.

Why this form: collecting multiple wrong answers per question (up to 4) via stochastic sampling rather than a single greedy decode increases the diversity of errors exposed to the teacher. A student might fail the same problem in different ways—mixing up the order of operations, misreading a quantity, or applying the wrong formula—and exposing multiple failure modes gives the teacher richer signal about the student's weaknesses. The choice of sampling parameters (temperature 1.0, top-k 50) balances diversity against coherence; higher temperature would produce more diverse but potentially nonsensical errors that contain no useful signal for the teacher. The paper does not conduct an ablation of these decoding parameters.

A critical design choice occurs in the first round: the untrained student LM produces outputs that are "noisy generations which are unhelpful as the feedback to the teacher LLM" (Section 3.4). For round 0, no exam is conducted; instead, the system directly requests rationales from the teacher for all training questions without any student feedback context (the $\hat{r}$ slot in the prompt template is left empty or replaced with a null value). The exam-collection loop only activates from round 1 onward, once the student has acquired baseline competence from the initial distillation round.

The Prompt Template: Conditioning the Teacher on Student Errors

The teacher LLM is ChatGPT, accessed via API with no weight access or fine-tuning capability. The only way to influence its outputs is through the prompt text. The paper's central technical contribution in the teacher interaction is the design of a prompt template $\mathcal{T}$ that integrates student feedback directly into the request for a corrected rationale.

The template, illustrated in Figure 3 and concretized with examples in Table 12, has the following structure:

Question: {x}
Wrong Solution: {r̂, ŷ}
Please correct the wrong solution by using better reasoning steps.
Hint: The final answer should be {y}.
Better Reasoning:

where:

• $\{x\}$ is the original question text,
• $\{\hat{r}, \hat{y}\}$ is the student's incorrect rationale-and-answer pair (concatenated exactly as the student generated it),
• $\{y\}$ is the ground-truth answer, and
• "Better Reasoning:" is the prompt for the teacher to begin generating the corrected rationale.

What this template computes (functionally): it takes three pieces of information—the question, the student's specific wrong solution path, and the correct answer—and packages them into an instruction that asks the teacher to produce a corrected reasoning chain. The teacher's output is the corrected rationale $r$. If the teacher's generated rationale leads to the correct answer (checked by extracting the answer from the teacher's output), the triple $(x, r, y)$ is added to the customized training set $\mathcal{D}_{\text{train}}$.

Operationally, the system requests 4 diverse rationales per mistake (using sampling with temperature 1.0, top-p 0.9, maximum 128 tokens; see Table 9), but only retains those whose final answer is correct. The paper calls this the "# Success" metric in Table 3: out of 5,701 requests to the teacher for GSM8K round 2, 5,250 produced correct rationales (a 92.1% success rate). The diversity constraint (top-p sampling rather than greedy decoding) is deliberate—prior work (Ho et al., 2023; Fu et al., 2023b) showed that training on multiple different correct reasoning paths for the same question improves the student's generalization, since it prevents the student from memorizing a single solution template.

Why this form (design choices):

1. The hint with the ground-truth answer. The template explicitly tells the teacher the correct answer ("Hint: The final answer should be {y}"). This follows Zelikman et al. (2022), who found that providing the answer increases the teacher's accuracy in generating correct rationales. Without the hint, the teacher might also make reasoning errors, producing incorrect rationales that get filtered out and waste API calls. The paper explicitly states the motivation: "to improve the LLM's accuracy and reduce the costs of calling APIs" (Section 3.2). This is a practical engineering decision, not a theoretical one—the teacher's rationales need to be correct to serve as training data, and giving the answer makes correctness much more likely.

2. Including the student's wrong solution, not just the question. This is the key innovation over prior distillation work. The student's wrong rationale serves two simultaneous functions for the teacher:

   • Diagnostic signal: it shows the teacher what specific error the student made. In the GSM8K example from Section 3 (Table 2), the student incorrectly computes "Harry has 68 − 30 = 38 apples" without then dividing by 2. The teacher's response explicitly addresses this missing step: "Harry has 38 / 2 = 19 apples."
   • Negative demonstration: it serves as a contrastive example that may help the teacher avoid reproducing the same mistake. The paper hypothesizes that seeing a wrong chain "can help LLM increase the chance of generating correct rationales" (Section 3.2), though the mechanism is not formally analyzed.

   The ablation in Table 3 confirms the causal effect: removing student feedback from the template (replacing it with just the question) reduces both the number of successfully generated correct rationales (from 5,250 to 4,641 for GSM8K, a 13.1% drop) and the downstream student accuracy (a 1.7 percentage point drop on GSM8K). The gap is even larger on StrategyQA (317 vs. 134 successful rationales), suggesting that the benefit of student feedback is larger when the questions require more nuanced reasoning where the teacher might otherwise produce diverse but incorrect answers without guidance.

3. The "Please correct" instruction framing. The template uses an explicit correction instruction rather than asking for a rationale from scratch. This frames the task as targeted remediation—the teacher should fix the specific error in the provided wrong solution, not produce a generic correct solution that might address different aspects of the problem. The case study in Table 2 supports this: for the StrategyQA question about mail carriers, the teacher without feedback produces a generic answer ("Mail carriers work outdoors... they need different clothes"), while the teacher with feedback produces a more detailed response that enumerates specific uniform types and directly counters the student's misconception that carriers have only one uniform.

4. Open-ended generation with post-hoc filtering. The teacher generates freely and correctness is verified after the fact by answer extraction, rather than constraining the teacher's generation (e.g., through constrained decoding). This preserves the teacher's full expressiveness but incurs the cost of generating and filtering rationales that turn out to be incorrect (roughly 8% for GSM8K). The paper does not explore whether iterative refinement (asking the teacher to retry on failures) would improve the yield rate.

Self-Reflection Learning: The Contrastive Triplet Objective

The student does not only learn from the teacher's correct rationales—it also learns from its own mistakes through a contrastive objective that shapes the model's internal representations of reasoning paths. The intuition, articulated in Section 3.3, is that "these mistakes can complement correct rationales collected from the teacher LLM to teach the student LM to distinguish bad and good reasoning steps."

The paper implements this as a margin-based triplet loss operating on the hidden states of the last token of each reasoning chain. Let:

• $h_{(r, y)}^{x}$ be the hidden state (output of the final transformer layer) at the position of the last token when the model processes the sequence $[x, r, y]$—that is, a correct reasoning path for question $x$.
• $h_{(\hat{r}, \hat{y})}^{x}$ be the corresponding hidden state for a wrong reasoning path $[x, \hat{r}, \hat{y}]$.
• $h_{(r', y)}^{x}$ be the hidden state for another correct reasoning path for the same question (the "positive" sample in triplet terminology).

The contrastive loss is:

$\mathcal{L}_{\text{cl}} = \mathbb{E}_{\mathcal{D}_{\text{train}}} \max\left(0, \rho - \cos(h_{(r,y)}^{x}, h_{(r',y)}^{x}) + \cos(h_{(r,y)}^{x}, h_{(\hat{r},\hat{y})}^{x})\right)$

where:

• $\cos(\cdot, \cdot)$ is cosine similarity between two vectors,
• $\rho$ is a margin hyperparameter set to $1.0$,
• $h_{(r,y)}^{x}$ is the anchor—a correct rationale for question $x$,
• $h_{(r',y)}^{x}$ is the positive—a different correct rationale for the same question,
• $h_{(\hat{r},\hat{y})}^{x}$ is the negative—a wrong rationale sampled from $\mathcal{D}_{\text{neg}}$ for the same question.

The expectation is taken over the training set; for each anchor sample, the positive and negative are randomly sampled from the set of available correct and wrong rationales for that question, respectively.

What it computes (operationally): for a given question $x$, the model encodes three reasoning paths (the anchor correct path, another correct path, and a wrong path) and extracts the last-token hidden state for each. It then computes two cosine similarities: (1) between the two correct paths (should be high—similar reasoning leads to similar representations), and (2) between the anchor correct path and the wrong path (should be low—different-quality reasoning leads to different representations). The loss penalizes the model when the cosine similarity between two correct paths is less than the cosine similarity between a correct path and a wrong path plus the margin $\rho$. When the inequality $\cos(\text{correct}_1, \text{correct}_2) \geq \cos(\text{correct}, \text{wrong}) + \rho$ holds, the loss is zero; otherwise, the model incurs a penalty proportional to the violation.

Why this form:

1. Cosine similarity on last-token hidden states. The choice of representation is deliberate and motivated by how autoregressive LMs encode sequences. The last token's hidden state aggregates information from the entire prefix through the causal attention mechanism, making it a natural sequence-level representation without requiring an explicit pooling operation or a separate encoder. Using the last-token state means the contrastive signal shapes how the model internally represents the entire reasoning chain, not just individual tokens. This is crucial because reasoning quality is a holistic property—a reasoning chain with a single fatal error near the end is as wrong as one that's nonsensical throughout, and the representation should capture this.

2. Triplet loss over pairwise contrastive loss. A simpler alternative would be a pairwise loss that only pushes correct and wrong representations apart (e.g., $\max(0, \rho - \cos(\text{correct}, \text{wrong}))$). The triplet form adds an explicit positive attraction term: it also encourages different correct reasoning paths for the same question to have similar representations. This is important because the teacher generates multiple diverse correct rationales per question (up to 4), and the student should learn that these different surface forms are semantically equivalent—they all represent valid reasoning for the same problem. Without the positive attraction, the model might learn to distinguish correct from wrong but fail to recognize that different correct solutions belong to the same equivalence class.

3. Margin $\rho = 1.0$. The margin enforces a minimum separation between the correct-correct similarity and the correct-wrong similarity. With cosine similarity bounded in $[-1, 1]$, a margin of 1.0 means the loss is only zero when the two correct chains have similarity at least 1.0 higher than the correct-wrong similarity—which, given the bounded range, effectively requires $\cos(\text{correct}_1, \text{correct}_2) \approx 1$ and $\cos(\text{correct}, \text{wrong}) \approx -1$, i.e., nearly perfectly separated representations. In practice, this strict margin prevents the model from finding degenerate solutions where all representations collapse to similar values; it must allocate representational capacity to distinguish correct from incorrect reasoning.

4. Same-question constraint. The positive and negative samples are always drawn from the same question $x$. This is critical because it forces the model to discriminate reasoning quality independent of topic. If negatives came from different questions, the model could trivially separate correct and wrong chains by detecting topic differences rather than reasoning quality (e.g., "this is about apples, that is about trains"). By keeping the question constant, the only systematic difference between correct and wrong chains is the logical structure of the reasoning, which is precisely what the model should learn to evaluate.

5. Connection to the t-SNE evidence. The paper provides an indirect but compelling validation of this design through Figure 4, which visualizes the last-token hidden states of correct and wrong rationales generated by the student on GSM8K before and after self-reflection training. Without self-reflection, the correct and wrong representations overlap substantially—different colors are intermingled in the 2D projection. With self-reflection, two distinct clusters emerge, with correct rationales concentrated in one region and wrong rationales in another. This suggests that the contrastive loss has indeed reshaped the internal representation space in a way that separates reasoning quality, which should make the model less likely to generate wrong chains (since wrong chains are represented differently from correct ones, the model's generation probability may implicitly favor the "correct" region of representation space).

The paper also quantifies this separation in Table 4 using two metrics: Euclidean distance (the mean Euclidean distance between correct and wrong rationales' last-token hidden states in the full high-dimensional space) and Preference (defined as the likelihood ratio of correct reasoning paths to wrong ones—that is, how much more probable the model considers correct chains than wrong chains averaged over the dataset). On GSM8K, self-reflection increases the distance from 51.00 to 65.08 and the preference from 73.63 to 79.11. On StrategyQA, the effect is even more dramatic: distance increases from 5.03 to 24.78 and preference from 96.54 to 98.91. The paper interprets this as evidence that "self-reflection contributes to aligning the preference of the student LM with correct reasoning paths, while away from self-made wrong ones."

Learning from Customized Feedback: The Language Modeling Objective

While the contrastive loss shapes the student's internal representations, the student still needs to learn how to actually generate coherent reasoning chains—the token-by-token autoregressive generation skill. This is handled by a standard language modeling (next-token prediction) objective on the teacher's customized rationales.

The teacher-generated correct rationales, collected across the current learning round, populate the training set $\mathcal{D}_{\text{train}}$. Each training example is a formatted string:

[demo, x, r, y]

where:

• $\text{demo}$ is a set of fixed demonstration examples prepended to every training instance (3-shot for mathematical tasks, 4–5-shot for commonsense tasks),
• $x$ is the question,
• $r$ is the teacher's correct rationale,
• $y$ is the ground-truth answer.

The language modeling objective is the standard autoregressive negative log-likelihood:

$\mathcal{L}_{\text{lm}} = \mathbb{E}_{\mathcal{D}_{\text{train}}} \log P_f([\text{demo}, x, r, y])$

where $P_f$ is the probability assigned by the student model $f$ to the concatenated sequence, factorized autoregressively across tokens. In practice, this is the standard cross-entropy loss between the model's predicted token distribution and the ground-truth next token at each position, summed over the sequence length.

What it computes (operationally): the student model reads the demonstration examples, the question, the teacher's reasoning, and the answer as a single continuous sequence, and is trained to predict each token conditioned on all previous tokens. The loss is computed over all tokens in the sequence, but the practical effect is concentrated on learning to generate the rationale $r$ and answer $y$ given the question $x$ and few-shot demonstrations $\text{demo}$.

Why this form (design choices):

1. Fixed demonstrations in the prefix. The paper prepends 3-shot (mathematical) or 4–5-shot (commonsense) demonstrations to every training example, following Min et al. (2022), Zelikman et al. (2022), and Fu et al. (2023b). The rationale is that "recent research have shown that training with demonstration examples can improve the in-context learning ability of LMs" (Section 3.3). During inference, the model also receives these same few-shot demonstrations as part of the prompt, so training with them ensures the model learns to use in-context examples effectively. The specific demonstrations are listed in Table 15 and are selected from the training set to provide diverse reasoning templates.

   An empirical nuance: the paper reports that "assigning less weight (0.1) to the fixed demonstration examples than the input sample helps the model focus on the input sample and yield better performance" (Appendix A.2). This means that during training, the loss computed on tokens belonging to the demonstration prefix is multiplied by 0.1, while the loss on the actual question-rationale-answer part receives full weight. This prevents the model from overfitting to reproducing the specific demonstration text at the expense of learning to generalize to new questions.

2. Training on diverse correct rationales per question. For each training question, the teacher generates up to 4 unique correct rationales, and all are included in $\mathcal{D}_{\text{train}}$. This means the student sees multiple valid solution paths for the same problem during training. The paper inherits this design from Ho et al. (2023) and Fu et al. (2023b), who found that training on diverse reasoning paths improves generalization. The intuition: if the student only sees one "canonical" solution per problem, it may memorize surface patterns rather than learning general reasoning strategies. Multiple paths prevent this by forcing the model to abstract across different valid approaches.

3. Full fine-tuning, not parameter-efficient adaptation. The paper fine-tunes all parameters of GPT-J (6B) rather than using adapters or LoRA (though they compare against Hu et al., 2023's LLM-Adapter in Table 1, which does use LoRA). The full fine-tuning approach requires more compute (8 Tesla V100 GPUs with FP16 precision, using DeepSpeed ZeRO optimization for memory efficiency) but avoids the potential capacity bottleneck of low-rank adaptation for a capability as complex as multi-step reasoning. The baseline comparison in Table 1 shows that full fine-tuning outperforms the LoRA-based LLM-Adapter (which achieves 10.6% on GSM8K vs. the paper's 15.6% in the first round alone).

4. Training hyperparameters. The paper uses AdamW optimizer with $\beta = (0.9, 0.999)$, $\epsilon = 10^{-8}$, weight decay 0.01, batch size 16, 10 epochs of training, and 100 warmup steps (Table 10). The learning rate is $1 \times 10^{-6}$ for the initial round and $7 \times 10^{-7}$ for subsequent rounds, with the lower rate for later rounds intended to "make the training more stable" as the model is already partly converged and large updates could cause catastrophic forgetting. A random seed of 42 is fixed for reproducibility. These hyperparameters are applied uniformly across all five datasets without per-task tuning, suggesting the method is relatively robust to hyperparameter choices.

5. Greedy decoding at evaluation. During inference, the model generates rationales using greedy decoding (selecting the highest-probability token at each step) rather than beam search or sampling. The paper notes this is for simplicity—"though beam search may further improve the performance" (Appendix A.2). Answer extraction is similarly simple: the system "simply using the first valid token after the 'Answer:'" delimiter, avoiding complex post-processing. Both choices prioritize reproducibility and ease of implementation over squeezing out marginal performance gains.

Joint Training: Balancing Language Modeling and Contrastive Signals

The final training objective combines the two losses through a single mixing coefficient $\lambda$:

$\mathcal{L} = \mathcal{L}_{\text{lm}} + \lambda \mathcal{L}_{\text{cl}}$

where $\mathcal{L}_{\text{lm}}$ is the language modeling loss on teacher rationales (Equation 3), $\mathcal{L}_{\text{cl}}$ is the contrastive triplet loss on self-reflection (Equation 2), and $\lambda$ controls the relative weight of the contrastive term.

What it computes (operationally): during each training step, the model processes a batch of training examples in parallel. For each example, the standard cross-entropy loss is computed on the teacher's rationale tokens. Simultaneously, for a subset of examples in the batch, the contrastive loss is computed over triplet pairs sampled from $\mathcal{D}_{\text{train}}$ and $\mathcal{D}_{\text{neg}}$ for the same question. The two losses are scaled and summed, and gradients are backpropagated through the full model. The model is updated to both better imitate the teacher's correct reasoning (via $\mathcal{L}_{\text{lm}}$) and better separate correct from incorrect reasoning in representation space (via $\mathcal{L}_{\text{cl}}$).

Why this form and the choice of $\lambda$:

1. $\lambda = 0.5$ as the default across all tasks. The paper states that "without any heavy tuning, $\lambda$ in Eq. (4) is set to 0.5 to control the impact of self-reflection" (Section 4.3). This is a significant practical claim: the method works well with a fixed $\lambda$ across five diverse benchmarks (mathematical and commonsense, varying in size and difficulty), suggesting that the language modeling and contrastive objectives are naturally well-balanced at roughly equal contribution to the total gradient norm. The paper does not provide a detailed sensitivity analysis across $\lambda$ values for all datasets, but Figure 5 shows the effect on MultiArith and StrategyQA for the initial round. The key finding: self-reflection helps up to $\lambda = 0.5$ but degrades performance at $\lambda \geq 0.75$.

2. The underfitting risk at high $\lambda$. Figure 6 provides the mechanistic explanation for why large $\lambda$ hurts. The plot shows the language modeling training loss $\mathcal{L}_{\text{lm}}$ over training steps for different $\lambda$ values on MultiArith. At $\lambda = 1.0$, the language modeling loss remains substantially higher (worse) than at $\lambda = 0.0$ throughout training, indicating underfitting. The paper's interpretation: "excessive emphasis on self-reflection learning (higher $\lambda$) can result in underfitting of these training data within a limited number of training steps." In other words, the contrastive gradient dominates the optimization, slowing down the model's ability to learn the actual token-generation task from the teacher's rationales. The model learns to tell good from bad reasoning but forgets (or never learns) how to produce good reasoning.

   This is a concrete example of the more general multi-task learning tradeoff: when two objectives compete for the same model capacity and are trained simultaneously, the relative learning rates matter. The paper's implicit solution is to keep $\lambda$ modest (0.5), which empirically provides enough contrastive signal to improve representations (as shown in Figure 4 and Table 4) without drowning out the primary language modeling task.

3. The contrastive loss converges regardless of $\lambda$. The paper also reports, in the caption of Figure 6, that "the loss of Eq. (2) with different $\lambda$ can all converge." This means the contrastive signal is relatively easy to optimize—the model can learn to push correct and wrong representations apart even when it hasn't fully mastered generating correct reasoning. This is expected: the contrastive task is simpler (a binary discrimination problem in representation space) than the generation task (predicting exact token sequences). The asymmetry—contrastive loss converges quickly, generation loss converges slowly—is what makes the $\lambda$ balance so important.

4. Interaction with the generalization findings. The generalization results in Table 11 show that self-reflection can help or hurt out-of-domain performance depending on the training dataset. For example, on SVAMP, adding self-reflection improves generalization to MultiArith (from 5.1 to 9.6 for the GSM8K-trained model, last round), but on CSQA it modestly degrades generalization to StrategyQA. The paper does not analyze this interaction, but it is consistent with the underfitting hypothesis: when the contrastive signal is too strong relative to the language modeling signal, the model overfits to discriminative features of the training distribution that don't transfer, at the expense of learning general reasoning patterns.

The Multi-Round Loop: Iterating Exams, Feedback, and Training

The components described above constitute one round of learning. The full method iterates this process, with each round's trained student becoming the starting point for the next round. The paper formalizes this as Algorithm 1:

1. Initialize the student $f^0$ with pre-trained GPT-J weights. Set round counter $r \leftarrow 0$.
2. Increment round: $r \leftarrow r + 1$; set $f^r \leftarrow f^{r-1}$ (start the new round from the previous round's trained weights).
3. Exam step: run the current student $f^r$ on the training set $\mathcal{D}_{\text{train}}$ and collect mistakes $\mathcal{D}_{\text{neg}}$ using Equation 1.
4. Rationale collection step:
   • If $r \leq 1$ (first round): request rationales from the teacher for all training questions using the template $\mathcal{T}(x, \text{null}, y)$—no student feedback is included because the untrained student's outputs are too noisy to be useful.
   • If $r > 1$ (subsequent rounds): request rationales only for mistakes using the template $\mathcal{T}(x, \hat{r}, y)$, where $\hat{r}$ comes from $\mathcal{D}_{\text{neg}}$. Questions the student already answers correctly are not re-sent to the teacher (saving API costs and focusing the teacher's effort on residual errors).
5. Training step: optimize the student $f^r$ using the joint loss from Equation 4, training on both the newly collected customized rationales and (in the contrastive term) the newly collected mistakes.
6. Convergence check: if the student's error rate on the training set has stopped decreasing (reached a plateau), stop. Otherwise, return to step 2.

What the loop computes (operationally): each round reduces the student's training error by targeting specifically the questions it still gets wrong. The first round provides broad coverage—the teacher generates rationales for the entire training set, giving the student a baseline reasoning capability. Subsequent rounds are increasingly focused: the teacher only generates rationales for the subset of questions the student still fails, and those rationales are explicitly conditioned on the student's specific errors. This is the "tailored" aspect of the learning: the feedback becomes more personalized and the training data more concentrated on the student's weaknesses as rounds progress.

Why this form (design choices):

1. Focusing teacher queries on residual errors saves cost. The paper reports the # Data column in Table 5: for GSM8K, round 1 uses 15k training examples (the full dataset, with multiple rationales per question), while round 2 uses 16k (still large, because the error rate is 76.3% so most questions still need teacher feedback) and round 3 uses 13k (less, because the error rate has dropped to 66.2%). For easier tasks like SVAMP, the reduction is dramatic: round 1 uses 2k examples, round 2 uses only 0.6k (error rate dropped to 24.0%), and round 3 uses 0.3k. This means the API cost per round decreases as the student improves for easy tasks—the method naturally allocates teacher attention where it's most needed.

2. The cold-start bootstrap in round 1. The untrained student's outputs are described as "noisy generations which are unhelpful as the feedback." Examining Table 14's GSM8K example confirms this: at round 0, the student generates "The number of short students is 2/5 of the total number of students. The number of tall students is 2/5 of the total number of students"—a confused, repetitive output that reveals no specific reasoning error (the statement about tall students being 2/5 of total is factually wrong and doesn't reflect a coherent mistake). Sending this to the teacher would likely produce a generic correction unrelated to any specific misunderstanding. By skipping the exam in round 1 and generating rationales unconditionally, the system bootstraps the student to a baseline competence level where its subsequent errors become informative enough to be useful feedback.

3. Convergence and early stopping. Table 5 shows diminishing returns: for GSM8K, accuracy improves by +12.9 (round 1), +12.6 (round 2), and +2.4 (round 3). For SVAMP, gains are +27.0, +3.6, and +1.0. The paper recommends "taking early stopping in the multi-round learning if the student can nearly reach its plateau" (Section 5.3). The convergence point varies by task difficulty: GSM8K (the hardest task) benefits from 3 full rounds before plateauing; SVAMP and CSQA (easier tasks) plateau after 2 rounds. This suggests a practical deployment strategy: monitor per-round accuracy improvements and stop when gains drop below a threshold (say, 2 percentage points), avoiding unnecessary API calls and training compute.

4. GPT-4-based quality evaluation across rounds. Table 6 provides a complementary view of multi-round progress using GPT-4 as an automatic evaluator of generated rationales (scoring from 1–5 based on "accuracy and quality of the reasoning path," though the prompt is not provided in the paper). Both correct and wrong rationales show quality improvements across rounds: on GSM8K, correct rationale scores increase from 4.50 (round 1) to 4.88 (round 2), and even wrong rationale scores increase from 1.15 to 1.26. This suggests that multi-round training improves not just answer accuracy but the overall coherence and quality of the rationales the model produces, even when those rationales are ultimately incorrect. The wrong rationales becoming "better" (more coherent, better-structured) while still wrong is consistent with the model learning better reasoning patterns overall, even if it still makes occasional logical errors.

5. The 4th round on GSM8K validates the plateau hypothesis. The paper reports conducting a 4th round on GSM8K to verify the plateau: "the ER remains unsatisfactory (51.8 ER) despite a marginal improvement (+1.4 $\Delta$) in accuracy." The error rate of 51.8% after round 4 is still high, and the accuracy gain is small, confirming that the model has reached the limit of what can be transferred from the teacher at this model scale. The paper attributes this to capacity: "the student is reaching its capacity after 3 rounds of learning." This is consistent with the fundamental scaling limitation—a 6B model cannot fully internalize the reasoning patterns of a 175B model, and additional training data beyond a certain point yields diminishing returns without increasing model capacity.

6. Static demonstrations persist across rounds. The same few-shot demonstrations (Table 15) are used in every round of training and inference. This means the demonstrations serve as a stable "curriculum framework" while the actual training data (the teacher's rationales) adapts to the student's changing needs. The paper does not explore whether dynamically updating demonstrations (e.g., selecting demonstrations that are most similar to the student's current weak areas) would improve results.

Summary of Design Choices and Their Justifications

• Exam-based mistake collection over random negative sampling: ensures the contrastive negatives reflect the student's actual error patterns rather than arbitrary incorrect reasoning, making the self-reflection signal genuinely diagnostic.
• Prompt template with ground-truth hint over open-ended rationale requests: maximizes the teacher's correct-rationale yield rate (92% for GSM8K), reducing wasted API calls and ensuring training data quality.
• Student feedback in the prompt over question-only prompts: provides both diagnostic signal (showing specific errors) and negative demonstration (potentially steering the teacher away from reproducing those errors), with empirical gains of +1.7% accuracy on GSM8K (Table 3).
• Triplet loss with positive attraction over pairwise contrastive loss: encourages different correct solutions to map to similar representations, teaching the model to recognize equivalence classes of valid reasoning rather than treating each as distinct.
• Last-token hidden state representation over pooled or dedicated encoder representations: leverages the natural sequence-aggregation property of causal attention without adding parameters, keeping the method architecture-agnostic.
• $\lambda = 0.5$ with no per-task tuning over task-specific $\lambda$ optimization: prioritizes simplicity and reproducibility; the sensitivity analysis (Figure 5) shows that the method is robust within a range (0.25–0.5), and the underfitting risk at higher $\lambda$ (Figure 6) provides a clear failure mode that is easy to detect and avoid.
• Multi-round cold-start bootstrap over including student feedback from round 0: avoids contaminating the teacher's initial training data with incoherent noise from the untrained student, establishing a baseline competence before the interactive feedback loop activates.
• Focusing teacher queries on residual errors over full-dataset regeneration each round: reduces API costs as the student improves, making the method more practical and scalable.
• Full fine-tuning over parameter-efficient methods: provides the full model capacity for reasoning acquisition, justified by the performance gap over LoRA-based baselines (Table 1), though at the cost of requiring 8 V100 GPUs.
• Greedy decoding at evaluation over beam search or sampling: prioritizes simplicity and deterministic reproducibility, with the acknowledgment that beam search might further improve results at the cost of additional inference compute.

4. Key Insights and Innovations

Innovation 1: Reframing the LLM from Data Annotator to Reasoning Teacher via Bidirectional Feedback

The dominant paradigm across all prior work on distilling reasoning from LLMs (Ho et al., 2023; Fu et al., 2023b; Magister et al., 2023; Shridhar et al., 2023) treated the large model as a static rationale factory: feed it questions, collect the outputs, train a student on those outputs. This is a one-directional pipeline—LLM → student—that implicitly assumes the LLM's value lies solely in generating high-quality text that the student can imitate.

This paper makes the conceptually distinct move of treating the LLM as a teacher rather than an annotator, and the difference is not rhetorical—it demands a fundamentally different interaction structure. A teacher needs to see the student's work to diagnose specific weaknesses and provide targeted remediation. An annotator only needs the question. The paper operationalizes this reframing through the multi-round interactive loop, where the student's mistakes become the teacher's primary conditioning signal (Section 3.2, prompt template in Figure 3).

Why this is a fundamental shift rather than an incremental improvement: prior work implicitly assumed that the bottleneck in reasoning distillation was data quality—if you could just get enough high-quality rationales, any student would learn. This paper identifies a different bottleneck: relevance of the teaching signal to the student's specific deficiencies. The ablation in Table 3 provides direct causal evidence for this claim. When student feedback is removed from the prompt, the teacher produces fewer correct rationales (5,250 → 4,641 on GSM8K, a 13.1% drop in success rate) and the downstream student accuracy drops 1.7 points. More tellingly, on StrategyQA—a task requiring nuanced reasoning rather than arithmetic—the success rate nearly halves without feedback (317 → 134 correct rationales out of 328 requests), suggesting that student error information is most valuable precisely when the reasoning space is large and diverse correct solutions exist. The teacher without feedback produces plausible-sounding but incorrect answers that get filtered out, wasting API calls and providing no training signal; with feedback, the teacher is constrained by the specific error it needs to correct, making its generation more targeted and more likely to be correct.

The broader implication: this reframing implies that the quality ceiling for reasoning distillation is not determined by how many rationales you can generate but by how well the teacher's feedback addresses the student's actual gaps. This shifts the optimization problem from "generate more data" to "make the teaching signal more diagnostic," which is a qualitatively different research direction that prior work had not articulated.

Innovation 2: Self-Reflection as Representation Learning, Not Behavior Learning

The field's prior attempts at self-reflection or self-correction for language models (Huang et al., 2022; Madaan et al., 2023; Shinn et al., 2023; Pan et al., 2023) all operated at the behavioral level: the model generates a critique or identifies errors in its output, then produces a revised version. This is an emergent capability that only large models reliably possess. Small LMs cannot spontaneously self-correct—asking them to "find and fix your mistake" produces either vacuous or nonsensical output.

This paper takes an orthogonal approach. Instead of trying to make the student generate self-corrections (a behavioral skill), it makes the student learn better internal representations of reasoning quality (a representational skill). The contrastive triplet loss (Equation 2) does not teach the model to say "here's my mistake and here's how I fixed it." It teaches the model to encode correct and incorrect reasoning chains into sharply separated regions of the hidden state space, as visualized in Figure 4. The model's own wrong rationales serve as the negative contrastive samples, making the self-reflection literal: the model learns to pull away from representations that correspond to its own error patterns.

Why this is a conceptual innovation rather than just a different loss function: it sidesteps the capability gap entirely. The behavioral approach to self-reflection requires the model to already understand what good reasoning looks like in order to critique its own output—which is precisely the capability smaller LMs lack. The representational approach only requires the model to compare correct and incorrect chains after seeing them, which is an easier discriminative task that doesn't depend on generation quality. The table of results bears this out: Table 4 shows that self-reflection increases the Euclidean distance between correct and wrong rationale representations by 27.6% on GSM8K (51.00 → 65.08) and by 392.6% on StrategyQA (5.03 → 24.78), while simultaneously increasing the model's likelihood preference for correct chains. These are representation-space improvements that occur even before the model has fully mastered generating correct chains itself—the contrastive loss converges quickly across all λ values (Figure 6 caption), confirming that discrimination is learned faster than generation.

The significance beyond the specific loss function: this suggests a general principle for capability transfer in language models—when a target capability (like reasoning) is out of reach for a small model at the behavioral level, you can still transfer aspects of that capability at the representational level, which then facilitates better behavioral learning. The contrastive objective doesn't replace behavioral training (language modeling on teacher rationales); it augments it by structuring the representation space in a way that makes the behavioral task easier. This is akin to curriculum learning but operating on internal representations rather than on training data order.

Innovation 3: Empirical Demonstration That Multi-Round Interactive Distillation Compounds Gains Beyond Single-Round Transfer

The paper provides the first systematic evidence—across five benchmarks spanning mathematical and commonsense reasoning—that the benefits of interactive student-teacher distillation compound over successive rounds rather than being captured in a single transfer step. The quantitative pattern in Table 5 tells the story: on GSM8K, Round 1 adds +12.9 points, Round 2 adds +12.6 points, and Round 3 adds +2.4 points before hitting a capacity-driven plateau. On SVAMP, Round 1 adds +27.0, Round 2 adds +3.6, and Round 3 adds +1.0.

Why this is a diagnostic finding rather than just a performance result: it reveals that the knowledge transferable from a 175B teacher to a 6B student is not monolithic—it arrives in layers that correspond to different levels of reasoning difficulty, and each round unlocks a new layer that was previously inaccessible. The paper's analysis of error rates across rounds (Table 5, "ER" column) shows that each round of customized training reduces but does not eliminate the training error, and the residual errors become increasingly concentrated on harder problems. The teacher's feedback in Round 2 addresses errors the student made after already learning from Round 1's generic rationales—errors that represent more subtle reasoning failures rather than complete inability to structure a solution.

The case study in Table 14 concretizes this. In Round 1, the student fails the GSM8K problem entirely, producing a single incorrect multiplication. The teacher provides a full correct solution. In Round 2, the student makes a more specific error (confusing "2/5 of 400" with "2/5 of 90"), and the teacher's feedback directly addresses this confusion. The Round 1 teacher feedback could not have targeted this specific error because the student hadn't yet exhibited it—it was masked by the more fundamental inability to structure a solution at all. This layered remediation, where each round's training surfaces new, more specific weaknesses that the next round can address, is the mechanism underlying the compounding gains.

The contrast with one-shot distillation (the "One-Round Distillation" baseline in Table 1, which achieves 15.6% on GSM8K vs. 33.1% for the full three-round method) demonstrates that the interactive feedback loop contributes more total gain (+17.5 points) than the entire first round of static distillation (+12.9 points). In other words, the dynamic component of the method matters as much or more than the static component. This has significant implications for how the field should think about distillation: treating it as a one-time data generation problem dramatically underuses the teacher's diagnostic capacity.

The finding that the 4th round on GSM8K yields marginal gain (+1.4 Δ) despite persistent high error rate (51.8% ER) also establishes a clear capacity boundary: even with perfect, targeted teacher feedback, a 6B model cannot fully absorb the reasoning patterns needed to solve the hardest problems in GSM8K. This negative result is informative—it suggests that multi-round distillation can transfer procedural reasoning patterns (how to structure multi-step solutions) but not necessarily the representational capacity needed for complex symbolic manipulation, which may be fundamentally tied to model scale. The paper's own generalization results (Table 11) showing out-of-domain degradation for some task pairs further supports this capacity-bounded interpretation.

Innovation 4: Training on Multiple Diverse Correct Rationales Per Question Combined with Contrastive Discrimination Creates a Richer Learning Signal Than Either Alone

The paper demonstrates—without making this a formal theoretical contribution—that the combination of diverse positive examples (multiple correct rationales per question from the teacher) and explicit negative examples (the student's own wrong rationales) creates qualitatively better learning than either signal alone. This is not obvious: one might expect that simply training on many correct rationales would be sufficient, since language modeling on correct chains should implicitly teach the model to avoid incorrect patterns. Or one might expect that contrastive learning on correct/wrong pairs would be sufficient, since it teaches quality discrimination.

The paper shows both are necessary and complementary. The diverse correct rationales (up to 4 per question, generated with top-p sampling for diversity) prevent the student from overfitting to a single solution template—a known problem in distillation that Fu et al. (2023b) also recognized. The contrastive loss on wrong rationales forces the student to actively separate its representations of correct and incorrect chains, which the language modeling objective does not explicitly optimize. Evidence for complementarity comes from the ablation pattern: removing student feedback (which reduces both rationale diversity and negative sample quality) hurts accuracy (Table 3, -1.7 points on GSM8K); removing contrastive loss entirely (λ = 0 in Figure 5) also hurts accuracy compared to λ = 0.5.

Why this is a diagnostic innovation: it reveals that the representation-space structure visualized in Figure 4—clean separation of correct and wrong rationales after self-reflection training—is not an automatic byproduct of behavioral training on correct examples. The language modeling loss alone (λ = 0) does not produce this separation, even though it does improve answer accuracy. The contrastive loss explicitly constructs this separation, and the separation in turn facilitates better generation (as measured by the preference metric in Table 4 and downstream accuracy). This is an instance of a more general principle: representation learning and behavioral learning can cooperate when the representation objective is aligned with the behavioral objective. The triplet loss's "positive attraction" term (pulling different correct solutions together in representation space) ensures that the representation clusters correspond to reasoning quality rather than to surface-form similarity, making the representation structure genuinely useful for the generation task rather than being an orthogonal objective that competes for model capacity.

The broader significance: this finding suggests that future work on reasoning distillation should not treat "generating more training data" and "improving training objectives" as independent axes. The way training data is structured—specifically, the availability of multiple diverse correct solutions per question and multiple wrong solutions that reflect the student's actual errors—determines what kinds of training objectives are effective. The paper's specific combination (diverse teacher rationales + contrastive triplet loss on own mistakes) is one concrete instantiation, but the principle that diverse positives and diagnostic negatives are complementary training signals is transferable to other distillation settings beyond reasoning.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on five benchmarks spanning two reasoning categories. Mathematical reasoning: GSM8K (Cobbe et al., 2021) — 7,473 training / 1,319 test questions at primary-school math level; MultiArith (Roy and Roth, 2015) — 420 training / 180 test questions (split 70:30 by the authors) of multi-step arithmetic; SVAMP (Patel et al., 2021) — 700 training / 300 test questions (split 70:30) with structural variations over existing math problems. Commonsense reasoning: CSQA (Talmor et al., 2019) — 9,741 training / 1,221 test questions (original split) of multi-choice QA; StrategyQA (Geva et al., 2021) — 1,603 training / 687 test questions (split 70:30) requiring implicit reasoning for yes/no answers. Dataset statistics are summarized in Appendix Table 8. The choice of both mathematical and commonsense tasks tests whether the method generalizes across reasoning types that differ in structure (multi-step arithmetic vs. implicit inference).

• Base model(s). The student LM is GPT-J 6B (Wang and Komatsuzaki, 2021), a publicly available decoder-only transformer. The teacher LLM is ChatGPT (the 175B GPT-3.5 model provided by OpenAI, accessed via API). The paper states that GPT-J is chosen because it is "representative of the capabilities of many contemporary [open-source] LLMs" and its 6B scale sits well below the ~100B threshold where emergent reasoning abilities appear, making it an appropriate test case for democratization. For the feasibility study (Section 5.4), four additional smaller models are tested: T5-Large (760M, encoder-decoder), GPT-2 Large (770M, decoder-only), OPT-IML (1.3B, decoder-only), and GPT-Neo (2.7B, decoder-only). The teacher model selection is driven by "pricing and availability" considerations at the time of the research (early 2023).

• Metrics. The primary metric throughout is accuracy (%) — the fraction of test questions for which the extracted final answer matches the ground truth. For mathematical datasets, answer extraction parses the first valid token after the "Answer:" delimiter in the generated output. For CSQA, the model selects among 5 multiple-choice options (a–e). For StrategyQA, the output is yes/no. The paper reports per-round error rate (ER) on the training set (Table 5) as a progress metric during multi-round learning, defined as the fraction of training questions the student still answers incorrectly after a given round. Additionally, GPT-4 score (Table 6) is used as an automatic quality evaluation of generated rationales, scored on a 1–5 scale based on "accuracy and quality of the reasoning path" (though the evaluation prompt is not provided in the paper). Representation-space metrics in Table 4 include Euclidean distance (mean distance between correct and wrong rationale hidden states) and Preference (the likelihood ratio of correct reasoning paths to wrong ones, i.e., how much more probable the model considers correct chains).

• Baselines. The paper compares against seven baselines of increasing sophistication:

  1. Student (w/o Fine-tuning): GPT-J evaluated zero-shot or few-shot on each dataset without any fine-tuning — this is the lower bound showing what the base model can do without distillation.
  2. Student (w/ Fine-tuning): GPT-J fine-tuned to directly generate answers (without rationales) on each dataset — tests whether answer-only supervised learning suffices without chain-of-thought.
  3. Teacher LLM: ChatGPT evaluated on each dataset (presumably few-shot prompted, though the exact prompting protocol for the teacher's evaluation is not specified in detail) — establishes the upper bound of what reasoning capability exists in the 175B model.
  4. STaR (Zelikman et al., 2022): The self-taught reasoner approach where the model iteratively generates its own rationales, filters for correct answers, and retrains on those. Results marked with * are taken from the original STaR paper or reproduced by the current authors on GPT-J.
  5. LLM-Adapter (Hu et al., 2023): Uses LoRA adapters for parameter-efficient fine-tuning of GPT-J on LLM-generated CoT data. Results are taken from the original paper where available.
  6. Specializing (Fu et al., 2023b): Fine-tunes FlanT5-XXL (11B parameters) on 130k LLM-generated rationales for mathematical reasoning. Results taken from the original paper.
  7. CoT Fine-tuned (Magister et al., 2023): Fine-tunes T5-11B on CoT data generated by GPT-3 175B. Results taken from the original paper.
  8. One-Round Distillation: The authors' own ablation — a single round of the proposed method (teacher generates rationales for all training questions, student trains on them with λ = 0.5) without any multi-round iteration. This isolates the contribution of the multi-round feedback loop.

  All baselines are evaluated under the same answer extraction protocol and (where applicable) the same few-shot demonstration format.

• Generation budget / compute accounting. The primary unit of "compute" in this paper is number of API calls to the teacher LLM, not floating-point operations. The paper reports request counts per round (Table 3 and Table 5, "# Request" column) and success rates ("# Success" — requests that produced correct rationales). For GSM8K round 2, 5,701 requests are made, yielding 5,250 correct rationales (92.1% success). For student-side training, all experiments use 8 Tesla V100 GPUs with FP16 precision, with full fine-tuning (not parameter-efficient), batch size 16, 10 epochs per round, and learning rates of 1e-6 (first round) and 7e-7 (subsequent rounds). The paper does NOT conduct FLOPs-matched comparisons between teacher and student (there is no section analogous to "FLOPs-Matched Comparison" in the compute-optimal test-time scaling literature). The cost metric is fundamentally asymmetric: teacher API calls are measured by count, student training by GPU-hours, and these are not converted to a common unit. The feasibility study (Section 5.4) explores whether smaller models (760M–2.7B) can benefit from the approach using "individual affordable computation resources," but without quantitative FLOPs comparisons.

• Cross-validation / statistical protocol. The paper does not employ k-fold cross-validation or report confidence intervals. The main results in Table 1 are single-run accuracies on the standard test splits. For the multi-round analysis, the stopping criterion is based on observing when the student "reaches a plateau" (Section 5.3) — this is a manual inspection rather than a statistical test. The GPT-4 evaluation scores in Table 6 report means with standard deviations (e.g., 4.50 ± 0.18 for GSM8K round 1 correct rationales), suggesting multiple rationales were scored and aggregated, but the sample size per mean is not specified. The method's robustness claim ("without any heavy tuning, λ in Eq. (4) is set to 0.5") is supported by a parameter sweep in Figure 5 over λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0} on two datasets (MultiArith and StrategyQA) for the initial round only — this is a limited sensitivity analysis focused on a single hyperparameter. All experiments use a fixed random seed of 42 for reproducibility.

Main Quantitative Results

Overall Performance Against Baselines (Table 1)

The full method (three rounds + self-reflection, denoted as "+ Self-Reflection" in Table 1) achieves the best performance among all small-LM methods across all five benchmarks:

• GSM8K: 33.1% accuracy — surpassing One-Round Distillation (15.6%) by +17.5 points, STaR (10.7%) by +22.4 points, and the best concurrent distillation method Specializing (27.1%, using an 11B model with 130k rationales) by +6.0 points while using only a 6B model and 54k rationales.
• MultiArith: 85.4% — up from 81.5% (One-Round) and 53.9% (STaR).
• SVAMP: 55.0% — up from 47.7% (One-Round) and 26.7% (STaR).
• CSQA: 71.3% — up from 68.1% (One-Round), nearly matching the teacher LLM's 76.0% despite being ~29× smaller.
• StrategyQA: 65.9% — up from 63.8% (One-Round), close to the teacher's 68.6%.

The incremental contributions are visible in the indented rows of Table 1. Starting from One-Round Distillation as the base, adding Multi-round alone yields average gains of +5.1 points across datasets (e.g., GSM8K 15.6 → 32.0, SVAMP 47.7 → 51.3). Adding Self-Reflection on top of Multi-round yields a further average improvement, with notable gains on SVAMP (+3.7) and MultiArith (+2.3), though the marginal benefit varies — StrategyQA gains only +0.4 from self-reflection in the full multi-round setting, suggesting that the contrastive loss adds less value when the task depends more on factual knowledge recall than on multi-step reasoning structure.

The student with no fine-tuning achieves abysmal performance on mathematical tasks (2.7% GSM8K, 9.0% MultiArith, 20.7% SVAMP), confirming that GPT-J 6B lacks emergent reasoning. Fine-tuning without rationales (answer-only) helps modestly (7.2%, 18.0%, 32.3% respectively), but the gap between answer-only and CoT-based distillation shows that learning to generate intermediate reasoning steps is crucial — the One-Round Distillation already doubles or triples the answer-only performance on math tasks.

Contribution of Multi-Round Learning (Tables 3 and 5)

Table 5 tracks the student's progress through successive rounds:

• GSM8K: Error rate drops from 98.3% (untrained) → 76.3% (after Round 1) → 66.2% (Round 2) → 49.2% (Round 3). Accuracy gains are +12.9 (Round 1), +12.6 (Round 2), +2.4 (Round 3). The large Round 2 gain (+12.6) nearly matches Round 1 (+12.9), demonstrating that the interactive feedback loop provides compounding benefits rather than saturating after one round. A 4th round yields only +1.4 Δ with 51.8% ER remaining, indicating a capacity ceiling.

• SVAMP: Error rate drops from 76.0% → 24.0% (Round 1) → 16.7% (Round 2) → 17.6% (Round 3). Accuracy gains: +27.0, +3.6, +1.0. The Round 3 error rate slightly increases (16.7 → 17.6) and accuracy gain is marginal, suggesting the model plateaus by Round 2 on this easier task.

• CSQA: Error rate drops from 67.8% → 18.9% (Round 1) → 7.6% (Round 2) → 9.2% (Round 3). Accuracy gains: +31.8, +3.9, -0.6. The Round 3 degradation (-0.6) and slight error rate increase (7.6 → 9.2) indicate possible overfitting or the exhaustion of useful teacher feedback when training error is already very low.

The "# Data" column shows how the teacher's effort is focused: for GSM8K, Round 2 requests rationales for roughly 16k examples (still large because 76.3% of training questions are wrong), while for SVAMP Round 2 only needs 0.6k examples (most questions are now correct). This demonstrates the method's natural allocation of teacher attention to residual errors.

Effect of Student's Feedback on Teacher Quality (Table 3)

The ablation comparing teacher rationale generation with vs. without student feedback in the prompt template shows:

• GSM8K: With feedback — 5,701 requests, 5,250 successful rationales (92.1%), student accuracy 28.2%. Without feedback — 5,701 requests, 4,641 successful rationales (81.4%), student accuracy 26.5% (−1.7). The success rate drops by 10.7 percentage points when student feedback is removed.
• SVAMP: With feedback — 168 requests, 166 successful (98.8%), accuracy 51.3%. Without feedback — 168 requests, 140 successful (83.3%), accuracy 48.3% (−3.0).
• StrategyQA: With feedback — 328 requests, 317 successful (96.6%), accuracy 65.5%. Without feedback — 328 requests, 134 successful (40.9%), accuracy 63.9% (−1.6). The success rate more than halves, indicating that student error information is especially critical for commonsense tasks where the reasoning space is large and the teacher otherwise generates many plausible-but-wrong answers.

These results are from the 2nd round of learning, starting from the same Round 1 checkpoint, isolating the effect of feedback on the current round's data generation.

The case study in Table 2 provides qualitative evidence for why feedback helps. For the GSM8K example, the student's feedback reveals a specific error (computing "68 − 30 = 38" without the subsequent division by 2). The teacher with feedback explicitly addresses this missing step. The teacher without feedback produces a generic solution that also contains an error (computing the total as 190 instead of 196), showing that the feedback not only makes rationales more targeted but also more likely to be correct.

Effect of Self-Reflection Learning (Figure 4, Table 4, Figures 5–6)

Representation-space evidence (Figure 4, Table 4): The t-SNE visualization of GSM8K rationale hidden states shows that without self-reflection, correct (blue) and wrong (red) representations are intermingled with substantial overlap. With self-reflection, two distinct clusters emerge. Quantitatively, Table 4 shows:

• GSM8K: Euclidean distance between correct and wrong representations increases from 51.00 → 65.08 (+27.6%). Preference (likelihood ratio of correct to wrong chains) increases from 73.63 → 79.11 (+7.4%).
• StrategyQA: Distance increases from 5.03 → 24.78 (+392.6%). Preference increases from 96.54 → 98.91 (+2.5%).

The dramatically larger relative improvement in distance for StrategyQA (nearly 5×) compared to GSM8K (27.6%) is notable. The paper does not discuss why, but a plausible interpretation is that StrategyQA rationales are shorter and more diverse in structure, making the untrained model's representations more entangled initially, giving the contrastive loss more room to create separation.

The λ sensitivity analysis (Figures 5–6):

• Figure 5 sweeps λ ∈ {0.0, 0.25, 0.5, 0.75, 1.0} for the initial round on MultiArith and StrategyQA. On MultiArith: λ = 0.25 achieves best accuracy (84.9%), λ = 0.5 yields 82.0%, λ = 1.0 drops sharply to 77.2%. On StrategyQA: λ = 0.25 gives 64.9%, λ = 0.5 gives 63.6%, λ = 1.0 gives 64.1%. StrategyQA is relatively flat across λ values, while MultiArith shows a clear peak at moderate λ with degradation at high λ. The paper's choice of λ = 0.5 as the default is reasonable for MultiArith and conservative for StrategyQA, but the lack of a StrategyQA-like peak suggests that on simpler reasoning tasks, the contrastive loss provides less marginal benefit and might even be unnecessary.

• Figure 6 explains the degradation at high λ: the language modeling loss $\mathcal{L}_{\text{lm}}$ on MultiArith fails to converge within the training budget when λ ≥ 0.75. At step 1000, the loss for λ = 1.0 is approximately 0.09 vs. approximately 0.02 for λ = 0.0 — a 4.5× gap. This is the underfitting mechanism: the contrastive gradient dominates the optimization, slowing progress on the primary generation task. The contrastive loss itself converges for all λ values (reported in the Figure 6 caption), confirming that discrimination is easier to learn than generation.

Generalization Across Tasks (Table 11)

The generalization matrix shows that models trained on one dataset are evaluated on all others, revealing transfer patterns:

• GSM8K-trained model (last round): Achieves strong transfer to MultiArith (80.3%, nearly matching the MultiArith-specialized model's 83.1%) and moderate transfer to SVAMP (42.3%). However, it degrades on CSQA (30.0%, below the untrained student's 34.5%) and StrategyQA (38.3%, below 47.2%). This suggests that training on mathematical reasoning actually damages commonsense reasoning capability, likely through catastrophic forgetting of the pre-trained knowledge needed for CSQA and StrategyQA.

• MultiArith-trained model: Shows minimal transfer to GSM8K (5.0%) but strong transfer to StrategyQA (52.1% vs. untrained 47.2%), suggesting MultiArith's simpler arithmetic structure does not transfer to harder math but may improve general reasoning patterns that help on yes/no inference tasks.

• CSQA-trained model: Shows improvement on StrategyQA across rounds (48.0% → 51.1%), indicating positive transfer between commonsense tasks, but degradation on all math tasks (GSM8K drops from 2.7% to 2.3%).

This mixed transfer pattern has an important implication that the paper does not fully explore: the method produces specialists, not general reasoners. Each distilled model excels in its training domain but may lose capabilities in others. The multi-round paradigm partially mitigates this (note the CSQA-trained model improves StrategyQA from 48.0% to 51.1% over rounds), but the overall pattern is consistent with standard fine-tuning causing catastrophic forgetting.

Feasibility Study on Smaller Models (Table 7)

Testing on models from 760M to 2.7B parameters on SVAMP and StrategyQA:

• SVAMP: All models benefit substantially from distillation. T5-Large (760M) goes from 0.0% → 11.0% (distillation) → 15.3% (+ self-reflection + multi-round). GPT-Neo (2.7B) goes from 3.7% → 34.3% → 36.0%. The 2.7B model with distillation nearly matches GPT-J 6B's One-Round performance (34.3% vs. GPT-J's 47.7% at round 1), suggesting reasonable scaling with model size.

• StrategyQA: All models converge to ~62–65% after distillation, regardless of size (760M: 62.0%, 770M: 62.2%, 1.3B: 62.0%, 2.7B: 62.2%). This striking size-invariance suggests that StrategyQA performance is dominated by the teacher's rationales encoding the relevant commonsense knowledge (which the student can memorize regardless of parameter count) rather than by the student's reasoning capacity. Adding self-reflection and multi-round learning on StrategyQA produces mixed results: some configurations improve (760M: 64.8%), others degrade (770M: 62.4% — below the 62.2% one-round baseline). The paper notes that "there are no more gains for StrategyQA, as it heavily relies on the memorization of commonsense knowledge mostly acquired from the pre-training stage, rather than on complex reasoning."

• Multi-round benefits are most consistent on mathematical tasks for these smaller models. The 760M model gains +0.6 from multi-round on SVAMP, the 1.3B model gains +3.0. This pattern aligns with the main results: multi-round interaction helps most when the task requires genuine multi-step reasoning that benefits from targeted error correction, rather than factual recall.

Ablation Studies and Robustness Checks

• Student feedback in teacher prompt (Table 3, discussed above): Removing student feedback reduces teacher success rate by 10.7 pp on GSM8K and 55.7 pp on StrategyQA, with corresponding accuracy drops of 1.7 and 1.6 points. This is the central ablation supporting the "LLM as teacher, not annotator" claim.

• Multi-round vs. one-round distillation (Table 5, aggregate across all datasets): Multi-round adds an average of +5.1 accuracy points across the five benchmarks beyond one-round distillation. However, the gain is task-dependent: large for GSM8K (+16.4 points comparing 1st vs. multi-round without self-reflection), small for SVAMP (+3.6), and zero or negative for CSQA in Round 3 (-0.6). This establishes a boundary condition: multi-round interaction helps when the task is hard enough that the student makes systematic, correctable errors, but adds little or can even hurt when training error is already low.

• Self-reflection weight λ (Figures 5–6, discussed above): λ = 0.5 provides consistent benefits; λ ≥ 0.75 causes underfitting of the language modeling objective. The robustness is established on two datasets for the initial round only — the paper does not report λ sensitivity for later rounds or for all five benchmarks.

• Model scale for feasibility (Table 7): The approach works with models as small as 760M parameters, with distillation providing the bulk of the benefit and self-reflection/multi-round providing smaller incremental gains. The finding that StrategyQA performance is size-invariant (~62–65% for all models after distillation) is a non-obvious result suggesting that for certain commonsense tasks, the knowledge is in the training data, not in the model's reasoning capacity.

• GPT-4-based rationale quality evaluation (Table 6, discussed in Section 5.3): Both correct and wrong rationales improve in GPT-4-assessed quality across rounds. On GSM8K, correct rationale scores increase from 4.50 (Round 1) to 4.88 (Round 2); wrong rationale scores increase from 1.15 to 1.26. This provides weak but suggestive evidence that multi-round training improves overall reasoning coherence, not just final-answer accuracy. The lack of a detailed evaluation prompt, inter-annotator agreement, or sample sizes makes these numbers difficult to interpret rigorously.

• Demonstration weight for training (Appendix A.2): "Assigning less weights (0.1) to the fixed demonstration examples than the input sample helps the model focus on the input sample and yield better performance." This is a minor but practical finding — without this reweighting, the model overfits to reproducing the demonstration text. No ablation table is provided for this parameter.

• Negative result with 4th round on GSM8K (Section 5.3): "The ER remains unsatisfactory (51.8 ER) despite a marginal improvement (+1.4 Δ) in accuracy." This confirms a capacity ceiling for the 6B model on challenging math problems — the teacher can provide perfectly targeted feedback, but the student lacks the representational capacity to fully absorb it.

Critical Assessment

Claim 1: "The multi-round learning paradigm enables the teacher LLM to provide customized training data according to student feedback, yielding substantial gains over one-shot distillation."

This claim is supported with clear boundary conditions. The ablation in Table 3 provides the cleanest evidence: removing student feedback from the teacher's prompt in Round 2 reduces both rationale success rate and downstream accuracy. The quantitative impact is meaningful but modest on GSM8K (-1.7 accuracy points for a single round) and more substantial on StrategyQA (where the success rate halves). Table 5 demonstrates that across three rounds on GSM8K, the cumulative gain from multi-round interaction (+16.4 points for multi-round vs. one-round, excluding self-reflection) is larger than the gain from the first round of static distillation (+12.9 points). This supports the claim that interaction matters as much as initial data generation.

However, the claim's generality is limited by the single teacher-student pair (ChatGPT → GPT-J). The paper does not test whether the feedback mechanism works with different teacher models (e.g., GPT-4, Claude, open-source LLMs) or different student architectures (e.g., LLaMA, Mistral). The effectiveness of student feedback likely depends on the teacher's ability to interpret and correct specific errors — a weaker teacher might not benefit from feedback, or might even be confused by it. The paper also does not test whether the feedback benefit is monotonic (do successive rounds consistently show feedback benefits, or does the marginal value of feedback diminish?). These are not fatal omissions — the paper demonstrates the mechanism works in its tested configuration — but they limit the strength of the "democratization" framing, since the method still depends on a specific proprietary teacher.

A genuine weakness: the "One-Round Distillation" baseline already includes the full 15k training examples for GSM8K Round 1. The multi-round method adds additional training data in Rounds 2 and 3 (16k and 13k examples respectively, per Table 5). The fair comparison would control for total training data volume — does multi-round outperform simply collecting more data from the teacher in a single round? If the teacher generated 15k + 16k + 13k = 44k rationales all at once (without student feedback), would performance match the multi-round 33.1%? This ablation is not run. It is possible that the multi-round benefit comes partly from seeing more total training examples, not just from the interactive feedback. The paper's Table 1 comparison against Specializing (which used 130k rationales for an 11B model to achieve 27.1%) suggests data volume alone is not sufficient, but a direct same-model comparison controlling for total rationales would strengthen the claim considerably.

Claim 2: "Self-reflection learning motivates the student to distinguish correct rationales from wrong ones, improving reasoning performance."

This claim is supported with mechanistic evidence but has unclear boundary conditions. The t-SNE visualization (Figure 4) and quantitative metrics (Table 4) clearly show that the contrastive loss reshapes the representation space to separate correct and wrong reasoning chains. The accuracy improvements in Table 1 show that this representation change correlates with better downstream performance: +1.1 on GSM8K, +2.3 on MultiArith, +3.7 on SVAMP, +1.1 on CSQA, +0.4 on StrategyQA when self-reflection is added to the full multi-round method.

However, a non-trivial weakness is the absence of an ablation on the triplet loss structure itself. The paper does not compare the triplet loss against simpler alternatives: (a) a pairwise contrastive loss without the positive-attraction term, (b) adding wrong rationales as negative examples in the language modeling objective (trained to assign low probability), or (c) simply including both correct and marked-as-wrong rationales in the training data with different prefixes. Without these comparisons, it is unclear whether the triplet formulation specifically matters or whether any method that exposes the model to wrong examples with discriminative pressure would work. The representation-space improvements in Table 4 are clearly caused by the contrastive loss, but are those representation changes necessary for the accuracy gains, or could a simpler method achieve the same accuracy without the triplet loss?

The λ sensitivity analysis (Figures 5–6) exposes another boundary: self-reflection helps only when the contrastive gradient does not dominate the language modeling gradient. At λ = 1.0 on MultiArith, accuracy drops below the λ = 0 baseline (77.2% vs. 82.0%), making self-reflection actively harmful. The paper's default λ = 0.5 works well, but the optimal λ likely depends on dataset size, model capacity, and the quality of negative examples. The paper only sweeps λ on two datasets for the first round — a more thorough analysis on all five datasets and across rounds would be needed to claim robustness.

An important unexamined question: does self-reflection help because it improves representation separation, or does the accuracy gain come from the extra training signal (effectively more training steps with a different loss) independent of the contrastive formulation? A control experiment that adds an equivalent amount of additional language modeling training (matching the extra compute from the contrastive loss) would disambiguate these hypotheses.

Claim 3: "The approach outperforms concurrent distillation methods while using a smaller model and less training data."

This claim is supported by Table 1 comparisons but with important caveats about fairness. The paper's full method (GPT-J 6B, 33.1% GSM8K) outperforms Specializing (FlanT5-XXL 11B, 27.1%) and CoT Fine-tuned (T5-11B, 18.4%), using fewer student parameters and fewer teacher-generated rationales. This is a genuine efficiency improvement.

However, three fairness concerns arise:

1. Different base models. Specializing uses FlanT5-XXL (an encoder-decoder) while this paper uses GPT-J (decoder-only). CoT Fine-tuned uses T5-11B. These architectures have different inductive biases and pre-training data, making direct comparison of parameter counts potentially misleading. A 6B decoder-only model might have different effective capacity for reasoning than an 11B encoder-decoder.

2. Different teacher models and training data construction. Specializing uses GPT-3 (likely code-davinci-002 or similar) as the teacher, not ChatGPT. The quality of the teacher's rationales likely differs. The paper does not control for teacher quality when comparing against prior work.

3. The comparison against LLM-Adapter (Hu et al., 2023) is particularly weak as a "method" comparison. LLM-Adapter uses LoRA for parameter-efficient fine-tuning, which is a training efficiency technique, not a data generation technique. The paper's full fine-tuning outperforms it (33.1% vs. 10.6% on GSM8K), but this compares both data generation strategy AND training methodology simultaneously. A fairer comparison would be: LoRA fine-tuning on the paper's multi-round data vs. full fine-tuning on the paper's multi-round data, to isolate the effect of the training data design.

Claim 4 (implicit): "The method works across diverse reasoning tasks and model scales."

The five-benchmark evaluation (GSM8K, MultiArith, SVAMP, CSQA, StrategyQA) and the feasibility study (Table 7, 760M–2.7B models) provide reasonable breadth. However:

• All benchmarks are in English and reflect Western-centric reasoning patterns and commonsense knowledge. No multilingual or cross-cultural evaluation is attempted.
• The benchmarks are all relatively small — GSM8K has 1,319 test questions, MultiArith has 180, SVAMP has 300. On small test sets, a few lucky correct answers can swing accuracy by 1–2 percentage points, which is within the range of gains attributed to self-reflection on some tasks. The paper does not report confidence intervals or statistical significance tests.
• The feasibility study (Table 7) only tests SVAMP and StrategyQA, not the full five-benchmark suite. The paper's claim that "the reasoning abilities of these small LMs can all be enhanced" is based on two datasets only.
• The StrategyQA size-invariance finding (all models converge to ~62–65%) actually weakens the claim that the method effectively teaches reasoning for commonsense tasks — it suggests the models are primarily memorizing factual patterns from the teacher's rationales, not learning generalizable reasoning strategies. The paper acknowledges this interpretation (Section 5.4), but this should be seen as a limitation of the method's applicability to knowledge-recall-heavy tasks, not a success.

Missing Experiments That Would Strengthen the Paper

1. Data volume control: Single-round distillation with the same total number of rationales as the full multi-round method (approximately 44k for GSM8K). This would isolate whether the interactive feedback mechanism matters beyond simply having more data.
2. Triplet loss ablation: Compare the triplet formulation against (a) pairwise contrastive loss, (b) language modeling with negative examples, or (c) a simple binary classifier head trained to distinguish correct/wrong chains. This would identify which aspect of the contrastive design is necessary.
3. Teacher model ablation: Test with a weaker teacher (e.g., a 13B open-source model) to determine whether the feedback mechanism depends on the teacher's strength. If a weaker teacher cannot effectively use student feedback, the democratization framing is weakened.
4. Cross-validation or multiple seeds: Given the small test sets, reporting mean and standard deviation across 3–5 random seeds would establish whether the reported gains exceed run-to-run variance.
5. Retention of pre-existing capabilities: The generalization matrix (Table 11) reveals catastrophic forgetting on out-of-domain tasks. A more comprehensive evaluation of the distilled model on a broad capability benchmark (e.g., MMLU, BBH) would clarify how much general knowledge is sacrificed for domain-specific reasoning gains.
6. Combining search with revisions analog: While this paper's method is orthogonal to the test-time compute scaling literature, an experiment combining the distilled model with majority voting or best-of-N sampling at inference would test whether the acquired reasoning ability can be further amplified at test time, or whether the distilled model's reasoning is already near its ceiling.

Overall Assessment

The paper's central empirical contribution — that bidirectional student-teacher interaction with student-aware feedback improves reasoning distillation over static one-shot rationale generation — is convincingly demonstrated for the GPT-J + ChatGPT configuration on mathematical reasoning tasks. The effect is clearest on GSM8K, where the multi-round loop consistently reduces error and the ablation on student feedback (Table 3) isolates the causal mechanism. The representational evidence for self-reflection (Figure 4, Table 4) provides a plausible mechanistic explanation, though the lack of loss-function ablations limits the strength of the claim that the triplet formulation specifically is necessary.

The paper's broader democratization narrative is partially supported but overstated given the experimental scope. The method still requires a proprietary 175B teacher accessed via paid API — it democratizes the result (a downloadable reasoning-capable 6B model) but not the process (still depends on ChatGPT). The feasibility study with sub-3B models (Table 7) shows the approach works at smaller scales, but only on two benchmarks. The generalization results (Table 11) reveal that the distilled models are narrow specialists that lose broad capabilities, complicating the "democratization" story — a model that can solve math but has forgotten basic commonsense may not be practically useful.

The strongest empirical results are on GSM8K (33.1%, a +22.4 point gain over the untrained student), where the interactive method clearly adds value beyond static distillation. The weakest results are on StrategyQA (65.9%, only +2.1 points above one-round distillation), where the benefit of interaction and self-reflection is marginal and may not justify the additional API cost and training complexity. This task-dependent variability is honestly reported but not deeply analyzed — understanding when multi-round interaction helps and when it doesn't is as important as demonstrating that it sometimes does.

6. Limitations and Trade-offs

The Method Democratizes the Result but Not the Process — It Still Depends on a Proprietary Black-Box Teacher

The assumption or constraint. The entire training pipeline requires a large proprietary LLM (ChatGPT 175B) as the teacher, accessed via paid API. The paper explicitly positions itself around democratization: "we aim to harness the untapped reasoning potential of smaller LMs to democratize this important emergent ability" (Section 1) and emphasizes enabling "individual affordable computation resources" (Section 5.4). However, the method democratizes only the output — a downloadable 6B model — while the process of creating that model remains gatekept behind OpenAI's API pricing, availability, and terms of service. The paper acknowledges the dependency on proprietary models in its Ethics Statement, noting that "the annotated rationales in this paper are collected from close-source ChatGPT" and that OpenAI's terms of use contain "a prohibition against 'use output from the Services to develop models that compete with OpenAI'" (Ethics Statement, point 1). The authors also note that the copyright and ownership status of LLM-generated outputs remains legally ambiguous.

The consequence. This has three practical consequences that undermine the democratization framing. First, economic dependency: reproducing the method requires paying for thousands of API calls. Table 3 reports 5,701 requests for GSM8K Round 2 alone. Table 5 shows that GSM8K uses approximately 44k total training rationales across three rounds (15k + 16k + 13k). At ChatGPT's API pricing at the time of writing, this represents a non-trivial cost — and the paper never quantifies this cost or compares it to alternatives. A researcher or small lab wanting to apply the method to a new domain must budget for this expense. Second, legal and terms-of-service risk: the paper itself raises the concern that using ChatGPT outputs to train competing models may violate OpenAI's terms of use. This creates uncertainty for anyone wanting to deploy models trained with this method in commercial settings. Third, reproducibility fragility: the method's effectiveness depends on the specific teacher model's capability to interpret student errors and generate targeted corrections. If the teacher model changes (API deprecation, model update, different provider), the quality of generated rationales may shift in ways that are impossible to predict or control. The paper does not test with open-source teachers (e.g., LLaMA 70B) to establish whether the feedback mechanism works with non-proprietary alternatives.

What evidence exists in the paper. The paper explicitly acknowledges this limitation in the Ethics Statement: "it is noteworthy that many of [open-source LLMs] use the outputs from closed-source LLMs... for further improvements" and calls for "a responsible discussion about data collection." The dependence on a specific teacher is visible throughout the results — all experiments use ChatGPT, and the feasibility study (Table 7) uses the same teacher across all student model sizes. The paper does not contain an ablation with different teacher models.

Mitigation status. The paper does not attempt to mitigate this limitation. The authors suggest in Limitations point 1 that "training better foundation LMs should be the primary task for the open-source community" and that it would be "valuable to validate our findings using more powerful LMs (e.g., LLaMA)," but this validation is left to future work. No experiments use open-source teachers, and no cost analysis is provided for the API dependency.

The Difficulty Estimation Overhead Is Unaccounted For — Teacher Queries Scale with Student Error Rate

The assumption or constraint. The method's multi-round loop (Algorithm 1) generates teacher rationales for all training questions in Round 1, then only for mistakes in subsequent rounds. The paper presents this as a cost-saving feature: "Focusing teacher queries on residual errors saves cost" (Section 3.4 motivation). However, the actual number of teacher API calls is substantial and directly tied to how many errors the student makes. Table 5 reveals the scale of this overhead: GSM8K Round 2 requests rationales for approximately 16k examples (because 76.3% of the training set is still answered incorrectly after Round 1), and Round 3 requests 13k more (66.2% error rate). In total, across three rounds, GSM8K training involves roughly 44k teacher-generated rationales (15k + 16k + 13k). For an easy task like SVAMP, Round 1 uses 2k rationales, Round 2 drops to 0.6k (error rate falls to 24.0%), and Round 3 uses only 0.3k. The paper frames this decreasing cost as a feature, but the initial cost is high, and the total cost across rounds is never aggregated or compared to the performance gain.

The paper's headline accuracy numbers do not account for the fact that the "exam" step in Algorithm 1 — running the student on the entire training set to collect mistakes — also consumes inference compute. While this cost is likely small relative to training and API calls, it is not measured.

The consequence. The practical consequence is that the method's compute efficiency relative to one-shot distillation is unknown. The multi-round approach improves GSM8K accuracy from 15.6% (one-round, approximately 15k rationales) to 32.0% (three rounds, approximately 44k rationales). This is a 2.05× accuracy improvement for approximately 2.93× more teacher rationales. Without a controlled ablation that matches total data volume (e.g., one-round distillation with all 44k rationales generated in a single pass, without student feedback), it is impossible to determine how much of the multi-round gain comes from the interactive feedback mechanism versus simply having more training data. The paper acknowledges this implicitly by comparing against Specializing (Fu et al., 2023b), which used 130k rationales on an 11B model to achieve 27.1%, suggesting data volume alone is insufficient. But this comparison confounds model architecture, teacher model, and data volume — it does not isolate the feedback mechanism on the same student-teacher pair.

For practitioners, this creates an optimization problem the paper does not solve: at what error rate is it no longer worth requesting more teacher rationales? GSM8K Round 4 demonstrates diminishing returns (+1.4 accuracy for additional teacher queries on a still-large error set), but no principled stopping criterion is proposed beyond "observe performance plateau."

What evidence exists in the paper. Table 5 provides the per-round data volume and error rate, which are the raw numbers needed to compute the total teacher query cost. Table 3 shows the per-round request counts and success rates. The paper acknowledges the cost consideration in Section 5.3: "By prior estimation of the task difficulty and observing performance gains in each round, we can avoid excessive parameter tuning on the number of learning rounds and balance the reasoning performance and training costs." However, this advice is qualitative — no cost-benefit threshold is proposed.

Mitigation status. The paper provides two partial mitigations: (1) the residual-error focusing strategy (only querying the teacher for mistakes after Round 1) reduces API calls as the student improves, and (2) the early stopping recommendation based on observed plateaus. However, neither addresses the fundamental question of whether the total API cost of multi-round learning is justified relative to generating the same volume of rationales in a single round without student feedback. No cost-matched ablation is performed.

The Method Produces Narrow Specialists — Training for Reasoning on One Domain Degrades Performance on Others

The assumption or constraint. The paper evaluates its method by fine-tuning and testing on five individual benchmarks (GSM8K, MultiArith, SVAMP, CSQA, StrategyQA), reporting in-domain accuracy for each. The generalization matrix in Table 11 reveals what happens when a model trained on one benchmark is evaluated on others. The results are sobering: a model trained to the last round on GSM8K drops from the untrained student's 34.5% to 30.0% on CSQA, and from 47.2% to 38.3% on StrategyQA. A model trained on MultiArith drops from 20.7% to 5.0% on GSM8K. A model trained on CSQA drops from 2.7% to 2.3% on GSM8K.

This is catastrophic forgetting: the process of fine-tuning the student on domain-specific reasoning data erases pre-existing capabilities the model had acquired during pre-training. The paper's focus on "democratizing reasoning ability" implicitly assumes that the distilled model will be better overall — not that it will gain reasoning ability in one domain at the cost of losing general knowledge.

The consequence. For practitioners, this means the distilled models are specialists, not general-purpose reasoners. A model fine-tuned on GSM8K may solve elementary math problems better than the base GPT-J, but it performs worse than the untrained model on commonsense tasks. This undermines the practical utility of the democratized model: a user wanting a general reasoning assistant would need to either (a) accept degraded performance on non-math tasks, (b) deploy multiple domain-specific models and route queries to the appropriate one, or (c) train a single model on all five benchmarks simultaneously (not tested in the paper). Each of these options introduces additional complexity that the paper does not address.

The paper's framing of reasoning as a unified "emergent ability" that can be democratized is also challenged by this finding. If training for mathematical reasoning degrades commonsense reasoning, then reasoning ability is not a single transferable skill — it is a collection of domain-specific capabilities that may interfere with each other during fine-tuning. This aligns with broader findings in the continual learning literature (catastrophic forgetting during sequential fine-tuning) but is not discussed in those terms.

What evidence exists in the paper. Table 11 provides the full generalization matrix across all five benchmarks. The paper notes the mixed results: "the in-domain generalization performance is enhanced after the reasoning distillation, while the out-of-domain (OOD) performance is usually slightly decreased. This finding is consistent with Fu et al. (2023b) although our method is better than theirs in terms of OOD performance" (Appendix B). The paper also notes a positive exception: "the CSQA-trained model improves StrategyQA from 48.0% to 51.1% over rounds," which is attributed to both being commonsense tasks.

Mitigation status. The paper does not attempt to mitigate catastrophic forgetting. No multi-task training experiments (training on all five benchmarks jointly), no rehearsal or replay mechanisms, and no elastic weight consolidation are explored. The generalization results are reported in Appendix B rather than in the main paper, suggesting the authors view this as a secondary finding rather than a core limitation. However, the practical implication — that the method produces domain specialists — directly contradicts the framing of the paper as democratizing "the reasoning ability" (singular, general).

Answer Correctness as a Proxy for Rationale Quality Allows Flawed Reasoning Chains into Training Data

The assumption or constraint. The paper uses a simple criterion to determine whether a teacher-generated rationale is "correct" and should be included in training data: the final answer extracted from the rationale must match the ground-truth answer $y$. The paper states this explicitly: "Following most existing works, we simply judge the quality of the generated rationale by the correctness of its answer" (Section 3.1, footnote). The same criterion is used for collecting student mistakes (Equation 1: $\hat{y} \neq y$ means the rationale is wrong) and for filtering teacher-generated rationales (Section 3.2: "only those containing correct answers are retained").

This is a known limitation in the CoT distillation literature. A rationale can be logically incoherent or contain incorrect intermediate steps but still reach the correct answer through compensating errors. Conversely, a rationale can be near-perfect but contain a single arithmetic slip in the final step, making the answer wrong. The paper acknowledges this limitation directly: "Evaluating the correctness of generated rationale is mainly based on the final answer... we call attention to develop more trustworthy criteria to evaluate the quality of rationales" (Limitations, point 4).

The consequence. The practical consequence is training data contamination: some fraction of the "correct" rationales used to train the student contain flawed or misleading intermediate reasoning. The student learns to imitate these flawed patterns, potentially acquiring reasoning habits that produce correct answers for the wrong reasons. This is especially concerning for the self-reflection contrastive loss (Equation 2): the positive anchor $h_{(r,y)}^{x}$ is assumed to represent "good reasoning," but if the teacher's rationale $r$ contains logical errors, the contrastive signal is pushing the student toward a flawed representation. Similarly, some "wrong" rationales collected as negative contrastive samples may contain valuable reasoning patterns with only a minor final error; the model is trained to push away from these representations, potentially discarding useful partial knowledge.

The effect is asymmetric across tasks. On mathematical datasets like GSM8K, where answers are deterministic numbers, the answer-correctness criterion is relatively reliable — it is unlikely to reach the correct number through completely wrong reasoning (though possible). On commonsense tasks like StrategyQA, where answers are binary (yes/no) and multiple plausible reasoning paths exist, the criterion is much weaker. The teacher might generate a rationale that sounds plausible but uses faulty logic, and as long as it reaches the correct yes/no answer, it enters the training set as a "correct" example. This may explain why self-reflection provides smaller gains on StrategyQA (+0.4 in the full multi-round setting, Table 1) — if the teacher's positive examples are unreliable, the contrastive signal is noisy and the representation separation may reflect spurious patterns rather than genuine reasoning quality.

What evidence exists in the paper. The GPT-4-based quality evaluation in Table 6 attempts to partially address this by scoring rationales holistically rather than by answer correctness alone. However, the evaluation: (a) is performed only post-hoc on a subset of data, not used to filter training data; (b) uses an unspecified prompt and scoring rubric; and (c) shows that even "wrong" rationales improve in quality across rounds (1.15 → 1.26 on GSM8K), suggesting the binary correct/wrong categorization based on answer match is discarding potentially useful training signal from near-miss rationales. The paper explicitly calls for "more trustworthy criteria to evaluate the quality of rationales. Potential methods can be using GPT-4 or a process reward model for automatic evaluation" (Limitations, point 4).

Mitigation status. The paper does not implement any mitigation. The answer-correctness filter is used as-is for all training data construction. The authors note the limitation and suggest future directions (process reward models, GPT-4 evaluation), but do not explore even simple heuristics like length-based filtering, consistency-based filtering (multiple teacher rationales must agree on intermediate steps), or using the contrastive model itself to score rationale quality.

Single Teacher-Student Pair and English-Only Benchmarks Limit Generality Claims

The assumption or constraint. All main experiments use a single teacher-student combination: ChatGPT (175B) as teacher and GPT-J (6B) as student, evaluated on five English-language benchmarks. The feasibility study (Table 7) varies the student (T5-Large 760M, GPT-2 Large 770M, OPT-IML 1.3B, GPT-Neo 2.7B) but keeps the same teacher (ChatGPT) and only tests on two benchmarks (SVAMP and StrategyQA). No experiments vary the teacher model, test on non-English reasoning tasks, or evaluate on benchmarks outside the mathematical and commonsense categories tested (e.g., no logical deduction, scientific reasoning, or multi-hop QA tasks).

The paper acknowledges the teacher-student limitation: "Our experiments primarily utilize ChatGPT and GPT-J as the teacher LLM and student LM, respectively, due to the considerations of availability and costs. It is valuable to validate our findings using more powerful LMs (e.g., LLaMA)" (Limitations, point 1). The paper also implicitly acknowledges the benchmark scope by selecting two reasoning categories (mathematical and commonsense) and five datasets, but does not discuss the absence of other reasoning types.

The consequence. The generality of three key findings remains unestablished:

1. The student feedback mechanism may depend on teacher quality. The prompt template in Figure 3 asks the teacher to "correct the wrong solution by using better reasoning steps." This requires the teacher to: (a) recognize the specific error in the student's wrong rationale, (b) understand why it's wrong, and (c) produce a corrected chain that addresses that specific error. A weaker teacher (e.g., a 13B open-source model) might not reliably perform any of these steps — it might fail to identify the error, produce a generic solution that doesn't address the student's misconception, or even reproduce the same error. If the feedback mechanism only works with teachers at or above ChatGPT's capability level, the democratization claim is further weakened: you can only democratize reasoning if you already have access to a reasoning-capable model to serve as teacher.

2. The optimal λ for self-reflection may vary across teacher-student pairs. The paper's λ sensitivity analysis (Figure 5) shows that λ = 0.5 works well for GPT-J trained on ChatGPT rationales. But if the teacher's rationales are noisier (from a weaker teacher) or the student is much smaller relative to the teacher, the optimal balance between language modeling and contrastive loss may shift. The paper provides no guidance on how to set λ for new teacher-student configurations.

3. The tasks studied may not represent the full spectrum of reasoning. Mathematical reasoning involves deterministic, multi-step symbolic manipulation with unique correct answers. Commonsense reasoning involves implicit world knowledge and inference. Other reasoning types — logical deduction, causal reasoning, counterfactual reasoning, ethical reasoning, multi-hop question answering — may have different properties that affect how well the interactive feedback mechanism works. The paper does not test whether the method generalizes beyond its five benchmarks.

What evidence exists in the paper. The feasibility study (Table 7) provides the only cross-model evidence, showing that the method works with students as small as 760M parameters — but only on two benchmarks with the same teacher. The paper's limitation statement (point 1) explicitly calls for validation with LLaMA, implicitly acknowledging the single-model limitation.

Mitigation status. The paper does not mitigate these limitations. The feasibility study varies student size but not teacher model or benchmark diversity. No experiments with open-source teachers, non-English benchmarks, or additional reasoning categories are included. The paper suggests future work on more powerful teacher-student pairs but does not perform any of the suggested experiments.

The Contrastive Loss Formulation Lacks Ablation on Alternative Ways to Use Negative Examples

The assumption or constraint. The paper proposes a specific contrastive triplet loss (Equation 2) for self-reflection learning and demonstrates that it improves accuracy when combined with language modeling at λ = 0.5. However, the paper does not compare this triplet formulation against any alternative method for using the student's wrong rationales as a training signal. The ablation in Table 3 and Figure 5 compares "with self-reflection" vs. "without self-reflection" (λ = 0 vs. λ > 0), but this does not isolate whether the specific form of the contrastive loss matters.

Alternative approaches that are not tested include: (a) a simpler pairwise loss that only pushes correct and wrong representations apart without the positive-attraction term; (b) including wrong rationales as negative examples in the language modeling objective (training the model to assign low probability to wrong chains, perhaps with a label token like "[WRONG]" prepended); (c) a binary classification head trained to predict correctness from the last-token hidden state; (d) data augmentation where wrong rationales are included in the training set with instructions to generate corrections (behavioral self-reflection rather than representational); (e) margin-based ranking loss that scores correct rationales higher than wrong ones without representation-space constraints.

The paper's presentation implies that the triplet loss is self-reflection learning, but self-reflection is a broader concept — learning from one's own mistakes — that could be operationalized in many ways. The paper's own introduction frames self-reflection more broadly: "recent studies have also shown that LLMs can self-improve by reflecting on their own mistakes" and "we exploit the reasoning potential of smaller LM by eliciting it to take self-reflection on the mistakes" (Section 1). This framing suggests that any method making the student benefit from its own errors would count as self-reflection, yet only one specific loss function is tested.

The consequence. The consequence is uncertainty about which aspect of the self-reflection mechanism is responsible for the observed gains. The t-SNE visualization (Figure 4) and quantitative distance metrics (Table 4) clearly show that the triplet loss separates correct and wrong representations. But is this separation necessary for accuracy improvement, or is it a side effect? Could a simpler method — say, simply including wrong rationales in the training data with a "this is wrong" label, or using a pairwise contrastive loss — achieve similar or better accuracy with less implementation complexity and lower risk of the underfitting problem shown in Figure 6?

The underfitting problem itself (Figure 6) might be specific to the triplet formulation. The triplet loss computes gradients through three full forward passes (anchor, positive, negative), which is computationally more expensive than a pairwise loss (two forward passes) and may dominate the language modeling gradient more severely at high λ. A pairwise loss might provide the discrimination benefit with less interference with the language modeling objective, allowing for higher self-reflection weight without underfitting. The paper does not explore these tradeoffs.

For practitioners wanting to implement the method, the lack of ablation on the loss function means they must either: (a) implement the exact triplet loss as specified (which requires careful sampling of triplets with the same-question constraint), or (b) experiment with alternatives from scratch without guidance from the paper on what matters.

What evidence exists in the paper. There is no ablation on the form of the contrastive loss. The paper only varies λ (the weight of the existing triplet formulation) and compares λ = 0 vs. λ > 0. The representation-space metrics (Table 4, Figure 4) show that the triplet loss changes representations, but not whether those changes are necessary or whether another loss would produce different changes with different effects on accuracy. The paper acknowledges this limitation partially in point 3 of the Limitations section: "The training objectives or forms can be defined in various ways, such as ranking loss or verbal critic are expected to further help the smaller LMs to reflect and learn from mistakes."

Mitigation status. The paper does not mitigate this limitation. The authors note that "the core of self-reflection is learning from mistakes" and that alternative training objectives "are expected to further help," but no experiments compare loss formulations. This is an acknowledged open direction rather than a weakness the paper attempts to resolve.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing rather than a new architecture or loss function. Its primary contribution is shifting how the field thinks about knowledge distillation for reasoning: from a one-directional data generation pipeline (LLM → student) to a bidirectional teaching loop (student → LLM → student) where the teacher's output is conditioned on the student's specific errors. This is not a paradigm shift on the scale of chain-of-thought prompting or the discovery of scaling laws, but it is a diagnostic reorientation that changes what researchers optimize for when transferring capabilities from large to small models.

Before this work, the dominant assumption in reasoning distillation (Ho et al., 2023; Fu et al., 2023b; Magister et al., 2023) was that the bottleneck is data quality and quantity—generate enough high-quality rationales from a large model, and any sufficiently capable student will learn to reason. This paper identifies a different bottleneck: relevance of the teaching signal to the student's actual deficiencies. The evidence is most direct in Table 3: on StrategyQA, the teacher's success rate at generating correct rationales drops from 96.6% to 40.9% when student feedback is removed from the prompt. The teacher without feedback doesn't just generate less useful data—it generates mostly useless data (nearly 60% of rationales are wrong and get filtered out), wasting API calls. The same teacher, armed with the student's wrong answer, becomes dramatically more reliable. This means the quality ceiling for distillation is set not by how many rationales you can generate, but by how precisely you can target the student's gaps.

The paper also partially resolves a tension in the literature about whether smaller models can benefit from self-reflection. Large models like ChatGPT can improve their outputs through self-critique and revision (Madaan et al., 2023; Shinn et al., 2023), but smaller LMs cannot spontaneously self-correct—asking GPT-J to "find and fix your mistake" produces incoherent output. The paper shows that self-reflection can be operationalized not as a behavioral skill (generating corrections) but as a representational skill (separating correct and incorrect reasoning chains in hidden state space via a contrastive triplet loss). Figure 4 and Table 4 provide the mechanistic evidence: the model's internal representation of reasoning quality becomes more structured, and this structure correlates with better downstream accuracy. This suggests a general principle: when a capability is out of reach at the behavioral level, aspects of it can still be transferred at the representational level, and the improved representations facilitate better behavioral learning. This is a concrete conceptual contribution that transcends the specific triplet loss used in this paper.

A subtler implication: the results make the case that interactive distillation can compound beyond single-round transfer (+12.9 points on GSM8K in Round 1, +12.6 in Round 2, Table 5), but the gains are task-dependent and bounded by model capacity. The 4th round on GSM8K yields only +1.4 Δ, and the generalization matrix (Table 11) reveals that domain-specific distillation causes catastrophic forgetting of pre-existing capabilities. These negative results are as informative as the positive ones: they establish that multi-round interaction helps most on tasks requiring procedural reasoning (multi-step math) where errors are systematic and correctable, and helps least on tasks relying on factual recall (StrategyQA) or when the student is near its capacity ceiling. This shifts the conversation from "does multi-round distillation work?" to "under what conditions does each additional round of teacher feedback justify its cost?"—a more precise and practically useful framing.

The directions that become more attractive after this work: (1) research on diagnostic teaching signals—what information about a student's errors is most useful for a teacher, and how should that information be structured in the prompt; (2) representation-space interventions as a complement to behavioral fine-tuning for capability transfer; (3) cost-aware distillation scheduling that allocates teacher queries based on expected marginal improvement per API call. The directions that become less attractive: treating distillation as a one-shot data generation problem without an iterative feedback loop; using static, student-unaware rationales as the sole training signal for reasoning tasks where students exhibit systematic error patterns.

Follow-Up Research This Work Enables

Cost-controlled distillation: matching total teacher queries between multi-round and single-round strategies. The paper's claim that multi-round interaction improves over one-shot distillation is confounded by total data volume: the three-round GSM8K model sees roughly 44k rationales, while the one-round baseline sees approximately 15k. The critical missing experiment is a single-round distillation trained on the same total number of rationales (44k, all generated in one pass without student feedback) compared against the multi-round model. If the single-round data-matched model achieves comparable accuracy, then the interactive feedback mechanism provides no benefit beyond data volume—the multi-round gains would be fully explained by having more training examples. If the multi-round model substantially outperforms the data-matched single-round model, then the student-aware feedback mechanism is genuinely causal and the paper's central claim is validated. A strong follow-up would also vary the teacher's data generation strategy (e.g., generating 4× more rationales per question in a single round vs. spreading them across rounds with feedback) and measure accuracy as a function of total teacher API cost, producing a cost-efficiency curve analogous to the compute-optimal scaling curves in the inference-time compute literature.

Teacher model ablation: How capable must the teacher be for student feedback to provide value? The paper uses ChatGPT (175B) as the sole teacher across all experiments. The feedback mechanism requires the teacher to: (a) understand the specific logical error in the student's wrong rationale, and (b) produce a corrected chain that directly addresses that error. This is a non-trivial capability that may only exist in the strongest models. A systematic ablation would test teachers at different scales (e.g., LLaMA 7B, 13B, 70B; GPT-3 variants; Claude; open-source models) on the same student (GPT-J 6B) and same benchmarks, measuring both the teacher's rationale success rate with vs. without student feedback and the downstream student accuracy. The key question: is there a teacher capability threshold below which student feedback hurts—where the teacher cannot reliably interpret errors and produces rationales that are actually lower quality than generic one-shot rationales? The StrategyQA results (Table 3: success rate drops from 96.6% to 40.9% without feedback) suggest that on nuanced reasoning tasks, student feedback is particularly valuable, but this may only hold for teachers with strong reasoning ability. If small open-source teachers can effectively use student feedback, the democratization claim is strengthened; if only frontier models can serve as effective interactive teachers, the method remains gatekept behind proprietary APIs.

Contrastive loss formulation ablation: Which aspect of the triplet objective is necessary for the representation-structure benefit? The paper shows that a specific margin-based triplet loss (Equation 2) produces cleaner separation of correct and wrong rationales in representation space (Figure 4, Table 4) and correlates with accuracy gains. However, no ablation compares this formulation against simpler alternatives. A targeted follow-up would test: (a) a pairwise contrastive loss (pushing correct and wrong apart without positive attraction between different correct solutions), (b) a binary classification head trained on the last-token hidden state to predict correctness (discrimination without structured embedding constraints), (c) language modeling with explicit negative examples (appending wrong rationales with a "[INCORRECT]" label and training the model to assign them low probability), and (d) simple data augmentation (including both correct and wrong rationales in the language modeling training set, relying on the autoregressive objective to implicitly learn quality distinctions). The key measurement: do all discrimination methods produce similar accuracy gains and representation-space separation, or is the triplet loss's positive-attraction term (which pulls different correct solutions together) uniquely important? Figure 4 suggests the positive-attraction term matters because without it, different correct solutions might occupy distinct regions of representation space (they can be far apart as long as they're far from wrong solutions). The triplet loss explicitly prevents this by making $\cos(\text{correct}_1, \text{correct}_2)$ large, which may be important for generalization—the model learns that different valid reasoning paths are semantically equivalent rather than unrelated. Ablating this term would test whether this hypothesized benefit is real.

Multi-task and continual learning to mitigate catastrophic forgetting during distillation. Table 11 reveals that domain-specific distillation causes out-of-domain performance degradation: a GSM8K-trained model drops from 34.5% to 30.0% on CSQA and from 47.2% to 38.3% on StrategyQA. This limits the practical utility of the method—a user wanting a generally capable reasoning assistant cannot simply fine-tune on one benchmark. A natural extension is multi-task distillation: train on all five benchmarks jointly (or on a broader set of reasoning tasks) in each round, collecting mistakes and teacher feedback per-task but sharing the student model. This would test whether the representation-structure benefits of self-reflection transfer across tasks (do clearer correct/wrong representations on math help commonsense reasoning?) and whether the catastrophic forgetting is primarily a fine-tuning-order effect (which multi-task training would eliminate) or a capacity-limitation effect (which it wouldn't). Beyond simple multi-task training, continual learning techniques—elastic weight consolidation, experience replay, progressive network expansion—could be integrated into the multi-round loop to preserve pre-existing capabilities while acquiring new reasoning skills. The paper's own finding that the CSQA-trained model improves on StrategyQA across rounds (48.0% → 51.1%) hints at positive transfer between related tasks, suggesting multi-task distillation might produce gains beyond what single-task training achieves.

Rationale quality filtering beyond answer correctness: process reward models as training data filters. The paper acknowledges (Limitations, point 4) that "evaluating the correctness of generated rationale is mainly based on the final answer" and calls for "more trustworthy criteria to evaluate the quality of rationales." This is a direct bottleneck: some teacher-generated rationales that reach correct answers do so through flawed reasoning, and some that reach wrong answers contain valuable reasoning steps with a single terminal error. Both types of noise contaminate the contrastive signal in Equation 2. A concrete follow-up would train or use an existing process reward model (PRM; Lightman et al., 2023) to score individual reasoning steps in teacher-generated rationales, then filter training data based on step-level correctness rather than final-answer correctness. This would produce: (a) a cleaner set of positive examples for the language modeling objective (only rationales where all steps are rated correct), and (b) more informative negative examples for contrastive learning (wrong rationales can be categorized as "correct reasoning with terminal error" vs. "fundamentally flawed reasoning," allowing the contrastive loss to treat these differently). Training such a PRM would require step-level human annotations or Monte Carlo rollout supervision (similar to the approach in the compute-optimal test-time scaling paper), making this non-trivial but clearly tractable given existing work. The paper's own GPT-4 evaluation (Table 6) shows that even "wrong" rationales improve in quality across rounds (1.15 → 1.26 on GSM8K), suggesting that step-level quality assessment would recover training signal that the binary correct/wrong filter currently discards.

Extending the interactive feedback mechanism to code generation and other reasoning domains with execution-based verification. The paper evaluates exclusively on mathematical word problems and commonsense QA. Code generation presents a uniquely favorable testbed because correctness can be verified automatically and deterministically through unit test execution—no ground-truth answers needed, no ambiguity in answer extraction. In a code distillation setting, the "exam" step (Section 3.1) would collect student-generated programs that fail unit tests, the teacher's prompt template would include the failing code and test error messages as the student's "wrong rationale," and the teacher would generate corrected code. This eliminates the rationale-quality evaluation problem entirely: a generated solution is known to be correct if and only if it passes all tests. The multi-round loop could focus on progressively harder test cases, and the contrastive loss could operate on program representations (last-token hidden states of passing vs. failing implementations). A strong follow-up would test this on HumanEval or MBPP, comparing the interactive method against one-shot distillation from the same teacher, and measuring not just pass@1 but pass@k and the diversity of generated solutions—code generation values solution diversity more than mathematical reasoning does, and the contrastive loss's positive-attraction term might actually harm diversity by pulling different correct solutions together in representation space. This would stress-test the method's assumptions and identify boundary conditions for the representational approach to self-reflection.

Practical Applications and Downstream Use Cases

Low-cost reasoning APIs powered by small distilled models for educational technology. A math tutoring platform (e.g., Khan Academy, Photomath) needs to generate step-by-step solutions to student-submitted problems, provide targeted error feedback, and adapt to different difficulty levels. Deploying a 175B proprietary model for real-time tutoring is expensive and raises privacy concerns when handling student data. The paper's method enables training a 6B model (or even a 2.7B model, per Table 7) that achieves 33.1% on GSM8K—not competitive with ChatGPT (62.2%) for open-ended tutoring, but sufficient for generating solution templates on problems the model can solve, which can then be verified and curated by human instructors. More importantly, the contrastive self-reflection mechanism (Table 4: preference ratio of correct to wrong chains improves from 73.63 to 79.11 on GSM8K) means the distilled model has sharper internal quality discrimination—it can potentially score student-submitted solutions as "likely correct" vs. "likely incorrect" based on representation-space proximity to its own correct reasoning clusters, even if it can't solve every problem itself. A deployment would: (1) distill a domain-specific reasoning model using the multi-round method on a curriculum-aligned problem set; (2) deploy the 6B model on-device or in a low-cost cloud instance; (3) use the contrastive head's similarity scores to flag student solutions that may contain errors, routing only the hardest cases to a human tutor or a more expensive LLM. The paper's feasibility study (Table 7) shows this works with models as small as 760M parameters, making on-device deployment plausible.

Data augmentation for self-improvement loops in specialized reasoning domains. The STaR methodology (Zelikman et al., 2022) bootstraps reasoning by having a model generate its own rationales, filtering for correct answers, and retraining. The paper's results (Table 1: STaR achieves only 10.7% on GSM8K vs. 33.1% for the proposed method) show that self-training alone is insufficient for smaller models—the self-generated rationales are too low-quality to provide effective training signal. However, the paper's method can serve as a bootstrapping pre-processing step for STaR-like pipelines: (1) use the multi-round interactive method with a strong teacher to lift the student from near-zero (2.7% on GSM8K) to a moderate baseline (33.1%); (2) at this point, the student generates coherent rationales often enough that STaR-style self-training becomes viable (the model now has a non-trivial chance of generating correct chains); (3) continue iterative self-improvement without further teacher API calls. This hybrid pipeline amortizes the teacher's cost—the expensive interactive rounds provide the initial capability lift, and the cheaper self-training extends it further. The paper's finding that the 4th round on GSM8K plateaus at 51.8% ER suggests a natural transition point: when teacher-guided gains diminish (+1.4 Δ), switch to self-training to squeeze out additional improvements from the model's own improved reasoning. For a medical or legal QA domain where domain-specific training data is scarce and human annotation is expensive, this hybrid approach would reduce total teacher API costs while still leveraging the teacher's diagnostic capability where it matters most (the initial capability lift).

Open-source release of multi-domain reasoning models for resource-constrained settings. The paper's feasibility study (Table 7) demonstrates that models as small as 760M parameters can reach 15.3% on SVAMP and 64.8% on StrategyQA after distillation with self-reflection—well above the untrained baselines (0.0% and 0.0% respectively). While these accuracies are modest, they represent non-trivial reasoning capability in a model small enough to run on a consumer CPU or a mobile device. A practical deployment would train a suite of domain-specific reasoning models at the 1–3B parameter scale using the multi-round method on a diverse set of benchmarks (math, science, commonsense, logical reasoning), then release them as open-source checkpoints. Users could: (a) deploy the appropriate specialist model for their use case, (b) use the contrastive head's similarity metrics to route queries to the most appropriate model, or (c) ensemble multiple specialists via majority voting on final answers. The paper's generalization matrix (Table 11) suggests that specialists don't transfer well across domains, so the multi-model approach would be necessary rather than optional. The key practical benefit is inference cost: a 1.3B model running on a single CPU is orders of magnitude cheaper to serve than ChatGPT API calls for high-volume applications like automated grading, content moderation, or customer support reasoning tasks.