ArXiv: 2404.19733

🎯 Pitch

On reasoning tasks, standard iterative preference optimization often fails, but this work shows that adding a negative log-likelihood term to the DPO loss prevents the model from forgetting correct sequences and unlocks compounding gains over repeated iterations. Using only training-set examples, the method boosts Llama-2-70B-Chat’s zero-shot GSM8K accuracy from 55.6% to 81.6% without any external data.

1. Executive Summary

This paper introduces Iterative Reasoning Preference Optimization (Iterative RPO), an iterative training algorithm that improves chain-of-thought reasoning in LLMs by repeatedly generating candidate solutions, constructing preference pairs from correct vs. incorrect answers, and training with a modified DPO loss augmented by a negative log-likelihood (NLL) term. Applied to Llama-2-70B-Chat on GSM8K, MATH, and ARC-Challenge using only the provided training sets, the method lifts zero-shot GSM8K accuracy from 55.6% to 81.6% (and to 88.7% with majority voting over 32 samples), outperforming SFT on gold CoT examples (63.5%), standard DPO (61.8%), and the STaR method (65.2%). The NLL term proves essential—standard DPO causes chosen-sequence probabilities to decrease during training, while the combined DPO+NLL loss reverses this degradation—establishing that iterative preference optimization yields compounding reasoning gains across iterations only when the training objective actively preserves the likelihood of winning sequences.

2. Context and Motivation

The Core Problem: Making Iterative Preference Optimization Work for Reasoning

The fundamental question this paper tackles is: why do iterative preference optimization methods, which have proven highly effective for general instruction-following tasks, largely fail to improve—or even degrade—performance on reasoning tasks? This gap is both empirically puzzling and practically significant, because reasoning (particularly chain-of-thought reasoning over math problems, science questions, and logic puzzles) represents one of the most important capabilities we want from modern LLMs, yet the dominant alignment paradigm seems to hit a wall when applied to it.

The paper's framing is explicit about this contradiction. Yuan et al. (2024) showed that iterative DPO methods like Self-Rewarding LLMs yield "significant gains on general instruction following tasks" but "only modest gains on reasoning tasks" (Section 4). Similarly, SPIN (Chen et al., 2024) reports that once model generations approach human performance on reasoning benchmarks, iterative preference optimization becomes "bottlenecked." The authors cite these as motivating failures: the community has developed powerful iterative preference optimization machinery, but it doesn't transfer to the reasoning domain.

This matters because reasoning is not just another task category — it's a capability that underlies many of the most economically and scientifically valuable applications of LLMs, from mathematical problem-solving to code generation to scientific inference. If iterative preference optimization cannot improve reasoning, then the dominant approach to post-training alignment leaves a massive capability gap that can only be addressed through other means (e.g., curated datasets, supervised fine-tuning on human demonstrations, or RL-based methods that are more complex and less stable). Closing this gap would mean that a single, relatively simple training recipe could improve both instruction-following and reasoning, dramatically simplifying the post-training pipeline.

Why Existing Approaches Fail on Reasoning

The paper identifies several specific mechanisms by which prior methods fall short, and these failures are not incidental — they follow from structural properties of how reasoning tasks differ from general instruction-following.

The DPO Probability Collapse Problem

One of the paper's central empirical findings (Figures 3 and 4) is that standard DPO training causes a decrease in the log-probability of chosen (correct) sequences over the course of training. Figure 3(a) shows this clearly: when training with DPO without the NLL loss term, the solid orange curve (chosen sequence log-probabilities) trends downward as training progresses, even while the margin between chosen and rejected sequences widens. This is catastrophic for reasoning tasks because:

  • Reasoning requires faithful reproduction of correct reasoning chains. Unlike general instruction-following, where multiple surface forms can express the same intent, mathematical reasoning has stricter correctness constraints — changing any step in a chain-of-thought can break the logical derivation. If the model's probability of generating previously correct solutions decreases, it may lose the very capability that made those solutions correct in the first place.
  • The DPO objective only cares about relative preference, not absolute quality. DPO can satisfy its loss by decreasing both chosen and rejected probabilities, as long as the chosen decreases less than the rejected. For reasoning, this is perverse — it means the model can "improve" in DPO terms while actually becoming worse at producing correct reasoning.

The paper connects this to concurrent observations in the literature (Pal et al., 2024; Hong et al., 2024; Xu et al., 2024) but provides the first systematic evidence that this phenomenon is the primary blocker for preference optimization on reasoning tasks. Figure 3(b) shows an even more extreme case: when standard DPO is initialized from an SFT model already trained on chosen sequences, the chosen-sequence probability still decreases, undermining the benefits of SFT initialization that Rafailov et al. (2023, 2024) argued would help DPO.

SFT's Implicit Probability Inflation Problem

Another crucial failure mode the paper identifies concerns supervised fine-tuning (SFT), which underlies methods like STaR (Zelikman et al., 2022). When SFT trains on correct (chosen) sequences only, it increases their probability — but Figure 2(a) reveals a hidden problem: SFT also increases the probability of incorrect (rejected) sequences, nearly as much as it increases the chosen ones. The dashed green curve (rejected sequence log-probabilities under SFT) closely tracks the solid green curve (chosen sequence log-probabilities), meaning the model is not learning to distinguish between correct and incorrect reasoning — it's just becoming more confident overall.

This happens because the base model assigns non-trivial probability to both correct and incorrect solutions, and SFT's log-likelihood maximization on correct sequences has a smoothing effect that lifts neighboring (incorrect) sequences as well. For reasoning, where the difference between correct and incorrect chains often hinges on subtle logical steps, this indistinguishability is particularly damaging. The model remains perfectly capable of producing plausible-sounding but wrong reasoning — and now does so with higher confidence.

Figure 2(b) shows that SFT on gold (human-written) chain-of-thought examples partially mitigates this — the gap between chosen and rejected probabilities is slightly larger — but the fundamental issue persists: chosen-sequence probabilities "barely increase" even on gold data, and rejected probabilities still rise. This explains why SFT on gold CoT examples in Table 1 reaches only 63.5% accuracy on GSM8K, a modest gain over zero-shot (55.6%) that plateaus quickly, while Iterative RPO reaches 81.6%.

The Verification Bottleneck in Iterative Methods

The paper explicitly contrasts its approach with Self-Rewarding LLMs (Yuan et al., 2024), identifying two design decisions that make that method unsuitable for reasoning:

  1. Self-Rewarding requires an LLM-as-judge to evaluate generation quality, and "these two omitted steps are challenging for reasoning tasks because they require a language model to verify correctness, which is known to be difficult" (Section 2, citing Huang et al., 2024). Unlike instruction-following, where a model can reasonably judge whether a response is helpful or harmless, mathematical correctness cannot be reliably assessed by the same model that generated the answer — the generator's errors are correlated with its judgment errors, creating a self-reinforcing failure mode.

  2. Self-Rewarding generates new prompts at each iteration to explore the input distribution. For reasoning tasks, generating novel but well-formed math problems or science questions is itself a hard reasoning task, and poorly formed prompts compound the verification problem. By contrast, Iterative RPO fixes the prompt set to the training data, eliminating this source of error.

Methods That Ignore Negative Examples

STaR and its descendants (ReSTEM^\text{EM} by Singh et al., 2024; V-STaR by Hosseini et al., 2024) all share a common architecture: generate candidate solutions, filter to keep only correct ones, and fine-tune on those. This is fundamentally a positive-example-only approach — it learns from what correct reasoning looks like but never explicitly learns to avoid incorrect reasoning. The paper's Figure 2 provides the mechanistic explanation for why this is insufficient: without negative examples (rejected sequences) in the training objective, the model cannot learn to decrease the probability of wrong reasoning chains relative to correct ones, and the probability of both rises in tandem.

V-STaR partially addresses this by using DPO to train a separate verifier model, but the generator itself is still trained only on positive examples via SFT. This is a weaker signal than directly training the generator with preference pairs, because the verifier can only post-hoc select among the generator's outputs — it cannot improve the generator's raw proposal distribution.

Difficulty of Obtaining High-Quality Negative Examples

A subtle but important challenge the paper implicitly addresses: for reasoning tasks, what constitutes a good "rejected" example? In instruction-following, any suboptimal response serves as a negative. But in reasoning, a randomly incorrect solution may be wrong for trivial reasons (e.g., an arithmetic error in the final step after correct reasoning) or for fundamental reasons (completely wrong approach). The paper's approach of generating both chosen and rejected sequences from the same model at each iteration ensures that rejected examples are "near-misses" — solutions that the current model actually produces, with errors that are representative of the model's actual failure modes — rather than arbitrary wrong answers that provide no useful learning signal. This on-policy negative sampling is a key difference from methods that use externally sourced negative examples or fixed training sets.

The Position This Paper Takes

The paper positions Iterative RPO as occupying a specific, previously unfilled niche in the design space of reasoning improvement methods (Section 2, final paragraphs). The explicit comparison to prior work defines this position along three axes:

Versus SFT-based iterative methods (STaR, ReSTEM^\text{EM}): These methods use preference optimization only implicitly — they filter training data by correctness, then apply standard SFT. Iterative RPO instead applies preference optimization directly to the generator through the DPO+NLL loss, which the paper argues (and Figure 2 supports) is necessary to decrease the probability of incorrect reasoning chains while increasing the probability of correct ones. This is the core claimed advantage: "Preference optimization allows the use of negative examples of reasoning chains and answers, which we show improves performance."

Versus DPO-based methods that fail on reasoning (Iterative DPO, Self-Rewarding): These methods use standard DPO without the NLL term. The paper's key empirical claim is that the NLL term is not just helpful but essential — without it, DPO causes the probability collapse documented in Figures 3 and 4, and performance degrades (as seen in the MATH results where standard DPO drops accuracy from 12.5% to 12.4%, and even further to 10.5% when initialized from SFT). The NLL term directly counteracts this by maintaining an upward pressure on chosen-sequence probability.

Versus methods using augmented data (MetaMath, WizardMath): These methods improve reasoning by sourcing additional training data or curated prompt sets, which the paper explicitly notes is "orthogonal" to its contribution. Iterative RPO operates on a fixed training set — the same prompts over all iterations — and improves by iteratively refining the model's own generation and preference pairs. This is positioned as a simpler, more self-contained approach that does not require external data pipelines, while remaining compatible with data augmentation methods.

The paper's title — "Iterative Reasoning Preference Optimization" — signals its thesis: that preference optimization can work for reasoning, contrary to recent negative results, if the training objective is modified to address the specific failure modes (probability collapse, inability to depress rejected sequences) that arise uniquely in the reasoning domain. The paper does not claim to invent iterative preference optimization (it credits Self-Rewarding, Iterative DPO, SPIN) nor does it claim to invent reasoning improvement through iterative training (it credits STaR, ReSTEM^\text{EM}). Its contribution is the synthesis: showing that iterative preference optimization works for reasoning if and only if you add the NLL term, and demonstrating that this combination yields compounding improvements across multiple iterations on three distinct reasoning benchmarks.

3. Technical Approach

3.1 Reader Orientation

The paper constructs an iterative self-training pipeline where a language model repeatedly generates its own chain-of-thought solutions, automatically labels them as correct or incorrect by checking the final answer against ground-truth, pairs correct and incorrect solutions as preference data, and then trains on those pairs using a modified DPO objective that includes an NLL term to prevent the model's probability of correct solutions from collapsing. The core problem this system solves is that standard DPO and iterative SFT methods fail to improve reasoning because DPO allows the absolute probability of correct sequences to decrease (the "probability collapse" problem), while SFT inflates probabilities of incorrect sequences alongside correct ones — the DPO+NLL combination creates an objective that simultaneously widens the margin between correct and incorrect sequences in the model's output distribution while maintaining or increasing the absolute likelihood of correct reasoning chains.

3.2 Big-Picture Architecture (Diagram in Words)

The Iterative RPO system has four major components that operate in a loop across iterations $t = 0, 1, \dots, T$:

  1. Base Model $M_t$ — the current LLM (initialized as Llama-2-70B-Chat, then updated each iteration) that serves as both the generator of candidate solutions and the starting point for the next round of training. At iteration 0, this is $M_0$, the instruction-tuned base model.

  2. Chain-of-Thought & Answer Generator — takes the current model $M_t$ and a fixed set of training questions $\{x_i\}$, samples $N$ candidate solutions per question (each containing CoT reasoning $c_i^n$ and a final answer $y_i^n$), and labels each solution as correct ($r_i^n = 1$) or incorrect ($r_i^n = 0$) by checking whether the extracted final answer exactly matches the known gold answer $y_i$. This produces, for each question, a set $G_i$ of generations annotated with binary rewards.

  3. Preference Pair Constructor — splits each question's generated solutions into winning set $G_i^w$ (correct answers, $r=1$) and losing set $G_i^l$ (incorrect answers, $r=0$), then pairs them by iterating through both sets simultaneously to produce $K$ preference pairs per question $\{(c_i^{w_k}, y_i^{w_k}), (c_i^{l_k}, y_i^{l_k})\}$, forming the training dataset $D_t^{\text{pairs}}$ for the next iteration.

  4. Preference Optimizer with DPO+NLL Loss — initializes parameters $\theta$ from $M_t$, then trains on $D_t^{\text{pairs}}$ using a combined loss $\mathcal{L}_{\text{DPO+NLL}}$ that adds a negative log-likelihood term on the winning sequences to the standard DPO loss, with $M_t$ serving as the reference model in the DPO denominator. The output is the next model $M_{t+1}$.

Information flows cyclically: training questions → $M_t$ generates $N$ candidate solutions per question → answers are scored against gold labels → correct/incorrect solutions are paired into $K$ preference pairs → these pairs train $M_{t+1}$ via DPO+NLL → $M_{t+1}$ becomes the generator for the next iteration. The training question set remains fixed across all iterations — only the model and the preference pairs it generates evolve.

3.3 Roadmap for the Deep Dive

The technical breakdown proceeds in this order:

  • First, the generation and reward assignment procedure — how solutions are sampled, what temperatures are used, how correctness is judged, and why the gold solution is injected into the winning set. This is the raw material for all subsequent training.
  • Second, the preference pair construction algorithm — how winning and losing sets are formed and paired, how many pairs are produced per question, and what filtering is applied. This determines what the model actually learns from.
  • Third, the DPO+NLL loss function — the mathematical form, the role of the NLL term, the hyperparameters $\alpha$ and $\beta$, and why this specific combination addresses the probability collapse problem. This is the core algorithmic contribution.
  • Fourth, the iterative training schedule — the number of iterations, how each model initializes from the previous one, the optimization hyperparameters (learning rate, batch size, step counts, early stopping protocol), and how temperatures are adjusted across iterations.
  • Fifth, the relationship to prior methods — exactly how Iterative RPO differs from Self-Rewarding LLMs (fixed prompts, gold-label rewards, DPO+NLL), from STaR (preference optimization vs. SFT), and from standard DPO (the NLL term).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithm design paper whose core idea is that iterative preference optimization can improve chain-of-thought reasoning in LLMs if and only if the training objective includes an NLL term that prevents the probability of correct reasoning sequences from degrading during DPO training, and that this recipe yields compounding gains across iterations when the model generates its own on-policy training data.

Chain-of-Thought & Answer Generation (Step i of Each Iteration)

The generation procedure is the data engine that produces the raw material for preference optimization at each iteration $t$. It operates over a fixed set of training questions $D = \{(x_i, y_i)\}_i$ where $x_i$ is a question and $y_i$ is the ground-truth correct answer. The current model $M_t$ (initialized as Llama-2-70B-Chat at $t=0$) samples $N$ complete responses for every training question:

(cin,yin)∼Mt(xi)for all xi∈D and n∈[N](c_i^n, y_i^n) \sim M_t(x_i) \quad \text{for all } x_i \in D \text{ and } n \in [N]

where $c_i^n$ is the chain-of-thought reasoning text (intermediate steps), $y_i^n$ is the extracted final answer, and $[N]$ denotes $\{1, 2, \dots, N\}$. Sampling is done with temperature $\tau$, where $\tau = 0.8$ for iterations 1–2 and $\tau = 1.3$ for iterations 3–4 (on GSM8K; for ARC $\tau = 0.8$ on iterations 1–2 and $\tau = 1.3$ for iteration 3; for MATH $\tau = 0.8$ for iterations 1–2 and $\tau = 1.0$ for iteration 3). The higher temperatures in later iterations are chosen deliberately because, as the model becomes more accurate, it produces fewer incorrect solutions at low temperature — switching to higher temperature ensures "there is a significant number of incorrect generations in later iterations" so that preference pairs are not dominated entirely by correct solutions (which would provide no useful contrastive signal).

The generation prompt is task-specific and designed to produce an easily extractable final answer. For GSM8K, a zero-shot prompt instructs the model to "Give step by step reasoning before you answer, and when you're ready to answer, please use the format 'Final answer: …'" For MATH, a 4-shot prompt with fixed in-context examples from the training set is used, with the instruction to "wrap your answer in \boxed, and conclude using the format 'Final answer: …'" — the few-shot examples are necessary so that the model produces properly LaTeX-formatted answers that can be reliably matched to ground-truth. For ARC-Challenge, a zero-shot prompt with the multiple-choice options listed asks the model to "conclude using the format 'Final answer: (insert letter here)'".

Reward assignment. Each generated solution receives a binary reward based on exact match between the extracted final answer and the gold answer:

rin=R(yin,yi)={1if yin=yi0otherwiser_i^n = R(y_i^n, y_i) = \begin{cases} 1 & \text{if } y_i^n = y_i \\ 0 & \text{otherwise} \end{cases}

This simple binary reward — exact string match after normalization — is possible because all three tasks (GSM8K, MATH with the provided grading function, and ARC with multiple-choice letters) have unambiguous correctness criteria. The paper notes that the framework could extend to "settings with more general reward models" but does not explore this empirically. The output of this step is, for each question $x_i$, an augmented set:

Gi={cin,yin,rin}n∈[N]G_i = \{c_i^n, y_i^n, r_i^n\}_{n \in [N]}

Injecting the gold solution. A critical practical detail: "Since some problems might not have any model-generated correct solution, we include the gold human written solution $(c_i, y_i)$ in the winning set $G_i^w$ so it is not empty." This ensures that even for problems where the model gets all $N$ generations wrong, there is still at least one correct example available for pairing. The gold solution provides the correct chain-of-thought reasoning from the dataset (GSM8K and MATH include human-written CoT; for ARC there is no gold CoT, so this injection is not possible, and questions with zero correct model generations are simply excluded from training).

Concrete generation budgets per task:

  • GSM8K: $N = 30$ solutions per question, across ~7.5k training questions
  • MATH: $N = 20$ solutions per question, across ~12.5k training questions
  • ARC-Challenge: $N = 30$ solutions per question, across ~7.7k training questions (both easy and challenge sets)

Preference Pair Construction

Given the reward-annotated generations $G_i$ for each training question, the algorithm constructs a dataset of preference pairs $D_t^{\text{pairs}}$ that will train the next model $M_{t+1}$. The procedure separates correct and incorrect solutions into two disjoint sets:

Giw={cin,yin∣rin=1}G_i^w = \{c_i^n, y_i^n \mid r_i^n = 1\}

Gil={cin,yin∣rin=0}G_i^l = \{c_i^n, y_i^n \mid r_i^n = 0\}

The winning set $G_i^w$ contains all model-generated correct solutions (plus the injected gold solution if applicable), while the losing set $G_i^l$ contains all model-generated incorrect solutions.

Pairing algorithm. The paper uses a straightforward simultaneous-iteration scheme to produce $K$ preference pairs per question: iterate over $G_i^w$ and $G_i^l$ simultaneously, pairing the current element from each set, and if the iteration reaches the end of either set, restart from the first element of that set. This produces $K$ pairs of indices $\{(w_k, l_k)\}$ where each pair consists of one correct solution and one incorrect solution. The goal is to "ensure we use as much of the data as possible" — rather than sampling a subset, the circular iteration pairs every correct solution with as many incorrect solutions as $K$ requires, cycling through incorrect solutions as needed.

If $G_i^w$ is empty (no correct solutions generated and no gold CoT available), that question is excluded from training entirely for that iteration. If $G_i^l$ is empty (all generated solutions are correct), the question is also excluded because there is no contrastive signal to learn from — the model already solves it perfectly under the current sampling distribution. The resulting preference dataset for iteration $t$ is:

Dtpairs={(ciwk,yiwk),(cilk,yilk)∣for all xi∈D and k∈[K]}D_t^{\text{pairs}} = \{(c_i^{w_k}, y_i^{w_k}), (c_i^{l_k}, y_i^{l_k}) \mid \text{for all } x_i \in D \text{ and } k \in [K]\}

Concrete pair counts per task:

  • GSM8K: $K = 10$ pairs per question, yielding "around 55–60k pairs for training, per iteration" after filtering. If the number exceeds 60k, it is randomly subsampled to 60k to "not introduce another source of variability."
  • MATH: $K = 15$ pairs per question, yielding "around 75k example pairs" per iteration after filtering.
  • ARC-Challenge: $K = 20$ pairs per question, yielding approximately 20k pairs in iteration 1, decreasing to 11k in iteration 2 and 5k in iteration 3. The decrease occurs because, as the model improves, fewer incorrect solutions are available to form pairs — "the decrease in the number of examples is due to the lack of incorrect samples for a number of questions in later iterations."

Filtering. The paper applies two filtering criteria: (1) pairs where the total sequence length overflows the maximum context length are removed, and (2) questions that have no incorrect generations (empty $G_i^l$) are excluded, as noted above.

Why this pairing strategy? The design choice of pairing any correct solution with any incorrect solution (rather than, say, pairing the highest-confidence incorrect with a correct on the same reasoning approach) is deliberately simple. The paper does not attempt to match solutions by semantic similarity or edit distance — any incorrect reasoning chain serves as a contrastive example for any correct one. This simplicity is enabled by the binary nature of the reward signal (exact match with gold answer): the contrast is between "any reasoning that leads to the right answer" and "any reasoning that leads to a wrong answer," without needing to assess the quality of intermediate steps. This also avoids the difficulty of constructing meaningful continuous preference scores for reasoning, which would require a trained verifier model that the paper explicitly avoids using (in contrast to Self-Rewarding LLMs, which rely on an LLM-as-judge).

The DPO+NLL Loss Function

This is the paper's central technical contribution — the specific training objective that makes iterative preference optimization work for reasoning. The loss combines the standard DPO loss with a sequence-length-normalized negative log-likelihood (NLL) term on the winning (chosen) response, weighted by a hyperparameter $\alpha$.

For a single preference pair consisting of winning sequence $(c_i^w, y_i^w)$ and losing sequence $(c_i^l, y_i^l)$ for input $x_i$, the full loss is:

LDPO+NLL=LDPO(ciw,yiw,cil,yil∣xi)+α⋅LNLL(ciw,yiw∣xi)\mathcal{L}_{\text{DPO+NLL}} = \mathcal{L}_{\text{DPO}}(c_i^w, y_i^w, c_i^l, y_i^l \mid x_i) + \alpha \cdot \mathcal{L}_{\text{NLL}}(c_i^w, y_i^w \mid x_i)

Expanding the two terms:

LDPO=−log⁡σ(βlog⁡Mθ(ciw,yiw∣xi)Mt(ciw,yiw∣xi)−βlog⁡Mθ(cil,yil∣xi)Mt(cil,yil∣xi))\mathcal{L}_{\text{DPO}} = -\log \sigma \left( \beta \log \frac{M_\theta(c_i^w, y_i^w \mid x_i)}{M_t(c_i^w, y_i^w \mid x_i)} - \beta \log \frac{M_\theta(c_i^l, y_i^l \mid x_i)}{M_t(c_i^l, y_i^l \mid x_i)} \right)

LNLL=−log⁡Mθ(ciw,yiw∣xi)∣ciw∣+∣yiw∣\mathcal{L}_{\text{NLL}} = -\frac{\log M_\theta(c_i^w, y_i^w \mid x_i)}{|c_i^w| + |y_i^w|}

where $M_\theta$ is the model being trained (initialized from $M_t$, the previous iteration's model), $M_t$ is the reference model frozen from the previous iteration, $M(x)$ denotes the probability of sequence $x$ under model $M$, $\sigma$ is the sigmoid function, $\beta$ is the DPO temperature hyperparameter controlling how strongly the loss penalizes violations of the preference ordering, $\alpha$ is the weight balancing the DPO and NLL terms, and $|\cdot|$ denotes sequence length in tokens.

What the DPO term computes: The DPO loss is a binary cross-entropy objective that compares the ratio of the new model's probability to the reference model's probability for the winning sequence against the same ratio for the losing sequence. The difference between these two log-ratios — $\beta \log \frac{M_\theta(c^w)}{M_t(c^w)} - \beta \log \frac{M_\theta(c^l)}{M_t(c^l)}$ — measures how much more the new model favors the winning sequence over the losing sequence compared to the reference model. The sigmoid $\sigma$ squashes this difference to $[0, 1]$, and the negative log turns it into a loss that is minimized when the winning sequence's relative probability is substantially larger than the losing sequence's relative probability. Because the reference model $M_t$ appears in the denominator of both ratios, the DPO loss only cares about relative changes — it can be satisfied by decreasing $M_\theta(c^l)$ more than $M_\theta(c^w)$, or by increasing $M_\theta(c^w)$ more than $M_\theta(c^l)$, or by decreasing both but decreasing the losing probability more aggressively.

What the NLL term computes: The NLL term is the standard language modeling loss — negative log-probability of the correct sequence — normalized by the total token count $|c_i^w| + |y_i^w|$. Length normalization is important because winning sequences vary in length, and without it, longer sequences would dominate the gradient. This term directly rewards the model for assigning high probability to the correct chain-of-thought and answer, regardless of what happens to the losing sequence. It provides an absolute upward pressure on the probability of correct solutions.

Why this form — the probability collapse problem: Standard DPO (where $\alpha = 0$) can decrease the absolute probability of the chosen (correct) sequence while still satisfying the preference ordering, as long as the losing sequence's probability decreases even more. Figures 3 and 4 in the paper document exactly this phenomenon: the log-probability of chosen sequences under standard DPO "decreases over training steps" (solid orange curves trend downward). For reasoning tasks, this is catastrophic — when the model becomes less likely to generate correct solutions, its accuracy degrades even though the relative preference between correct and incorrect solutions has improved. The NLL term directly counteracts this by adding a gradient that pushes $M_\theta(c^w, y^w \mid x_i)$ higher. The combination $\mathcal{L}_{\text{DPO}} + \alpha \mathcal{L}_{\text{NLL}}$ means the model must simultaneously (a) increase the probability gap between winning and losing sequences (from DPO) and (b) maintain or increase the absolute probability of winning sequences (from NLL).

Why this form — comparison to SFT: SFT training on chosen sequences alone ($\mathcal{L}_{\text{NLL}}$ only, no DPO term) increases the probability of correct solutions but, as Figure 2 shows, it also "increases the probability of rejected sequences" nearly as much. The DPO term provides the counterbalancing force that explicitly pushes down the probability of incorrect solutions, creating a separation between correct and incorrect reasoning in the model's output distribution that pure SFT cannot achieve.

Hyperparameter $\alpha$: This controls the trade-off between preference optimization (widening the gap between chosen and rejected) and likelihood preservation (keeping chosen sequences probable). The paper tunes $\alpha$ in $\{0.25, 0.5, 1, 2\}$ when training $M_1$ (the first iteration) and "end up using 1 for all experiments in the paper." The fact that $\alpha = 1$ is optimal — giving equal weight to the DPO and length-normalized NLL terms — suggests that neither term dominates the training dynamics, and that both objectives contribute meaningfully to the final performance. With $\alpha = 0$, the loss reduces to standard DPO, which Table 1 shows yields only 61.8% on GSM8K vs. 73.1% for DPO+NLL (first iteration). With $\alpha \to \infty$ (pure NLL/SFT on chosen), the loss becomes STaR-like, which Table 1 shows reaches only 65.2%.

Hyperparameter $\beta$: This is the temperature in the DPO loss that controls how sharply the loss penalizes preference violations. A small $\beta$ allows the log-ratio difference to be small (lenient), while a large $\beta$ requires a large margin between the winning and losing probability ratios. The paper tunes $\beta$ in $\{0.05, 0.1, 0.5, 1.0\}$ and "end up using 0.1" for GSM8K. This relatively small value suggests that aggressive preference optimization (large $\beta$) is counterproductive for reasoning — it might force the model to drastically reduce losing-sequence probabilities, which could destabilize the output distribution or cause overfitting to the specific pairs in the training data.

Reference model choice: The paper uses "the previous iteration's model $M_t$ as the reference model in the denominator of the DPO term." This is a natural on-policy choice — it measures how much the new model $M_\theta$ diverges from the model that generated the training data. Using a fixed initial model as reference (e.g., always $M_0$) would make the DPO loss increasingly stringent as training progresses, because later models would accumulate probability shifts relative to the fixed reference. Using $M_t$ resets the reference at each iteration, allowing each round of training to make modest adjustments from its immediate predecessor rather than large corrections from the initial model.

Length normalization in the NLL term: The denominator $|c_i^w| + |y_i^w|$ is the total token count of the winning sequence. Without this normalization, the NLL loss would be dominated by long CoT solutions (which accumulate more negative log-probability simply because they have more tokens), creating a perverse incentive for the model to prefer shorter solutions. Normalizing by length converts the loss to a per-token average negative log-probability, which is the standard language modeling metric and ensures fair comparison across sequences of different lengths.

Practical optimization across pairs: The loss is optimized over all $K$ pairs for every training input, meaning the total loss for the dataset is the sum (or average) of $\mathcal{L}_{\text{DPO+NLL}}$ over all $(x_i, k)$ pairs. Each pair independently contributes its DPO term (comparing its specific winning and losing sequences) and its NLL term (on its specific winning sequence). The paper does not use any pair weighting or curriculum — all pairs contribute equally to the gradient.

Iterative Training Schedule and Optimization Details

The overall procedure trains a sequence of models $M_1, M_2, \dots, M_T$ where each model is initialized from its predecessor and trained on preference pairs generated by that predecessor. The experimental protocol specifies four iterations for GSM8K and three iterations for ARC-Challenge and MATH.

Model sequence:

  • $M_0$: The instruction-tuned Llama-2-70B-Chat, used only to generate the first round of training data.
  • $M_1$: Initialized from $M_0$, trained on preference pairs $D_0^{\text{pairs}}$ (generated by $M_0$) using $\mathcal{L}_{\text{DPO+NLL}}$.
  • $M_2$: Initialized from $M_1$, trained on preference pairs $D_1^{\text{pairs}}$ (generated by $M_1$) using $\mathcal{L}_{\text{DPO+NLL}}$.
  • $M_3$: Initialized from $M_2$, trained on preference pairs $D_2^{\text{pairs}}$ (generated by $M_2$) using $\mathcal{L}_{\text{DPO+NLL}}$.
  • $M_4$ (GSM8K only): Initialized from $M_3$, trained on preference pairs $D_3^{\text{pairs}}$ (generated by $M_3$) using $\mathcal{L}_{\text{DPO+NLL}}$.

Optimization hyperparameters (consistent across all iterations and tasks unless noted):

  • Optimizer: AdamW
  • Learning rate: $7 \times 10^{-7}$
  • Batch size: 16
  • Maximum training steps per iteration: 5000 for GSM8K and MATH, 4000 for ARC-Challenge
  • DPO temperature $\beta$: 0.1 (GSM8K; values for ARC and MATH are not separately reported, suggesting the same)
  • NLL weight $\alpha$: 1.0 (tuned in $\{0.25, 0.5, 1, 2\}$ on iteration 1 GSM8K and fixed thereafter)

Checkpoint selection protocol. The paper uses a two-stage validation procedure to avoid overfitting:

  1. For each iteration's training, 1000 samples are held out from the training set as a validation set. The model is trained for up to the maximum step count, and the best checkpoint is selected based on performance on this held-out set. The validation metric is not explicitly stated but is presumably accuracy on the held-out questions (measured by generating from the training checkpoint and checking against gold answers).

  2. After selecting the optimal number of steps, the model is retrained from the same initialization for exactly that many steps, but this time including the previously held-out 1000 samples in the training data. This ensures that the final model at each iteration sees all available training data while still using principled early stopping.

Temperature adjustment across iterations. The generation temperature increases in later iterations to maintain a supply of incorrect solutions. On GSM8K, $\tau = 0.8$ for iterations 1–2 and $\tau = 1.3$ for iterations 3–4. On ARC, $\tau = 0.8$ for iterations 1–2 and $\tau = 1.3$ for iteration 3. On MATH, $\tau = 0.8$ for iterations 1–2 and $\tau = 1.0$ for iteration 3. This is a pragmatic design choice: as the model's accuracy improves, lower temperatures would produce mostly correct solutions, leaving too few incorrect solutions to form meaningful preference pairs. Higher temperatures increase diversity and error rate, ensuring the preference dataset remains informative. However, this also means later iterations train on noisier data — the incorrect solutions at $\tau = 1.3$ may include errors that the model would rarely make at $\tau = 0.8$, potentially reducing the relevance of the contrastive signal.

Hardware and scale. All generation (sampling $N$ solutions per question) is done on "one node containing eight V100 GPUs (32G memory)." All training is done on "eight nodes each containing eight A100 GPUs (80G memory)" — a total of 64 A100 GPUs per training run. This substantial compute requirement reflects the cost of training 70B-parameter models with long-context sequences (CoT reasoning chains can be hundreds of tokens).

Key Design Choices and Their Justifications

Why fixed training prompts across iterations? Self-Rewarding LLMs generate new prompts at each iteration to explore the input distribution, but Iterative RPO deliberately uses the same fixed set of training questions throughout. The justification is twofold: (a) for reasoning tasks, generating novel well-formed questions is itself a hard reasoning problem that introduces errors, and (b) using gold-labeled training prompts eliminates the need for a learned reward model — correctness can be judged by simple exact match against the known answer. This simplification removes the "verification bottleneck" that the paper identifies as the key failure mode of Self-Rewarding for reasoning tasks. The trade-off is that the model sees a narrower distribution of questions, which may limit generalization — but the paper's strong test-set results (which are held-out questions from the same distribution) suggest this is not a major issue for the benchmarks studied.

Why preference pairs instead of SFT on correct solutions? The paper's ablation comparing DPO+NLL to STaR (SFT on correct solutions only, which is equivalent in spirit to their method without the DPO term) shows a clear performance gap: 73.1% vs. 65.2% on GSM8K after one iteration. The mechanistic explanation in Figure 2 — SFT increases rejected-sequence probability alongside chosen-sequence probability — underscores why preference pairs are necessary. The DPO term provides explicit negative signal: "this specific chain of reasoning leads to the wrong answer, make it less likely." SFT cannot provide this signal because it only sees correct solutions.

Why DPO+NLL instead of PPO or other RL methods? The paper positions DPO as the preference optimization backbone because it is "simpler and more efficient" than PPO — it does not require training a separate reward model, online sampling during training, or KL-penalty tuning against a reference policy. DPO converts the preference learning problem into a straightforward supervised loss on static preference pairs. The paper does not empirically compare against PPO or other RL methods, but the implicit argument is that DPO+NLL achieves the desired outcome (widened preference margin + preserved correct-sequence probability) with minimal complexity. The NLL term is the paper's key modification to make this work for reasoning — prior work had already established that iterative DPO (without NLL) works for instruction-following but not reasoning, and the paper's innovation is identifying and fixing the specific failure mode.

Why include gold solutions in the winning set? This is a pragmatic hedge against the case where $G_i^w = \emptyset$ (model produces no correct solutions for a question). Without this injection, such questions would be excluded from training, which is particularly problematic for hard questions where the model's initial performance is poor — excluding them would create a selection bias where training data skews toward questions the model already solves, potentially limiting improvement on harder problems. By injecting the gold CoT, even hard questions contribute training signal: the model sees at least one correct reasoning path paired against its own incorrect attempts. This is similar in spirit to STaR's "rationalization" step, where the model is given the correct answer and asked to generate reasoning that leads to it — but Iterative RPO uses the dataset-provided gold reasoning directly rather than having the model generate it post-hoc.

Why five iterations for GSM8K but three for ARC and MATH? The paper does not explicitly justify this choice, but the performance curves in Table 1 and Table 2 provide the implicit rationale. On GSM8K, the gain from iteration 3 to iteration 4 is only 0.5% (81.1% → 81.6%), indicating saturation. On ARC, iteration 3 reaches 86.7%, and on MATH, iteration 3 reaches 20.8%. The diminishing returns suggest that further iterations would yield negligible improvements at best, and the paper likely stopped when gains became marginal. The faster saturation on ARC (fewer iterations) may reflect the multiple-choice nature of the task — with only 3–5 possible answers, the model has less room for incremental improvement through better reasoning, since chance-level performance is already 25%. The slower saturation on MATH reflects the much harder nature of the task (baseline only 12.5%), where each iteration provides meaningful new training signal because the model still generates many incorrect solutions even at $\tau = 1.0$.

Why not use data from previous iterations? The paper explicitly notes this as a limitation (Appendix A): "When training iteration $t$ using iterative RPO, we do not make use of the collected data in previous iterations. Utilizing those data could potentially boost the performance even more." This is a deliberate simplification — each iteration trains only on freshly generated on-policy data from the immediately preceding model. Using all historical data (from $M_0, M_1, \dots, M_{t-1}$) might provide more diverse training examples and prevent catastrophic forgetting of reasoning patterns learned in earlier iterations, but it would also mix off-policy and on-policy data in ways that might destabilize the DPO loss (which assumes the reference model matches the data distribution). The paper leaves this exploration to future work.

Why normalize the NLL term by sequence length? Without normalization, the NLL loss $-\log M_\theta(c_i^w, y_i^w \mid x_i)$ accumulates over all tokens in the winning sequence. Longer CoT solutions would contribute disproportionately large gradient magnitudes, creating an implicit bias toward solutions that happen to be verbose (or, conversely, toward solutions that happen to be short, if the model learns to reduce loss by shortening its output). Normalizing by $|c_i^w| + |y_i^w|$ converts the loss to a per-token average, making the gradient contribution independent of solution length and ensuring that the model optimizes the quality of each reasoning step rather than optimizing length.

Why initialize $M_{t+1}$ from $M_t$ rather than from $M_0$? Each iteration starts from the model produced by the previous iteration, not from the original instruction-tuned model. This is what makes the process "iterative" — the model that generates training data is always the most recently improved model, creating a virtuous cycle where better data generation → better training → better data generation. The paper's ablation showing that two iterations (78.0%) outperform one iteration with twice as much data (74.8%) on GSM8K demonstrates that this iterative refinement of the data-generating model is more valuable than simply having more static data from a fixed model. The underlying mechanism: as $M_t$ improves, its incorrect solutions become "harder" negatives — they are errors that a reasonably competent model actually makes, not random mistakes, providing a more informative contrastive signal for the DPO loss.

4. Key Insights and Innovations

Innovation 1: NLL as a Necessary Stabilizer, Not an Auxiliary Regularizer — Reframing the Inference-Time Objective

The dominant assumption across iterative preference optimization methods, from Iterative DPO (Xu et al., 2023; Xiong et al., 2023) through Self-Rewarding LLMs (Yuan et al., 2024) to SPIN (Chen et al., 2024), has been that the DPO objective alone is sufficient for effective preference learning — or, in the view of Rafailov et al. (2023, 2024), that combining DPO with an SFT initialization (rather than a joint loss) resolves any stability issues. The NLL term, when included at all, has been treated as an optional regularizer to prevent forgetting or as a preliminary fine-tuning step separate from preference optimization. This paper fundamentally reframes the NLL term's role: it is not a nice-to-have stabilizer but a necessary component that prevents the DPO objective from actively degrading the model's ability to generate correct reasoning.

The evidence for this reframing is stark. Figures 3 and 4 show that standard DPO causes the log-probability of chosen (correct) sequences to decrease over training steps — not to plateau or grow slowly, but to deteriorate. The DPO loss, by design, only cares about the relative gap between chosen and rejected probabilities ($\beta \log(M_\theta(c^w)/M_t(c^w)) - \beta \log(M_\theta(c^l)/M_t(c^l))$) and can satisfy itself by lowering both, as long as the rejected probability drops faster. For general instruction-following, where multiple valid surface forms express the same intent, this absolute probability decline may be harmless or even beneficial (the model redistributes probability mass among semantically equivalent phrasings). For reasoning, where correctness depends on the logical validity of each step and small changes to a chain-of-thought can break the derivation, this decline is catastrophic: the model literally becomes less likely to produce solutions that it previously knew how to generate correctly. This is why Iterative DPO and Self-Rewarding LLMs report "only modest gains on reasoning tasks" — their objective is structurally misaligned with the requirements of the domain, not just undertuned.

The conceptual move here is recognizing that reasoning tasks impose a stricter constraint on preference optimization than instruction-following tasks: the chosen sequences must not merely be preferred relative to rejected ones, but must remain absolutely accessible under the model's output distribution. The NLL term is not a convenience; it is the mechanism that enforces this constraint. This is a genuinely novel diagnostic contribution because prior work had attributed the failure of preference optimization on reasoning to unrelated issues — insufficient data diversity, weak reward models, the inherent difficulty of the tasks — rather than identifying the probability collapse as the root cause. The paper's finding that DPO+NLL reaches 73.1% on GSM8K after one iteration versus 61.8% for standard DPO (Table 1), and that on MATH standard DPO actually degrades performance from 12.5% to 12.4% (or 10.5% when initialized from SFT, Table 2), demonstrates that this is not a marginal improvement but a qualitative difference in whether preference optimization helps or hurts.

This reframing has implications beyond reasoning: it suggests that any domain where correctness depends on the specific content of generated text (code generation, formal proofs, factual accuracy) may suffer from the same probability collapse under standard DPO, and that the NLL term — or some equivalent mechanism for preserving absolute likelihood of correct outputs — may be necessary rather than optional. The paper does not explore this generalization, but the conceptual framework it establishes makes the prediction testable.

Innovation 2: On-Policy Negative Mining as a Learning Signal — Why the Model's Own Errors Are the Best Teacher

Prior iterative methods for reasoning improvement — STaR (Zelikman et al., 2022), ReSTEM^\text{EM} (Singh et al., 2024), and V-STaR (Hosseini et al., 2024) — all share a common architecture: generate candidate solutions, filter to keep only the correct ones, and train on those. This is fundamentally a positive-example-only approach. Even V-STaR, which uses DPO, applies it only to train a separate verifier, leaving the generator trained via SFT on correct examples. The implicit assumption across these methods is that seeing more correct reasoning examples is sufficient for improvement — that the model will implicitly learn to avoid errors by being shown correct patterns, or that the verifier can compensate for the generator's weaknesses at inference time.

This paper demonstrates that this assumption is mechanistically incorrect for chain-of-thought reasoning, and the evidence comes from a diagnostic experiment that prior work never performed. Figure 2(a) shows that SFT on chosen (correct) sequences increases their probability — but it also increases the probability of rejected (incorrect) sequences nearly as much. The two curves (solid and dashed green) track closely together throughout training. The model is not learning to distinguish between correct and incorrect reasoning; it is simply becoming more confident overall — including confidence in wrong answers. The conceptual problem is that SFT optimizes $-\log M_\theta(c^w, y^w \mid x)$ with no countervailing signal on incorrect sequences, and because the base model already assigns non-trivial probability to both correct and incorrect solutions (they share similar surface forms, vocabulary, and partial reasoning patterns), maximizing the likelihood of correct ones has a smoothing effect that lifts nearby (incorrect) sequences in the output distribution. This is not just an inefficiency — it is an active failure mode where training on correct examples makes the model more likely to produce plausible-sounding-but-wrong reasoning.

The paper's contribution is identifying that negative examples — specifically, the model's own incorrect reasoning chains — provide a learning signal that no amount of positive examples can replicate. The DPO term in the DPO+NLL loss explicitly pushes down the probability of incorrect sequences that the model actually generates, creating separation between correct and incorrect reasoning in the model's output distribution. This is fundamentally different from simply training on more correct examples: it teaches the model not just "this is what good reasoning looks like" but "this specific pattern of reasoning leads to a wrong answer, make it less likely." The pairing strategy — any correct solution with any incorrect solution — is deliberately simple because the binary reward signal already captures the essential contrast: "reasoning that reached the right answer" versus "reasoning that reached a wrong answer." The model's own errors serve as the negative examples because they are the most informative counterfactuals — they represent failure modes the model actually exhibits, not arbitrary wrong answers that provide no useful gradient.

This is not an incremental refinement of positive-example-only methods; it is a categorical shift in what kind of data trains the model. The experimental comparison makes this clear: STaR (first iteration) reaches 65.2% on GSM8K, while Iterative RPO (also first iteration) reaches 73.1% using the same base model and the same training prompts (Table 1). The only structural difference is that STaR uses only correct solutions for SFT, while Iterative RPO uses both correct and incorrect solutions in a preference optimization framework. The 7.9 percentage point gap on GSM8K (and similar patterns on MATH and ARC) is substantial enough to rule out sampling noise or hyperparameter differences — it reflects a fundamentally more effective learning mechanism.

Innovation 3: Iterative On-Policy Self-Improvement Without External Data or Reward Models — A Closed-Loop Reasoning Improvement Framework

The dominant paradigms for improving LLM reasoning capability fall into two camps. The first sources additional training data — MetaMath (Yu et al., 2024) augments the training set with 395k new math questions, WizardMath (Luo et al., 2023) uses reinforcement learning from evol-instruct data, and OpenMathInstruct (Toshniwal et al., 2024) provides 1.8M instruction-tuning examples. These methods improve reasoning by expanding the data distribution. The second uses human or model-based reward signals — RLHF with human-in-the-loop (Bai et al., 2022; Touvron et al., 2023), Self-Rewarding LLMs with LLM-as-judge (Yuan et al., 2024), or expert iteration with trained verifiers (Havrilla et al., 2024). These methods improve reasoning by learning from richer feedback than binary correctness.

Iterative RPO occupies a previously unfilled niche: it achieves substantial reasoning improvements (55.6% → 81.6% on GSM8K, 12.5% → 20.8% on MATH, 77.8% → 86.7% on ARC-Challenge) using only the original training set (no additional prompts, no external corpora, no data augmentation) and only binary correctness labels that are provided with the dataset (no trained reward model, no LLM-as-judge, no human feedback during training). The system operates in a closed loop: the model generates its own training data, the gold answers in the existing dataset serve as the sole verification mechanism, and the preference pairs constructed from this self-generated data drive the next round of training. There is no external dependency beyond what is already in the benchmark's training set.

This is significant not because it outperforms methods that use additional data — MetaMath reports 82.3% on GSM8K, comparable to Iterative RPO's 81.6%, while using far more training data — but because it demonstrates that iterative self-improvement on reasoning tasks is possible without the verification bottleneck that prior work assumed was unavoidable. The paper explicitly identifies this bottleneck: "These two omitted steps are challenging for reasoning tasks because they require a language model to verify correctness, which is known to be difficult" (Section 2, citing Huang et al., 2024). Self-Rewarding LLMs need an LLM to judge generation quality at each iteration — but for reasoning, the same model that generates errors cannot reliably detect them, creating a self-reinforcing failure mode. Iterative RPO sidesteps this by exploiting the fact that reasoning benchmarks already come with gold answers in the training set, making the verification step trivial (exact string match). The iteration occurs not on the verification side but on the generation side: as the model improves, it generates better-quality incorrect solutions (harder negatives) and a higher proportion of correct solutions (more positive examples), providing an increasingly informative training signal without ever needing to judge the correctness of novel answers.

The upshot is a practical recipe: if you have a reasoning benchmark with a training set that includes ground-truth answers (which GSM8K, MATH, and ARC-Challenge all do), you can apply Iterative RPO as a drop-in training procedure without building any additional infrastructure — no data collection, no reward model training, no human annotation. This dramatically lowers the barrier to improving reasoning capabilities compared to methods that require curated data pipelines or trained verifiers. The paper acknowledges that data augmentation methods are "orthogonal" and that combining both approaches could yield further gains, but the self-contained nature of Iterative RPO is positioned as a deliberate simplification that makes iterative preference optimization for reasoning accessible and reproducible.

The iteration dynamics provide a conceptual validation of the closed-loop approach. The paper's ablation showing that two iterations (78.0%) outperform one iteration with twice as much paired data (74.8%) on GSM8K is a key result — it demonstrates that the iterative refinement of the data-generating model is more valuable than simply having more static data from a fixed model. This is because the quality of the training signal improves: as $M_t$ becomes more capable, its incorrect solutions transition from "random errors a weak model makes" to "near-miss errors a competent model makes," and these near-misses provide a sharper contrastive signal for the DPO loss. The model learns not just from being shown correct reasoning but from having its own specific, subtle error patterns corrected — a form of targeted feedback that fixed datasets cannot provide. The paper does not deeply analyze the properties of these evolving negative examples (e.g., how their distribution shifts across iterations), but the performance trajectory itself — compounding improvements that slow but do not reverse across 3–4 iterations — suggests that the self-generated training data remains informative even as the model approaches saturation.

Innovation 4: The SFT Probability Inflation Problem as a Diagnostic Concept — Explaining Why Positive-Example-Only Training Plateaus

The paper introduces — through Figure 2 and the surrounding analysis — a diagnostic concept that had not been explicitly named or systematically studied in the reasoning literature: SFT probability inflation. When supervised fine-tuning trains on correct reasoning chains only, the model's probability increases not just for those correct chains but for a surrounding neighborhood of semantically similar incorrect chains. The mechanism is intuitive: correct and incorrect solutions share vocabulary, sentence structure, partial reasoning steps, and formatting, so maximizing the likelihood of correct sequences has a smoothing effect that lifts nearby sequences in the output distribution, regardless of their correctness. The result is that SFT makes the model more confident — in both correct and incorrect reasoning — without substantially improving its ability to discriminate between them.

This concept explains a set of otherwise puzzling empirical observations that have accumulated in the reasoning literature. STaR and ReSTEM^\text{EM} report improvements from iterative SFT on filtered correct solutions, but these improvements plateau at levels well below what the training data quality would suggest is achievable. The paper's GSM8K results crystallize this: SFT on gold (human-written) chain-of-thought solutions reaches only 63.5% accuracy, a modest 14% relative improvement over the zero-shot baseline of 55.6% — despite training on perfectly correct, human-authored reasoning for 7.5k problems. Why does training on ideal examples produce such limited gains? The SFT probability inflation diagnostic provides the mechanism: the model cannot learn to avoid incorrect reasoning because it never sees incorrect reasoning during training, and the smoothing effect of maximum-likelihood estimation on correct examples prevents it from developing strong preference gradients between correct and incorrect outputs.

The diagnostic value of this concept extends beyond explaining SFT's limitations. It also explains why DPO without NLL fails on reasoning: DPO can decrease chosen-sequence probabilities because it only cares about the relative gap between chosen and rejected, and once SFT has inflated both probabilities to similarly high levels, DPO may find it easier to satisfy its objective by pushing both down (with rejected pushed down faster) rather than pushing chosen up while holding rejected down. The probability inflation from SFT creates a starting condition that makes the probability collapse under DPO more severe. This is exactly what Figure 3(b) shows: when DPO (without NLL) is initialized from an SFT model trained on chosen sequences, the chosen-sequence log-probability decreases sharply — the model is undoing the inflation, but in the wrong direction.

This concept is incremental rather than revolutionary — it formalizes an intuition that SFT smooths probability mass — but it serves a crucial scientific function by providing a unified mechanistic explanation for why multiple distinct methods (vanilla SFT, STaR, DPO without NLL) all underperform on reasoning tasks while Iterative RPO does not. It is the conceptual thread that ties the paper's experimental findings together and justifies the specific components of the DPO+NLL loss: the NLL term counteracts the collapse of chosen-sequence probabilities, while the DPO term counteracts the inflation of rejected-sequence probabilities that SFT alone would cause. The combination specifically addresses both halves of the probability inflation problem, and the ablation experiments (comparing DPO+NLL to DPO-only, SFT-on-chosen, and SFT-on-gold) demonstrate that both terms are necessary and neither is sufficient alone.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on three reasoning benchmarks: GSM8K (~7.5k training, held-out test set of grade-school math word problems; Cobbe et al., 2021), MATH (~12.5k training, 5,000 test examples of competition-level math problems; Hendrycks et al., 2021), and ARC-Challenge (7.7k training across easy and challenge sets, 1,172 test examples of multiple-choice science questions; Clark et al., 2018). Training uses only the provided training splits with no additional sourced data. For MATH, the accompanying pretraining corpus is not used. For ARC-Challenge, the supporting ARC Corpus is not used.

  • Base model(s). All experiments initialize from Llama-2-70B-Chat (Touvron et al., 2023), the instruction-tuned chat version of Llama 2 at 70 billion parameters. This model is chosen as a representative large-scale instruction-following model that exhibits non-trivial but far-from-saturated reasoning performance (e.g., 55.6% zero-shot on GSM8K), leaving substantial room for improvement. It also serves as a standard baseline that many prior reasoning-improvement methods have built upon (MetaMath, WizardMath), enabling indirect comparison.

  • Metrics. The primary metric is test accuracy — the fraction of test questions for which the model's extracted final answer exactly matches the gold answer after normalization. For GSM8K and MATH, this uses exact string match (with MATH answers normalized through the provided grading function). For ARC-Challenge (multiple-choice), the model must output the correct letter option. All main results report single-generation accuracy using greedy decoding (temperature 0), with majority voting over 32 samples (sampling with temperature 0.8) reported as a secondary metric to assess the model's best-achievable performance when test-time compute is scaled.

  • Baselines. The paper compares against five categories of baselines, all initialized from the same Llama-2-70B-Chat model and using the same training data:

    • Zero-shot CoT: The base model prompted with chain-of-thought instructions but no training.
    • SFT on gold CoT examples: Supervised fine-tuning on the dataset-provided human-written chain-of-thought solutions (for GSM8K and MATH only; ARC has no gold CoT).
    • Standard DPO: Direct Preference Optimization (Rafailov et al., 2023) using the same preference pairs $D_0^{\text{pairs}}$ as Iterative RPO's first iteration, but with $\alpha = 0$ (no NLL term). Tested both initialized from Llama-2-70B-Chat and initialized from an SFT model trained on chosen sequences.
    • STaR (Zelikman et al., 2022): One iteration of the Self-Taught Reasoning method — generate candidate solutions, filter for correct ones, and SFT on those. Also tested with twice as much training data.
    • Iterative RPO variants: The first iteration of Iterative RPO initialized from the SFT-on-chosen model (to isolate the effect of the DPO+NLL objective vs. SFT initialization), and Iterative RPO with twice as much paired data (to test whether iteration count or data quantity drives gains).

  • Generation budget / compute accounting. The paper measures per-iteration training cost in terms of (a) number of generated solutions per question ($N = 20–30$ depending on task), (b) number of training pairs constructed ($K = 10–20$), and (c) total training steps (maximum 4,000–5,000 per iteration, batch size 16). No FLOPs accounting is provided. Test-time compute for majority voting uses 32 samples per question. All generation runs on 8×V100 GPUs; all training runs on 64×A100 GPUs (80G). The paper does not compare total FLOPs across methods or account for the cost of generating preference pairs vs. the cost of SFT.

  • Cross-validation / statistical protocol. For each iteration's training, 1,000 samples are held out from the training set as a validation set. The model is trained up to the maximum step count, the best checkpoint is selected based on held-out performance, and then the model is retrained for exactly that many steps including the previously held-out 1,000 samples in the training data. This two-stage procedure ensures principled early stopping without sacrificing training data. Hyperparameters ($\alpha$, $\beta$) are tuned on the first iteration (GSM8K) using this protocol and then fixed for all subsequent iterations and tasks. No cross-validation across folds or statistical significance testing is reported.

Main Quantitative Results

Iterative RPO Performance Trajectory on GSM8K

The headline result in Table 1 shows Iterative RPO lifting zero-shot CoT performance on GSM8K from 55.6% to 81.6% after four iterations — a 47% relative improvement. The performance compounds across iterations: 73.1% (iteration 1), 78.0% (iteration 2), 81.1% (iteration 3), and 81.6% (iteration 4). The per-iteration gains decay rapidly — 17.5, 4.9, 3.1, and 0.5 percentage points respectively — indicating saturation by iteration 4. Majority voting over 32 samples further lifts iteration 3 to 88.2% and iteration 4 to 88.7%, compared to 70.7% for zero-shot CoT with majority voting.

Compared to baselines using the same base model and training data, Iterative RPO's first iteration (73.1%) substantially outperforms:

  • SFT on gold CoT examples: 63.5% (a 9.6 percentage point gap)
  • STaR (1 iteration): 65.2% (a 7.9 percentage point gap)
  • STaR with twice as much data: 66.9%
  • Standard DPO initialized from Llama-2-70B-Chat: 61.8% (an 11.3 percentage point gap)
  • Standard DPO initialized from SFT on chosen sequences: 60.3%

The SFT-on-chosen baseline (equivalent to the positive-example-only component of Iterative RPO's first iteration, without the DPO term) is not directly reported in Table 1 as a standalone number, but Figure 2 analysis implies its performance would be similar to STaR's 65.2%. The gap between SFT-on-chosen and Iterative RPO iteration 1 (65.2% vs. 73.1%) quantitatively isolates the contribution of the DPO term beyond what SFT alone achieves.

The comparison of one iteration with twice as much data (74.8%) versus two iterations with the standard data quantity (78.0%) demonstrates that iterative refinement of the data-generating model is more valuable than simply having more static data. The 3.2 percentage point gap (78.0% − 74.8%) indicates that on-policy data from an improved model provides more informative training signal than additional off-policy data from a weaker model.

ARC-Challenge Results

Table 2 reports Iterative RPO's performance on ARC-Challenge across three iterations: 84.8% (iteration 1), 86.2% (iteration 2), 86.7% (iteration 3), with majority voting reaching 87.9%. The gains are more compressed than on GSM8K — only 1.9 percentage points total across three iterations versus 8.5 percentage points on GSM8K — reflecting the multiple-choice structure (25% chance baseline) and the smaller room for improvement above the already-strong zero-shot baseline of 77.8%.

Compared to baselines:

  • Zero-shot CoT: 77.8% (an 8.9 percentage point gap to iteration 3)
  • SFT on chosen sequences: 79.8% (a 6.9 percentage point gap)
  • Standard DPO initialized from Llama-2-70B-Chat: 82.8% (a 3.9 percentage point gap)
  • Standard DPO initialized from SFT on chosen: 83.5% (a 3.2 percentage point gap)

Notably, standard DPO performs substantially better on ARC (82.8%) than on GSM8K (61.8%), narrowing the gap to Iterative RPO. This may be because the multiple-choice format makes the DPO probability collapse less damaging — even if chosen-sequence probabilities decrease, the model only needs to select among 3–5 options rather than generate a free-form answer, so the collapse of absolute probability matters less for final accuracy. Conversely, the SFT-on-chosen baseline (79.8%) actually underperforms standard DPO on ARC, unlike on GSM8K where the opposite holds, suggesting the probability inflation problem identified in Figure 2 is less severe for multiple-choice tasks (where incorrect answers cover a smaller, more structured output space).

The paper notes a robustness finding specific to ARC: because correct answers can be arrived at by luck (random guessing is correct 25% of the time), "the generated data in step (i) of Iterative RPO may provide a CoT and a final answer that is correct by luck," introducing noise into the preference pairs. The fact that Iterative RPO still improves across iterations despite this noise demonstrates a degree of robustness to incorrectly-labeled training data that the paper does not further analyze.

MATH Results

Table 2 shows Iterative RPO improving MATH accuracy from the 4-shot CoT baseline of 12.5% to 17.7% (iteration 1), 19.9% (iteration 2), and 20.8% (iteration 3), with majority voting reaching 29.1%. The absolute gains are modest (8.3 percentage points total) but represent a 66% relative improvement over the baseline. The incremental gains across iterations (5.2, 2.2, 0.9 percentage points) show significant but rapidly diminishing returns.

The MATH results expose a critical failure of standard DPO: it achieves only 12.4% when initialized from Llama-2-70B-Chat — marginally worse than the 12.5% zero-shot baseline — and drops further to 10.5% when initialized from the SFT-on-chosen model. This is the clearest evidence that standard DPO's probability collapse (documented in Figure 4b) actively degrades reasoning performance on difficult tasks. The SFT-on-chosen baseline (16.8%) outperforms standard DPO by a wide margin but still falls 0.9 percentage points below Iterative RPO's first iteration (17.7%), again demonstrating the value of the DPO term beyond SFT alone.

The stronger performance of majority voting on MATH (29.1% vs. 20.8% single-sample) compared to GSM8K (88.7% vs. 81.6%) reflects the higher difficulty of MATH — the model's correct solutions are less consistent, so aggregating over 32 samples provides a larger relative boost. The paper does not report majority voting for other baselines on MATH, making it difficult to assess whether Iterative RPO's majority-vote gain is larger than what a simpler method would achieve.

Cross-Task Comparison of Iteration Dynamics

Across all three tasks, the paper observes a consistent pattern: diminishing returns across iterations with eventual saturation. On GSM8K, gains decrease from 17.5 → 4.9 → 3.1 → 0.5 percentage points. On MATH, from 5.2 → 2.2 → 0.9. On ARC, from 7.0 → 1.4 → 0.5 (computed from Table 2). The absolute magnitude of improvement correlates with task difficulty as measured by baseline performance: GSM8K (55.6% baseline) shows the largest gains, MATH (12.5% baseline) shows the smallest absolute gains but largest relative gains, and ARC (77.8% baseline) shows the most rapid saturation.

The paper attributes saturation to two factors: (1) the "upper limit on learning across iterations" imposed by the fixed set of training prompts — the model eventually runs out of new reasoning patterns to learn from the same questions, and (2) the decreasing availability of incorrect solutions for preference pair construction as the model improves — on ARC, the number of training pairs drops from ~20k to ~11k to ~5k across three iterations because "there is a lack of incorrect samples for a number of questions in later iterations." This progressive reduction in training data size may itself contribute to the diminishing returns, independently of the fixed-prompt limitation.

Ablation Studies and Robustness Checks

NLL term ablation (DPO+NLL vs. standard DPO): This is the paper's central ablation, appearing across all three tasks. On GSM8K (Table 1), DPO+NLL (iteration 1) achieves 73.1% vs. 61.8% for standard DPO initialized from Llama-2-70B-Chat, and 73.1% vs. 60.3% for standard DPO initialized from SFT on chosen sequences. On MATH (Table 2), the gap is even starker: 17.7% vs. 12.4% (standard DPO from Llama) and 17.7% vs. 10.5% (standard DPO from SFT on chosen), with standard DPO actually degrading performance below the zero-shot baseline. On ARC (Table 2), the gap is narrower: 84.8% vs. 82.8% (standard DPO from Llama) and 84.8% vs. 83.5% (standard DPO from SFT on chosen). The consistent direction and magnitude of the NLL term's benefit across tasks — from +1.3 percentage points on ARC to +11.3 on GSM8K to +7.2 on MATH (vs. the worse standard DPO variant) — supports the claim that the NLL term is necessary, but the varying magnitude suggests the probability collapse is task-dependent.

Log-probability trajectory analysis (Figures 3 and 4): The paper traces chosen and rejected sequence log-probabilities over training steps to diagnose the NLL term's mechanism. Figure 3(a) (GSM8K, initialized from Llama) shows that under standard DPO, chosen-sequence log-probability decreases over training (solid orange curve trends downward), while under DPO+NLL, it increases (solid blue curve trends upward). In both conditions, the margin between chosen and rejected probabilities widens, confirming that DPO is functioning as designed — what differs is the direction of chosen-sequence probability change. Figure 3(b) (GSM8K, initialized from SFT on chosen) shows an even more dramatic version: chosen-sequence probability under standard DPO drops sharply from its elevated SFT starting point, while DPO+NLL maintains and slightly increases it. Figure 4(a) (ARC) and Figure 4(b) (MATH) replicate the qualitative pattern: chosen-sequence log-probabilities "barely increase over training steps when training with DPO" but "increase over training steps" with DPO+NLL.

SFT probability inflation diagnostic (Figure 2): The paper directly measures what happens when training on chosen sequences via SFT, without the DPO term. Figure 2(a) shows SFT trained on chosen sequences (from $D_0^{\text{pairs}}$): both chosen (solid green) and rejected (dotted green) sequence log-probabilities increase over training and remain close together, confirming that SFT "increases the probability of rejected sequences" nearly as much as chosen ones. Figure 2(b) repeats this with SFT on gold (dataset-provided) CoTs: the gap between chosen and rejected is slightly wider than in Figure 2(a), but chosen-sequence probabilities "barely increase" and rejected probabilities still rise. This ablation confirms that SFT — whether on model-generated correct solutions or on human-written ones — fails to create strong separation between correct and incorrect reasoning in the model's output distribution.

SFT initialization for Iterative RPO: When Iterative RPO's first iteration is initialized from the SFT-on-chosen model rather than from Llama-2-70B-Chat directly, GSM8K accuracy is identical at 73.1% (Table 1, bottom). The paper presents this as a practical advantage — "the training process is simpler without the SFT step" — but it also suggests that the DPO+NLL objective effectively overwrites whatever starting distribution the SFT initialization provides, converging to the same endpoint regardless. This is a robustness check for the method's sensitivity to initialization, though tested on only one task.

Data quantity vs. iteration count (Table 1, bottom rows): On GSM8K, STaR with twice as much data reaches 66.9% (vs. 65.2% for standard STaR), and Iterative RPO's first iteration with twice as much paired data reaches 74.8% (vs. 73.1% with standard data quantity). Both methods benefit from more data, but two iterations of standard Iterative RPO (78.0%) substantially outperform one iteration with double data (74.8%). This ablates the hypothesis that the gains from iteration are purely due to having more training pairs — the on-policy data from an improved model is qualitatively better than more off-policy data from a static model.

Temperature adjustment across iterations: For GSM8K, generation temperature increases from 0.8 (iterations 1–2) to 1.3 (iterations 3–4). For MATH, from 0.8 (iterations 1–2) to 1.0 (iteration 3). For ARC, from 0.8 (iterations 1–2) to 1.3 (iteration 3). The paper does not ablate this choice (e.g., by holding temperature constant or testing alternative schedules), so the claim that higher temperature is "hoping that there is a significant number of incorrect generations in later iterations" is justified only by necessity (without it, training data would shrink), not by experimental comparison to alternatives.

Number of iterations: The paper uses 4 iterations for GSM8K, 3 for ARC, and 3 for MATH. There is no ablation varying the number of iterations (e.g., stopping GSM8K at 3 to compare with ARC/MATH, or running ARC to 4 to see if further gains are possible). The stopping points appear to be determined by observed saturation, but the paper does not report intermediate checkpoints that would make this transparent nor does it test whether additional iterations would degrade performance (overfitting).

Gold solution injection: The paper injects the gold (dataset-provided) CoT solution into the winning set $G_i^w$ for GSM8K and MATH to prevent empty winning sets. The effect of this injection is not ablated — there is no comparison of Iterative RPO with and without the gold solution. This matters because the gold solution is human-written and may be qualitatively different from model-generated correct solutions (different style, different reasoning paths, different vocabulary), potentially providing a stronger training signal for some questions than the model's own correct generations. On ARC, gold CoT is not available and thus not injected, making the ARC results a partial ablation — but since ARC and GSM8K/MATH differ in many dimensions (task format, baseline performance, number of iterations), the effect of gold injection cannot be isolated.

Critical Assessment

Claim 1: Iterative preference optimization improves chain-of-thought reasoning across iterations.

The evidence in Tables 1 and 2 strongly supports that Iterative RPO yields compounding accuracy improvements on GSM8K (55.6% → 73.1% → 78.0% → 81.1% → 81.6%), MATH (12.5% → 17.7% → 19.9% → 20.8%), and ARC-Challenge (77.8% → 84.8% → 86.2% → 86.7%). The gains are monotonic and substantial relative to both the zero-shot baseline and all tested baselines using the same data.

However, the paper demonstrates improvement for only 3–4 iterations before saturation, and the per-iteration gains decay rapidly. On GSM8K, the fourth iteration adds only 0.5 percentage points — within the range that could be explained by checkpoint selection noise or the retraining-with-full-data procedure. The paper does not establish whether the iteration count itself is optimal or whether performance would degrade if iterations continued (overfitting to the fixed prompt set), which would materially affect the claim that "iterations of Iterative RPO yield improved reasoning."

The fixed-prompt limitation is significant: all iterations train on the same ~7.5k–12.5k questions with different model-generated CoTs. The saturation pattern is consistent with the model exhausting the useful learning signal from a finite prompt set rather than reaching a fundamental capability ceiling. Without testing Iterative RPO with an expanded or augmented prompt set, the paper cannot distinguish between "the method saturates because the prompts are exhausted" and "the method saturates because it has extracted all possible reasoning improvements from preference optimization on this data." The former would mean Iterative RPO's gains could be extended by adding more training questions (orthogonal data augmentation), while the latter would mean the method faces a hard ceiling regardless of prompt set size.

Claim 2: The NLL term in the DPO+NLL loss is essential — standard DPO degrades or fails to improve reasoning.

The ablation comparing DPO+NLL to standard DPO is the paper's strongest experimental result. On GSM8K, the 11.3 percentage point gap (73.1% vs. 61.8%, Table 1) is unambiguous. On MATH, standard DPO's degradation below the zero-shot baseline — 12.4% vs. 12.5% from Llama, and 10.5% from SFT initialization (Table 2) — provides even stronger evidence that standard DPO is not merely ineffective but actively harmful for difficult reasoning tasks. The log-probability trajectory analysis in Figures 3 and 4 provides a mechanistic explanation (chosen-sequence probability collapse) that is consistent across all three tasks.

The claim is well-supported but its generality is untested. The paper demonstrates the NLL term's necessity for three reasoning benchmarks on one model family (Llama-2-70B-Chat). Whether the probability collapse is specific to reasoning tasks, to this model scale, or to the DPO hyperparameter $\beta = 0.1$ cannot be determined from these experiments. A larger $\beta$ (which the paper tuned but rejected) might exacerbate the collapse; a smaller $\beta$ might mitigate it without needing the NLL term. The paper's $\beta$ tuning is reported only for GSM8K iteration 1, and the selected value (0.1) is used for all tasks and iterations without further ablation. If the probability collapse is $\beta$-dependent, the NLL term's "necessity" may be conditional on the specific $\beta$ chosen.

Additionally, the ARC results complicate the narrative: standard DPO reaches 82.8% on ARC, only 2.0 percentage points below Iterative RPO's first iteration (84.8%). This is a much smaller gap than on GSM8K or MATH, suggesting that the NLL term's importance varies substantially by task. The paper does not analyze why — one hypothesis is that ARC's multiple-choice format makes the probability collapse less damaging because the model only needs to rank a small set of answer letters rather than generate free-form CoT, but this is not tested.

Claim 3: Preference optimization with negative examples outperforms SFT-based iterative methods (STaR) that use only positive examples.

The comparison to STaR is limited to one iteration on GSM8K (Table 1): Iterative RPO iteration 1 (73.1%) vs. STaR (65.2%). This is a 7.9 percentage point gap, which is substantial. However, the paper does not compare multi-iteration STaR to multi-iteration Iterative RPO — it only runs STaR for one iteration. Prior work (Zelikman et al., 2022; Singh et al., 2024) has shown that STaR and ReSTEM^\text{EM} benefit from multiple iterations. Without running STaR for 3–4 iterations on GSM8K, the paper cannot establish whether Iterative RPO's iterative advantage is due to preference optimization or simply due to running more iterations. The single-iteration comparison shows that preference optimization outperforms SFT-on-chosen given the same iteration budget, but does not rule out the possibility that 3–4 iterations of STaR might close much of the gap.

The mechanistic evidence for negative examples' value comes from Figure 2, which shows SFT increasing rejected-sequence probabilities. But this figure shows log-probability trajectories, not downstream task accuracy. The implicit causal chain — SFT inflates rejected probabilities → this limits accuracy → DPO term fixes it by pushing rejected probabilities down → accuracy improves — is plausible but not directly tested. An ablation that would strengthen this claim: train with DPO+NLL but where the rejected sequences are randomly sampled incorrect answers (rather than model-generated near-misses), to test whether the value of negative examples comes from their contrastive nature or specifically from being on-policy errors.

Claim 4: Improvement comes from iterative refinement of the data-generating model, not just from additional data.

The ablation comparing one iteration with double data (74.8%) to two iterations with standard data (78.0%) on GSM8K supports this claim directionally, but the evidence is thin. The 3.2 percentage point gap could be influenced by differences in training dynamics (two separate training runs with checkpoint selection vs. one longer run) or by the fact that the doubled-data run uses pairs generated by the weaker $M_0$ model while the two-iteration run's second iteration uses pairs from the improved $M_1$ model. The paper does not control for total training steps — the two-iteration run trains twice (once for each iteration) while the doubled-data run trains once, potentially seeing different optimization trajectories independent of data quality.

A stronger test would be: run one iteration of Iterative RPO initialized from $M_0$, then generate new pairs using the resulting $M_1$ model, and compare training $M_2$ from $M_0$ on these new pairs (non-iterative — always initialize from $M_0$) to training $M_2$ from $M_1$ on these new pairs (iterative, as the paper does). This would isolate whether the benefit comes from starting from a better initialization (iterative) or from having better training data generated by a better model. The paper does not run this ablation, so the claim that "iterative refinement of the data-generating model" is the key mechanism conflates two factors: (a) better training data from an improved model and (b) initializing from that improved model.

General Experimental Weaknesses

Single model family, single scale. All experiments use Llama-2-70B-Chat. The paper claims the method provides "a simple recipe that has the potential to improve the reasoning ability of LLMs over a wide range of tasks, as shown on the three tasks we consider," but the "over a wide range of tasks" part is tested across only three reasoning benchmarks, all from standard evaluation suites. There is no evidence on how Iterative RPO behaves with smaller models (where the probability collapse might be more or less severe), with non-Llama architectures, or with models at different base capability levels. The 70B scale is substantial; it is unclear whether the gains would replicate at 7B or 13B, where the base model's reasoning is much weaker and might not produce enough correct solutions at any temperature to seed meaningful preference pairs.

The gold solution injection is untested. For GSM8K and MATH, the paper injects the human-written CoT into the winning set for questions where the model generates no correct solutions. This is a form of data augmentation (adding human demonstrations) that breaks the closed-loop self-improvement framing. The paper does not ablate this injection, so the contribution of human-written CoTs to the overall gains is unknown. On ARC, where no gold CoT exists and questions with zero correct generations are simply excluded, the iteration counts are lower (3 vs. 4), and the gains compress faster — but this could be due to the multiple-choice format rather than the absence of gold CoT injection. A version of Iterative RPO on GSM8K without gold injection would reveal whether the method can bootstrap from very low initial correctness (where many questions have zero correct model-generated solutions) without human-written fallbacks.

Narrow comparison to prior preference optimization methods. The paper compares to standard DPO and SFT, but does not compare to several recent preference optimization variants that also combine likelihood terms: DPO-Positive (Pal et al., 2024), which adds an NLL-like term to DPO; Contrastive Preference Optimization (Xu et al., 2024); or ORPO (Hong et al., 2024), which combines SFT and preference optimization in a single objective. The paper cites all three in Section 3.1 as "related observations" but provides no experimental comparison, even though these methods address the same probability collapse problem. This makes it difficult to assess whether the DPO+NLL formulation is specifically effective or whether any objective that combines preference optimization with likelihood preservation would work equally well.

No error analysis of final model behavior. The paper reports only aggregate accuracy numbers. There is no analysis of what kinds of errors the final model makes compared to the base model — whether improvement comes from fixing specific error types (arithmetic mistakes, logical leaps, misinterpretation of questions), from the model becoming better at recovering from its own mistakes, or from the model simply being more confident in answers it was already sometimes producing. This limits the scientific insight into what the method teaches the model beyond the aggregate metric. An error taxonomy across iterations would strengthen the mechanistic story considerably.

Fixed hyperparameters across tasks and iterations. The paper tunes $\alpha$ and $\beta$ only on GSM8K iteration 1, then uses the same values ($\alpha = 1$, $\beta = 0.1$) for all subsequent iterations and for all three tasks. This simplifies the method but leaves open the possibility that per-task or per-iteration tuning could yield substantially different results. Given that the NLL term's benefit varies across tasks (small on ARC, large on GSM8K and MATH), task-specific $\alpha$ tuning might narrow or widen these gaps.

The validation protocol introduces a subtle data dependency. The two-stage procedure — tune on held-out 1k, then retrain with those 1k included — means the final model at each iteration sees the entire training set, but the checkpoint selection is based on performance on data that was subsequently incorporated into training. This creates a form of implicit overfitting where the selected model may have memorized patterns that generalize to the former held-out set but not to the test set. The paper does not report whether test performance is sensitive to the specific 1k held-out split (e.g., through multiple random splits), which would be necessary to rule out this concern.

6. Limitations and Trade-offs

The Fixed Training Prompt Bottleneck: Finite Data Caps Iterative Improvement

The assumption or constraint. Iterative RPO operates over the same fixed set of training prompts across all iterations — the GSM8K training set (~7.5k questions), the MATH training set (~12.5k questions), and the ARC training set (~7.7k questions). The paper never introduces new questions, relying entirely on iterative refinement of the model's own reasoning on a static input distribution. The paper explicitly connects this to saturation: Section 3.1 notes that "the gain decays across the iterations (17.5%, 4.9%, 3.1%, 0.5%), indicating an upper limit on learning across iterations, especially as we are iterating across a fixed number of prompts." The authors also acknowledge in Appendix A that "we have experimented on three tasks. It is unclear how the approach would perform on general instruction tuning tasks without a clear best answer."

The consequence. The gains from Iterative RPO are bounded by the information content of the fixed training set, and this bound is reached quickly — by iteration 4 on GSM8K the gain over the previous iteration is only 0.5 percentage points. A practitioner cannot expect continued improvement by running additional iterations; they would need to source new training prompts, which is exactly the data-augmentation pipeline the method was designed to avoid. More subtly, this means Iterative RPO's headline improvements (55.6% → 81.6% on GSM8K) are specific to the size and composition of the particular training set used — a smaller training set would likely saturate earlier, and the paper provides no guidance on how much training data is required to achieve meaningful gains. For a benchmark with only a few hundred training examples, the method might exhaust its improvement potential after a single iteration. Furthermore, the fixed-prompt design means the model never sees novel problem types or reasoning patterns beyond those in the original training distribution, limiting its ability to generalize to out-of-distribution reasoning tasks.

What evidence exists in the paper. The diminishing-returns trajectory is documented quantitatively in Table 1 for GSM8K (gains: 17.5, 4.9, 3.1, 0.5 percentage points) and can be computed from Table 2 for MATH (5.2, 2.2, 0.9) and ARC (7.0, 1.4, 0.5). The progressive shrinkage of training data on ARC — "around 20k example pairs for iteration 1, 11k example pairs for iteration 2, and 5k example pairs for iteration 3" (Section 3.2) — provides direct evidence that the model exhausts its ability to generate incorrect solutions, leaving progressively less training signal. The comparison to methods that do use augmented data (MetaMath at 82.3%, WizardMath at 81.6% on GSM8K) shows that Iterative RPO reaches comparable performance without data augmentation, but provides no evidence about whether combining augmentation with Iterative RPO would push past the saturation point.

Mitigation status. The paper explicitly acknowledges this limitation: "The current recipe requires correct answers, and a clear metric for comparing a generated response with this correct answer" (Appendix A). It also notes that data-augmentation approaches "should be orthogonal to ours, and both can provide benefits" (Section 3.1). There is no experimental exploration of combining Iterative RPO with data augmentation, leaving the practical question of how to extend gains past the fixed-prompt saturation point to future work. The paper also does not explore curriculum strategies (e.g., starting with easier questions and progressively introducing harder ones across iterations), which might extract more learning signal from a fixed prompt set.

The Gold Solution Injection Breaks the Closed-Loop Self-Improvement Claim

The assumption or constraint. For GSM8K and MATH, the paper injects the dataset-provided human-written chain-of-thought solution $(c_i, y_i)$ into the winning set $G_i^w$ whenever the model fails to generate any correct solution for a question, "so it is not empty" (Section 3.1). This injection is applied at every iteration, meaning that for questions the model consistently gets wrong, the winning example in the preference pair is always the human-authored CoT — not a model-generated correct solution. The paper positions Iterative RPO as a closed-loop self-improvement method that "does not require human-in-the-loop or extra training data" (Section 5, Conclusion), but the gold CoT injection is precisely extra human-authored training data that the model would not have generated on its own.

The consequence. The gold solution injection introduces an unmeasured dependency on human demonstrations. For hard questions where the base model's pass@30 is near zero, the winning side of every preference pair is the human-written CoT, meaning the model is effectively doing standard supervised fine-tuning on those examples — not learning from its own correct reasoning. The paper cannot distinguish between two possible sources of the observed gains: (a) the iterative preference optimization mechanism working as claimed (model learns from its own errors contrasted against its own successes), or (b) the gold CoT injection providing a direct SFT signal on the hardest questions, with the DPO+NLL loss simply doing a better job of internalizing it than pure SFT. If (b) dominates, then the method's claimed advantage over SFT-based methods is partially an artifact of better optimization of the same human demonstrations, not a fundamentally different learning mechanism.

This is particularly problematic for the MATH results, where baseline accuracy is only 12.5% and many questions likely have zero model-generated correct solutions at early iterations. The gold CoT injection may be doing substantial work in bootstrapping the model's performance from 12.5% to 17.7% in the first iteration, but the paper provides no ablation that measures this contribution. For a practitioner with a reasoning dataset that lacks human-written chain-of-thought (like ARC, or a custom domain-specific task), the method's performance without this crutch is unknown.

What evidence exists in the paper. The paper does not ablate the gold solution injection. There is no experiment comparing Iterative RPO with and without gold CoTs on a task where human-written CoTs exist (GSM8K or MATH). ARC provides a partial but confounded ablation: ARC has no gold CoTs, so questions with zero correct generations are simply excluded from training. On ARC, the method reaches only 3 iterations (vs. 4 for GSM8K) and the training data shrinks from ~20k pairs to ~5k pairs — but ARC also differs in task format (multiple-choice vs. free-form), baseline performance (77.8% vs. 55.6%), and prompt format (zero-shot vs. few-shot), making it impossible to attribute these differences specifically to the absence of gold CoT injection. The paper is silent on what fraction of training questions on GSM8K and MATH rely on injected gold CoTs at each iteration — this number is essential for assessing how much of the training signal is model-generated vs. human-authored.

Mitigation status. Not addressed. The paper describes the injection as a practical workaround ("Since some problems might not have any model-generated correct solution, we include the gold human written solution") without acknowledging it as a limitation of the self-improvement framing. Appendix A's limitation statement does not mention it. The paper does not propose alternatives (e.g., using the model's own highest-confidence incorrect solution with the correct answer appended as in STaR's rationalization step) or discuss how the method would perform without this injection.

Computationally Prohibitive at Scale — Each Iteration Costs 64 GPUs and Generates Tens of Millions of Tokens

The assumption or constraint. The paper's experimental setup uses substantial computational resources: all training runs on "eight nodes each containing eight A100 GPUs (80G memory)" — 64 A100-80GB GPUs per training run — with up to 5,000 training steps per iteration, batch size 16, and a 70B-parameter model (Section 3.1). Each iteration also requires generating $N = 20–30$ full chain-of-thought solutions per training question, across thousands of questions, using 8×V100 GPUs. The total cost per iteration includes (a) sampling $N \times |D|$ long-context CoT sequences, (b) extracting and scoring answers, (c) constructing preference pairs, and (d) training a 70B model on these pairs for thousands of steps. The paper does not report total GPU-hours, FLOPs, or dollar cost for any experiment.

The consequence. A practitioner considering Iterative RPO faces a steep resource barrier. Training a 70B model on 64 A100 GPUs is beyond the reach of most academic labs and many industrial teams. Even at smaller model scales (7B, 13B), the cost of generating $N = 30$ CoT solutions per training question — where CoT solutions can be hundreds of tokens long — may dominate the training cost itself, especially for large training sets. The paper provides no guidance on how the method scales with model size: would a 7B model require the same number of iterations? The same number of generated solutions per question? The same $\alpha$ and $\beta$? Without this information, practitioners at smaller scales cannot reliably estimate whether the reported gains justify the computational investment. The paper's practical advice — "a simple recipe that has the potential to improve the reasoning ability of LLMs" — understates the resource requirements that make this recipe simple only at the scale of large industrial labs.

There is also a subtle data scaling cost: each iteration generates $N \times |D|$ CoT solutions, and for GSM8K with $N = 30$ and $|D| \approx 7,500$, this is ~225,000 full CoT generations per iteration. Across four iterations, this is nearly 1 million generated reasoning chains, each of which must be scored against gold answers and filtered for context length. For MATH with $N = 20$ and $|D| = 12,500$, the per-iteration cost is 250,000 generations. The paper does not compare this to the cost of alternative approaches: SFT on gold CoTs requires zero generation cost (just training on existing human-written data), and data augmentation methods like MetaMath incur a one-time data generation cost rather than a per-iteration cost. The 4×4\times iteration overhead (four separate generation + training cycles) may make Iterative RPO substantially more expensive than methods that achieve similar final accuracy with a single training stage.

What evidence exists in the paper. The hardware specification (Section 3.1: "eight nodes each containing eight A100 GPUs") and training hyperparameters (batch size 16, up to 5,000 steps per iteration, 70B model) allow a rough estimate of cost: training a 70B model for 5,000 steps at batch size 16 on 64 A100 GPUs is on the order of hundreds to thousands of A100-hours per iteration, plus the generation cost on V100 GPUs. The paper provides no further cost analysis. The fact that the method saturates in 3–4 iterations bounds the total cost, but the per-iteration cost is still high enough to be a practical barrier.

Mitigation status. Not addressed. The paper does not discuss computational cost as a limitation, does not report FLOPs or GPU-hours, and does not explore cost-reduction strategies (e.g., reducing $N$, using fewer iterations with more data per iteration, distilling the trained model into a smaller one, or using smaller models). Appendix A mentions that data from previous iterations is not reused, which could reduce per-iteration generation cost if implemented, but this is framed as a potential improvement rather than as a response to a recognized cost limitation.

No Evidence Beyond a Single Model Family — Llama-2-70B-Chat as a Fragile Single Point of Validation

The assumption or constraint. All experiments in the paper use exactly one base model: Llama-2-70B-Chat (Touvron et al., 2023). The paper's claims about Iterative RPO's effectiveness are therefore conditional on this specific model architecture, scale, and pretraining distribution. The paper states that the method "has the potential to improve the reasoning ability of LLMs over a wide range of tasks" (Section 5, Conclusion), implying generality across models, but provides no evidence beyond Llama-2-70B.

The consequence. Several components of Iterative RPO could be sensitive to model-specific properties that do not transfer to other architectures or scales. The probability collapse under standard DPO (Figures 3 and 4) — which motivates the entire DPO+NLL design — may be specific to Llama-2-70B-Chat's output distribution, calibration properties, or the way its instruction tuning interacts with chain-of-thought generation. A model with different calibration (e.g., better-calibrated uncertainty, different base rates of correct vs. incorrect CoT generation) might exhibit less severe probability collapse, making the NLL term unnecessary, or might exhibit more severe collapse, making the NLL term insufficient. The paper's $\alpha = 1$ and $\beta = 0.1$ hyperparameters were tuned on this specific model on GSM8K — there is no evidence these values generalize.

The paper's key finding — that the NLL term is "essential" — is model-conditional. If another model family (e.g., Mistral, Qwen, or a non-instruction-tuned base model) does not experience the same chosen-sequence probability collapse under standard DPO, then the DPO+NLL loss reduces to DPO with an unnecessary regularizer, and the paper's central claim about necessity would not hold. Similarly, the iterative gains depend on the model producing a sufficient diversity of both correct and incorrect solutions at each iteration. A model that is either much stronger (nearly always correct, producing few negatives) or much weaker (nearly always incorrect, producing few positives) would not generate informative preference pairs, and the method might fail to improve or even degrade.

At smaller scales (7B, 13B), the base model's reasoning capability is substantially weaker — GSM8K zero-shot accuracy for Llama-2-7B-Chat is far below 55.6%. Such a model would generate many more incorrect solutions and fewer correct ones, potentially requiring different $N$, $K$, and temperature schedules to construct informative preference pairs. The paper provides no scaling analysis across model sizes, leaving practitioners at smaller scales with no guidance on whether Iterative RPO is worth attempting.

What evidence exists in the paper. None beyond Llama-2-70B-Chat. The paper does not include experiments with Llama-2-7B-Chat, Llama-2-13B-Chat, any non-Llama model, any non-instruction-tuned base model, or any model with a different pretraining data mixture. The three benchmarks (GSM8K, MATH, ARC-Challenge) provide some task diversity, but the model family is held constant, making it impossible to separate model-specific effects from task-general effects.

Mitigation status. The paper does not acknowledge this as a limitation. Appendix A focuses on task-related limitations (only three tasks, unclear performance on general instruction tuning) and loss-function limitations (NLL necessity when training data comes from other models), but does not mention the single-model-family constraint. The generality claim in the conclusion is stated without qualification, and there is no call for future work to validate on other model families.

Evaluation on Narrow Reasoning Benchmarks Only — No Evidence of Broader Reasoning Improvement or Transfer

The assumption or constraint. The paper evaluates Iterative RPO exclusively on three standard reasoning benchmarks: GSM8K (grade-school math), MATH (competition math), and ARC-Challenge (multiple-choice science). All three tasks have the same fundamental structure: a question with a single unambiguous correct answer that can be verified by exact string match. The paper does not evaluate on tasks requiring open-ended reasoning (e.g., logical argumentation, multi-step planning, code generation), tasks where multiple valid reasoning paths exist, tasks requiring factual knowledge beyond what the model already possesses, or tasks where correctness is graded on a continuous scale rather than binary.

The paper acknowledges this scope limitation explicitly: "It is unclear how the approach would perform on general instruction tuning tasks without a clear best answer" (Appendix A), but frames this narrowly — the concern is about tasks lacking a clear reward signal, not about the broader question of whether improved math and science reasoning transfers to other reasoning capabilities.

The consequence. A practitioner seeking to improve an LLM's general reasoning capabilities cannot infer from these results that Iterative RPO will help. The benchmarks tested share a specific structure — formally verifiable answers — that makes them compatible with the binary reward mechanism the method requires. For reasoning tasks without clean answer verification (e.g., "explain the economic causes of the 2008 financial crisis"), the method's core loop — generate, extract answer, check exact match — breaks down entirely. The paper suggests that "more general reward models" could substitute for exact match, but provides no experiments with such reward models, leaving open the possibility that reward model errors compound across iterations (the verification bottleneck the paper itself identifies as fatal to Self-Rewarding LLMs).

Even within the class of answer-verifiable reasoning tasks, the paper does not establish that improvements on GSM8K, MATH, and ARC indicate a general improvement in mathematical or scientific reasoning capability — as opposed to improvement on the specific distribution of problems, question formats, and answer formats represented in these benchmarks. The fixed-prompt training design means the model is iteratively refined on the exact same distribution as the test set (just different questions), making it impossible to distinguish genuine reasoning improvement from distribution-level memorization or format adaptation. The paper does not test on held-out reasoning benchmarks (e.g., evaluating a GSM8K-trained model on a different math word problem dataset) to assess transfer.

The paper also provides no analysis of whether the improvements come from better chain-of-thought reasoning versus better extraction of the final answer — the "Final answer:" format is explicitly prompted, and improved accuracy could partly reflect the model learning to better format its output for the exact-match grading function rather than genuinely reasoning more accurately.

What evidence exists in the paper. The three benchmarks are evaluated separately (Tables 1 and 2), with no cross-task transfer experiments (e.g., training on GSM8K and testing on a held-out math benchmark, or training on ARC and testing on a different science QA dataset). The paper does not report per-category breakdowns within benchmarks (e.g., MATH has subcategories like algebra, geometry, number theory; ARC has physics, biology, chemistry questions). The prompts are task-specific (Section B.1), meaning the model is trained with task-specific formatting instructions ("Final answer:", "\boxed{}", multiple-choice letter selection), which may contribute to format overfitting.

Mitigation status. The paper acknowledges this limitation partially in Appendix A: "It is unclear how the approach would perform on general instruction tuning tasks without a clear best answer, but we argue that positive results on the three tasks in this paper can already prove the method useful." The hedging phrase "already prove the method useful" suggests the authors view the three benchmarks as sufficient evidence of utility, without acknowledging the gap between benchmark performance and general reasoning capability. The paper does not propose future work on extending Iterative RPO to tasks with learned reward models or open-ended evaluation, beyond the brief mention of "more general reward models" in Section 2.

Training Data Shrinks Across Iterations, Undermining the Claim of Iterative Improvement

The assumption or constraint. Iterative RPO constructs preference pairs from the current model's own generations, requiring both correct and incorrect solutions for each training question. As the model improves across iterations, the pool of incorrect solutions shrinks — and for many questions, it disappears entirely. The paper documents this directly for ARC-Challenge: "The decrease in the number of examples is due to the lack of incorrect samples for a number of questions in later iterations" (Section 3.2), with training pairs dropping from ~20k to ~11k to ~5k across three iterations. The same phenomenon presumably affects GSM8K and MATH, though the paper does not report the training data size per iteration for those tasks.

The consequence. The diminishing returns across iterations may be partly or largely attributable to shrinking training data size rather than (or in addition to) exhaustion of the fixed prompt set. As the model becomes more accurate, it generates fewer incorrect solutions, which means fewer preference pairs can be constructed — reducing the amount of training signal available per iteration. This creates a self-limiting dynamic: the better the model gets, the less data it has to learn from in the next iteration. The performance trajectory (rapid gains, then quick saturation) could be explained by this data shrinkage alone, without invoking any conceptual ceiling on what preference optimization can extract from the fixed prompts.

This is particularly problematic for the paper's claim that "iterative refinement of the data-generating model" drives improvement (Section 4). If later iterations have substantially less training data, the comparison between two iterations (78.0%) and one iteration with double data (74.8%) on GSM8K (Table 1) may underestimate the value of data quantity. The two-iteration run sees one full iteration with standard data quantity (~55–60k pairs) and a second iteration with potentially fewer pairs (since $M_1$ is more accurate than $M_0$ and generates fewer incorrect solutions), while the one-iteration-double-data run sees a single training stage with ~110–120k pairs. If the second iteration's data size dropped to, say, ~30k pairs, then the iterative advantage (78.0% vs. 74.8%) is actually an advantage of having higher-quality on-policy data from $M_1$ despite having much less of it — which would be an even stronger result, but the paper doesn't provide the data to support this interpretation.

For ARC, where the data shrinkage is most severe (a 75% reduction from iteration 1 to iteration 3), the diminishing returns are extreme — iteration 3 adds only 0.5 percentage points — and it is unclear whether this is because the model has reached a capability ceiling or simply because 5k training pairs provide insufficient signal for meaningful improvement on a 70B model.

What evidence exists in the paper. The ARC training data sizes are reported explicitly (Section 3.2). For GSM8K, only the iteration-1 data size is reported ("around 55–60k pairs for training, per iteration"), with the phrase "per iteration" implying it is stable, but this cannot be strictly true if the model improves — a more accurate model must, by definition, generate fewer incorrect solutions at the same temperature. The paper's temperature increase in later iterations (τ = 1.3 for GSM8K iterations 3–4, τ = 1.3 for ARC iteration 3) is an implicit acknowledgment that the model would otherwise produce too few incorrect solutions, but the paper does not report whether this temperature increase successfully maintained training data size, or whether the resulting higher-temperature incorrect solutions are qualitatively different from the lower-temperature ones used in earlier iterations. The paper does not ablate the temperature schedule to isolate its effect on training data quantity and quality.

Mitigation status. The temperature increase is a partial mitigation — it aims to maintain a supply of incorrect solutions by increasing sampling diversity. However, this mitigation is itself a tradeoff: higher-temperature incorrect solutions may include errors that are unrepresentative of the model's typical failure modes (random noise rather than systematic reasoning errors), reducing the informativeness of the preference pairs. The paper does not discuss this tradeoff or measure the distribution shift in incorrect solutions across temperatures. The paper also does not consider alternative strategies for maintaining training data size, such as reusing incorrect solutions from earlier iterations, generating additional incorrect solutions by prompting the model to produce common errors, or weighting training examples by difficulty rather than discarding questions that become "too easy."

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a methodological reframing rather than a paradigm shift. Prior work had established that iterative preference optimization works for instruction-following but not reasoning, and had established that iterative SFT works for reasoning but plateaus quickly. The field was converging on an implicit consensus: preference optimization and reasoning improvement are fundamentally different problems requiring fundamentally different approaches. Iterative RPO breaks this consensus by showing that preference optimization can work for reasoning — if and only if you add an NLL term that prevents the probability collapse that standard DPO induces on correct reasoning sequences.

The magnitude of this reframing is significant but bounded. It is not a paradigm shift because the core mechanisms (DPO, iterative self-training, chain-of-thought generation) are all inherited from prior work. But it is more than an incremental refinement because it identifies and solves a specific failure mode — the DPO probability collapse — that had been observed but not systematically diagnosed as the primary blocker for reasoning. The paper's diagnostic contribution (Figures 2, 3, 4) gives the field a concrete mechanistic explanation for why prior methods failed: standard DPO decreases chosen-sequence probabilities, and SFT inflates rejected-sequence probabilities alongside chosen ones. The DPO+NLL loss is the targeted fix for both halves of this problem.

This reframing reconciles a set of contradictory findings that had accumulated in the reasoning literature. Yuan et al. (2024) and Chen et al. (2024) reported that iterative DPO methods make "only modest gains on reasoning tasks." Huang et al. (2024) demonstrated that LLMs "cannot self-correct reasoning yet" when using standard prompting or SFT-based approaches. Meanwhile, STaR (Zelikman et al., 2022) and ReST^EM (Singh et al., 2024) showed that iterative SFT on filtered correct solutions does improve reasoning, but modestly and with diminishing returns. These findings appeared contradictory — why would iterative training help in some forms but not others? The probability collapse and probability inflation diagnostics resolve the puzzle: iterative SFT helps because it increases correct-sequence probability, but plateaus because it cannot discriminate against incorrect sequences; iterative DPO fails because it discriminates but at the cost of collapsing correct-sequence probability; Iterative RPO succeeds because the NLL term maintains correct-sequence probability while the DPO term provides the discrimination that SFT lacks. The contradiction was not in the empirical results but in the implicit assumption that any single mechanism (preference optimization, or positive-example SFT) could address both requirements.

The paper also redirects research attention in two specific ways. First, it makes the study of training dynamics under preference optimization — particularly the trajectory of chosen-sequence log-probabilities — a first-class diagnostic that future methods should report. Prior DPO papers typically report only final accuracy and win rates; Figures 3 and 4 demonstrate that the path the model takes during training reveals whether the optimization is functioning as intended or collapsing. Second, it shifts the burden of proof for new reasoning improvement methods: any method claiming to improve reasoning through preference optimization must now demonstrate that it does not exhibit the probability collapse documented here, or explain why its specific design avoids it. Methods that report only aggregate accuracy without probing the chosen-vs-rejected probability gap are operating with an incomplete diagnostic picture.

A research direction that becomes less attractive is the pursuit of ever-more-complex iterative DPO pipelines for reasoning without an NLL-like likelihood-preservation term. The paper's central empirical finding — that standard DPO degrades MATH performance from 12.5% to 12.4% (or to 10.5% when initialized from SFT, Table 2) — demonstrates that this is not a marginal tuning issue but a structural failure. Iterative DPO variants that do not address the probability collapse (e.g., using larger preference datasets, stronger reward models, or more iterations) are likely to hit the same wall.

Research directions that become more attractive include: (a) theoretical analysis of why DPO causes probability collapse specifically on reasoning tasks (is it a property of the long-context CoT sequences? the sharp correctness boundaries? the entropy of the base model's output distribution?), (b) alternative mechanisms for maintaining chosen-sequence probability beyond the NLL term (KL penalties, trust-region constraints, adaptive $\beta$ schedules), and (c) combining Iterative RPO with data augmentation — since the paper shows saturation on fixed prompts, adding new training questions is the obvious path to extending the gains beyond the 3–4 iteration ceiling.

Follow-Up Research This Work Enables

Diagnosing whether the probability collapse is task-specific, model-specific, or universal. The paper demonstrates the chosen-sequence probability collapse under standard DPO on three reasoning benchmarks with Llama-2-70B-Chat, but provides no evidence about its generality. A strong follow-up would replicate Figures 3 and 4 across: (a) non-reasoning tasks (instruction-following, summarization, translation) to test whether the collapse is reasoning-specific or hidden in domains where absolute probability matters less for accuracy; (b) multiple model families (Mistral, Qwen, Gemma) and scales (7B, 13B, 70B) to test whether it is a Llama-specific property or a general consequence of DPO optimization on long-context outputs; and (c) different $\beta$ values in standard DPO — the paper only tested $\beta = 0.1$, and it is plausible that much smaller $\beta$ (e.g., 0.01) would mitigate the collapse by reducing the gradient pressure on the chosen sequence, potentially making the NLL term unnecessary. A finding that the collapse is specific to Llama-2 or to $\beta \ge 0.1$ would substantially narrow the method's claimed generality.

Measuring how much of Iterative RPO's gains come from the gold CoT injection. The paper injects human-written chain-of-thought solutions into the winning set for GSM8K and MATH but never ablates this injection. A clean follow-up experiment would run Iterative RPO on GSM8K with and without the gold CoT injection, measuring both final accuracy and the per-iteration trajectory. If the gold CoT is essential for bootstrapping performance on hard questions (where the model initially generates zero correct solutions), then the method's claim to be a closed-loop self-improvement system would be weakened, and the practical applicability to domains without human-written reasoning demonstrations would be limited. Conversely, if the method performs nearly as well without gold CoTs — perhaps because questions with zero correct solutions are rare enough that excluding them does not hurt, or because the model can bootstrap from partial correctness — then the closed-loop claim is validated. The paper could also test an alternative bootstrapping mechanism: for questions with zero correct model generations, use the model's highest-log-probability incorrect solution paired with a prompt that gives the correct answer and asks the model to generate reasoning leading to it (STaR's rationalization step), then use that generated reasoning as the winning sequence. This would preserve the self-improvement framing by keeping all training data model-generated.

Combining Iterative RPO with data augmentation to break through the fixed-prompt saturation ceiling. The paper shows that gains saturate by iteration 3–4 on a fixed prompt set, and notes that data augmentation methods like MetaMath (82.3% on GSM8K) and WizardMath (81.6%) are "orthogonal" but never combines them. A natural follow-up would apply Iterative RPO starting from MetaMath's augmented training set — or, more interestingly, interleave data augmentation with Iterative RPO iterations: use the improved model from iteration $t$ to generate new training questions (following MetaMath's question bootstrapping approach), then use those new questions as the fixed prompt set for iteration $t+1$. This would test whether the iterative refinement mechanism (better model → better training data → better model) can be extended to the prompt distribution itself, not just the reasoning chains. The key metric would be whether the combined approach exceeds the sum of each method's individual gains (e.g., MetaMath's 82.3% plus Iterative RPO's 26 percentage point gain over the baseline, or whether the gains are largely overlapping). If the gains are additive, this would establish Iterative RPO as a general-purpose post-training step that can be applied on top of any data augmentation pipeline.

Stress-testing Iterative RPO at smaller model scales where initial correctness is low. The paper only evaluates on Llama-2-70B-Chat, which achieves 55.6% zero-shot on GSM8K — high enough to generate plenty of correct solutions for seeding preference pairs. At 7B or 13B scales, zero-shot GSM8K accuracy is much lower (likely 10–30% for instruction-tuned models), meaning the pool of correct solutions shrinks dramatically. A stress-test at 7B would answer: does Iterative RPO still work when $N = 30$ generations per question yields only a handful of correct solutions across the entire training set? If the method fails, is it because the preference pairs are too sparse, because the NLL term overfits to the rare correct sequences, or because the DPO term receives an uninformative contrastive signal from the low-quality incorrect solutions of a weak model? If the method succeeds at 7B but requires different hyperparameters (larger $N$, higher $\alpha$, more iterations), what is the scaling relationship? This experiment would determine whether Iterative RPO is practical for teams without access to 70B-scale initial models, which is most of the research community.

Measuring whether Iterative RPO improves genuine reasoning or adapts to benchmark format. The paper reports only aggregate test accuracy on the same benchmark distributions as training. A follow-up would conduct transfer experiments: train Iterative RPO on GSM8K and evaluate on (a) other math word problem datasets (SVAMP, ASDiv, MultiArith) to test distribution generalization, (b) MATH (a harder math benchmark with different formatting and problem types) to test difficulty generalization, and (c) a held-out set of GSM8K-style problems where the "Final answer:" format is changed or removed, to test whether the gains partly come from the model learning to better comply with the extraction prompt rather than genuinely reasoning better. Additionally, error analysis across iterations — categorizing errors into arithmetic mistakes, logical errors, misinterpretation, and formatting errors — would reveal what kind of reasoning improves. The paper claims the method improves "reasoning ability" but provides no evidence that the improvement is in the reasoning steps rather than in answer extraction or format compliance. A finding that most of the gain comes from fixing formatting errors or from better compliance with the "Final answer:" template would substantially weaken the reasoning-improvement claim.

Replacing the binary reward with a learned verifier to extend to tasks without gold answers. The paper's method requires exact-match verification against gold answers, which limits applicability to tasks with clean, unambiguous correctness criteria. A follow-up would train a verifier model (e.g., following the V-STaR approach of Hosseini et al., 2024, or the PRM approach from Lightman et al., 2023) on the same training data, use it to score generated solutions on a continuous scale rather than binary exact match, and construct preference pairs based on verifier scores rather than gold-label matching. This would test two hypotheses: (a) whether a continuous reward signal (allowing the method to distinguish between "slightly wrong" and "completely wrong" incorrect solutions, and between "barely correct" and "clearly correct" solutions) provides a richer training signal than binary rewards, and (b) whether the method can be extended to reasoning tasks where gold answers are unavailable or where correctness is multi-dimensional (e.g., "explain the solution" tasks, code generation with multiple valid implementations). The key risk is that verifier errors compound across iterations — the same verification bottleneck the paper identifies as fatal to Self-Rewarding LLMs — and measuring whether Iterative RPO is robust to imperfect verification would establish the boundary conditions for practical deployment beyond benchmark settings.

Practical Applications and Downstream Use Cases

Self-improving math tutors and educational tools. A deployed math tutoring system based on an open-weight model like Llama-2-70B-Chat could use Iterative RPO to continuously improve its reasoning on the specific distribution of problems students ask, without requiring new human-authored training data. The system would collect a dataset of student questions, generate multiple candidate solutions for each, verify correctness using a combination of exact-match answer checking and simple execution-based verification (running code for computational problems, checking against known answer keys for textbook problems), and periodically retrain using the DPO+NLL objective. The paper's GSM8K results — 55.6% → 81.6% accuracy improvement using only the original training set — suggest that a tutoring system starting from a reasonable baseline could substantially improve its reliability on its specific user distribution without any human annotation beyond what comes with the initial question bank. The diminishing returns across iterations (17.5 → 4.9 → 3.1 → 0.5 percentage points on GSM8K) indicate that periodic retraining (e.g., weekly or monthly as new questions accumulate) would be more practical than continuous iteration, since most of the gain comes in the first 2–3 iterations.

Distillation of iterative reasoning improvements into smaller, deployable models. The paper's training runs on 64 A100 GPUs for a 70B model are impractical for many deployment scenarios (on-device, edge, low-latency serving). However, the core insight — that Iterative RPO produces a model with substantially better reasoning than the base model on the same training data — enables a distillation pipeline: use Iterative RPO to train a 70B teacher model through 3–4 iterations, then use the final model to generate high-quality chain-of-thought solutions on the training set (or on an augmented prompt set), and distill these into a smaller student model (7B or 13B) via standard SFT. The paper does not explore distillation, but the performance trajectory — 81.6% on GSM8K after four iterations versus 55.6% zero-shot — represents a large teacher-student gap that distillation could exploit. A 7B model trained via SFT on the 70B Iterative RPO teacher's correct solutions might substantially outperform a 7B model trained via SFT on the original gold CoTs or on a weaker teacher's outputs, enabling on-device reasoning capabilities that currently require datacenter-scale models.

Cost-effective fine-tuning for domain-specific reasoning tasks with limited labeled data. For organizations with a domain-specific reasoning task (medical diagnosis, legal analysis, scientific inference) and a small set of labeled training examples (hundreds to low thousands), Iterative RPO provides a recipe for extracting maximum improvement from limited data without sourcing additional annotations. The method's core requirement — a set of questions with verifiable answers — is naturally satisfied by any domain where correctness can be programmatically checked (computation, database queries, formal proofs) or where a small labeled set already exists. The paper's results on MATH (12.5% → 20.8%, a 66% relative improvement) are particularly relevant: they demonstrate meaningful gains on a difficult task where the base model's initial accuracy is low, which is precisely the scenario facing teams with limited domain-specific data. The practical workflow would be: (1) few-shot prompt a strong base model on the domain-specific questions, (2) generate $N = 20–30$ candidate solutions per question, (3) verify correctness using whatever ground-truth labels or automatic checks are available, (4) run 2–3 iterations of Iterative RPO, and (5) deploy. The paper's finding that $\alpha = 1$ and $\beta = 0.1$ worked across all three tasks suggests these hyperparameters may transfer reasonably well to new domains without extensive tuning, lowering the barrier to adoption.

Improving open-source LLM reasoning without access to additional training data or reward models. For the open-source LLM community — where access to large-scale human preference data, curated reasoning datasets, and trained reward models is limited — Iterative RPO provides a self-contained improvement recipe that requires only a base model and a training set with ground-truth answers (which most reasoning benchmarks already provide). Unlike MetaMath (requires generating 395k new questions), WizardMath (requires RL infrastructure and evol-instruct data), or Self-Rewarding LLMs (requires an LLM-as-judge that is unreliable for reasoning), Iterative RPO can be implemented with standard training infrastructure (SFT + DPO training code, which is widely available in open-source libraries like TRL and Axolotl) and no external data dependencies beyond the benchmark's training set. The paper's reported hardware requirements (8×V100 for generation, 64×A100 for training) are high but within reach of well-resourced open-source efforts, and the method would likely scale down to 7B/13B models on smaller GPU clusters. The open-source community could apply Iterative RPO to the full suite of reasoning benchmarks (GSM8K, MATH, ARC, MMLU, BBH, etc.) and release the improved models as drop-in replacements for the base Llama, Mistral, or Qwen checkpoints, similar to how the community has released DPO-tuned and ORPO-tuned variants of base models for instruction-following — but now with demonstrated reasoning improvement.