ArXiv: 2312.06585

🎯 Pitch

Fine-tuning on model-generated solutions filtered by correctness feedback dramatically outperforms training on human-written data alone—PaLM 2-L gains over 6% on MATH and APPS, and larger models benefit more. The self-training procedure, ReST^EM, frames this as expectation-maximization, alternating between sampling many solutions and fine-tuning on those that pass a verifier. One iteration already captures most of the gain, but repeating can further improve or cause overfitting depending on dataset size.

1. Executive Summary

This paper introduces ReST^EM — a self-training method grounded in expectation-maximization for reinforcement learning — and demonstrates that iteratively generating solutions from a language model, filtering them with a binary correctness reward, and fine-tuning on the surviving samples substantially outperforms supervised fine-tuning on human-written data. Testing on the MATH reasoning benchmark and APPS code-generation benchmark using PaLM 2 models (S, S*, and L variants), ReST^EM yields gains that scale favorably with model size — for example, improving PaLM 2-L by 6.34% absolute on MATH and 6.4% on APPS — while the human-data baseline produces markedly smaller improvements. A single iteration captures most of the gain, with subsequent iterations providing diminishing returns and eventual overfitting on the smaller APPS training set, establishing that model-generated synthetic data can reduce dependence on human demonstrations but only when the training problem set is large enough to absorb multiple rounds of self-generated fine-tuning without memorization.

2. Context and Motivation

The Core Problem: Human Data Is the Bottleneck for Fine-Tuning LLMs

The fundamental constraint this paper tackles is straightforward but consequential: supervised fine-tuning (SFT) of large language models requires human-generated demonstrations, and acquiring high-quality human data for complex problem-solving tasks is expensive, slow, and difficult to scale. The paper opens by explicitly naming this as the central challenge:

"acquiring high-quality human data poses a significant bottleneck. This is particularly demanding for complex problem-solving tasks, requiring significant resources and expert knowledge."

This bottleneck manifests in several ways that the paper explores, both explicitly and implicitly. First, human data is fundamentally quantity-limited: for a given task like competition-level mathematics or code generation, only so many problems with verified solutions exist. The MATH dataset, for instance, contains 7,500 training problems — each with a single human-written solution. Second, human data is diversity-limited: a single human demonstrator (or even a team of them) produces solutions in a particular style, with particular reasoning patterns, and with particular blind spots. The model fine-tuned on these solutions inherits those stylistic constraints. Third, human data is cost-nonlinear: as task complexity increases, the expertise required to produce valid demonstrations grows, and the pool of qualified annotators shrinks — making the cost per demonstration rise steeply for the very tasks where fine-tuning would be most valuable.

This problem matters for two converging reasons. On the practical deployment side, organizations want to specialize large pretrained models for specific high-value tasks (medical diagnosis, legal reasoning, competitive programming), but the human expertise to generate sufficient fine-tuning data for these tasks is scarce and expensive. On the research side, the field is confronting an uncomfortable scaling dynamic: pretraining continues to benefit from larger models and more data (Chinchilla scaling laws), but fine-tuning — the step that makes these models useful for specific applications — is throttled by a fundamentally different, non-scalable resource (human time and expertise). This asymmetry creates a scaling mismatch where pretraining capabilities grow faster than our ability to harness them for specialized tasks.

The paper's response to this mismatch is to investigate whether model-generated synthetic data, filtered by an automated scalar feedback signal (e.g., answer correctness for math, test case passage for code), can substitute for — and potentially outperform — human-written data in the fine-tuning pipeline. This is not merely a cost-saving argument. The implicit claim is more ambitious: that model-generated data may be qualitatively better for fine-tuning than human data because it is more "in-distribution" relative to the model's own generation patterns, and because the model can generate multiple diverse correct solutions per problem rather than the single solution a human annotator typically provides.

The Broader Landscape: Why Self-Training for LLMs Is Not a Solved Problem

Before this paper, there was already a substantial literature on training language models with self-generated data. However, as the paper's literature review (Section 4) makes clear, this prior work concentrated on two regimes that left a significant gap:

1. Small models, simpler tasks. Methods like Iterative Maximum Likelihood (IML; Agarwal et al., 2019; Liang et al., 2016; Wu et al., 2016), Self-Taught Reasoner (STaR; Zelikman et al., 2022), and early rejection sampling approaches demonstrated that self-training works for semantic parsing, machine translation, and elementary math reasoning (GSM8K) — but typically with models in the sub-10B parameter range. The paper is explicit about this limitation:

"prior works primarily applied training with self-generated data to relatively small language models (up to 7B parameters), with limited scalability observed for larger models (Yuan et al., 2023)"

This is not a trivial gap. Scaling behavior is not guaranteed to be monotonic or even positive — a method that works at 7B parameters may degrade, saturate, or exhibit qualitatively different dynamics at 100B+ parameters. Yuan et al. (2023), studying rejection sampling fine-tuning (RFT) on GSM8K, observed exactly this: performance gains from self-generated data diminished as model capacity increased, suggesting that larger models might not benefit from self-training in the same way smaller ones do. This finding — that self-training might be most useful for weaker models and less useful for stronger ones — would, if generally true, severely limit the practical relevance of self-training for state-of-the-art LLMs.

2. Single-iteration or tightly-coupled approaches. Many prior methods either (a) performed only a single round of generation and fine-tuning (Rejection Sampling Fine-Tuning / RFT; Yuan et al., 2023), (b) tightly coupled data collection and policy updates in an online loop (IML; Agarwal et al., 2019), or (c) used greedy decoding or rationalization rather than temperature sampling for exploration (STaR; Zelikman et al., 2022). Each of these design choices carries limitations:

  • Single iteration (RFT): The model never gets to learn from its own improved outputs. After one round of fine-tuning, the model is better than the base model, but its improved outputs are never harvested for a second training round. This leaves potential gains on the table, though whether multiple iterations help or overfit is an empirical question the paper investigates directly.

  • Online coupling (IML): Data collection and policy optimization happen over mini-batches, meaning the policy changes continuously and the data distribution shifts with it. This is computationally expensive for large models (every mini-batch requires new generations) and can cause the model to diverge significantly from its pretrained distribution — what the paper calls "task-specific overfitting, where the model performs well on the target task but loses its ability to generalize to other tasks or domains." For large-scale LLMs, this is a serious concern: you don't want your math-specialized model to forget how to reason about coding, or vice versa.

  • Greedy/constrained generation (STaR): Using greedy decoding means only one solution per problem is generated and potentially used. This eliminates the exploration benefit that comes from sampling diverse solutions, many of which may be correct but reason differently. STaR's rationalization mechanism — providing the model with the correct answer and asking it to generate a reasoning chain that leads to that answer — introduces a different problem: the model can produce solutions that reach the correct answer through incorrect reasoning (false positives), contaminating the training data.

3. No systematic comparison of self-generated vs. human-generated data at scale. Perhaps the most important gap the paper identifies is empirical, not methodological. Prior work had shown that self-training works — models improve when fine-tuned on self-generated data. But the crucial question for the field is: does self-training work better than fine-tuning on human data? If self-training merely matches human-data SFT at lower cost, that's useful. If it substantially outperforms human-data SFT, that fundamentally changes how practitioners should approach fine-tuning. The paper frames this as an open question for complex problem-solving domains, noting the "less explored" nature of competition-level math and code generation for self-training research.

Where Existing Self-Training Approaches Fall Short

The paper identifies specific failure modes and limitations in prior self-training approaches across several axes:

Scaling with model size is not guaranteed. Yuan et al. (2023) demonstrated that RFT on GSM8K showed "diminishing returns from model-generated data... when scaling model capacity." This is a direct empirical counterexample to the intuitive expectation that better models produce better self-training data. If larger models generate higher-quality solutions (more correct, more diverse), then self-training should yield larger gains. The fact that it didn't in the GSM8K setting suggests either (a) GSM8K is too easy — even small models saturate the benefit — or (b) there is a methodological issue with how self-training data is collected and used. The paper engages with this directly by testing on harder benchmarks (MATH, APPS) and demonstrating the opposite trend: larger models benefit more from ReST^EM. This is a critical empirical correction to the emerging narrative that self-training doesn't scale.

Task-specific overfitting from online methods. IML and related approaches that continuously update the policy on self-generated data risk catastrophic forgetting of broader capabilities. The model becomes a narrow expert on the training task but degrades on everything else. This is particularly problematic for LLMs, which are valued precisely because of their generality. A math-specialized model that can no longer write code or reason about everyday situations is a failed outcome unless the deployment context is extremely narrow. The paper notes this explicitly:

"the learned policy can significantly diverge from the initial pretrained model, which can manifest as task-specific overfitting, where the model performs well on the target task but loses its ability to generalize to other tasks or domains."

The computational cost of online coupling — generating fresh samples after every policy update — is also prohibitive for models with tens or hundreds of billions of parameters.

Rationalization introduces false positives. STaR's approach of providing the correct answer as a hint and asking the model to generate reasoning for it is intuitive: if the model can't solve a problem end-to-end, maybe it can reason backward from the answer. However, the paper reports that this "leads to substantial increase in false positive solutions that result in correct answer but with incorrect reasoning." This is a subtle but important point: for self-training to work, the training data must contain not just correct answers but correct reasoning processes. If the model learns from solutions that coincidentally reach the right answer through flawed logic, it internalizes those flawed patterns. This is particularly dangerous for math reasoning, where the reasoning chain — not just the final answer — is what generalizes to new problems.

Single-iteration approaches leave gains unrealized. RFT, as the simplest self-training baseline, performs one round of generation and fine-tuning. While this is computationally cheap and easy to implement, it raises an obvious question: if the fine-tuned model is now better than the base model, shouldn't its own outputs be better training data than the base model's outputs? Multiple iterations allow the model to bootstrap — generating from progressively stronger policies, filtering with the same reward function, and training on progressively higher-quality data. Whether this actually works or leads to overfitting is an empirical question, but single-iteration approaches don't even test it.

How ReST^EM Positions Itself

The paper positions ReST^EM as a simple, scalable, theoretically grounded self-training algorithm that addresses the limitations above through three design choices:

Decoupling data generation from policy optimization (EM framework). Drawing on the expectation-maximization formulation for RL (Dayan and Hinton, 1997; Peters and Schaal, 2007), ReST^EM separates the E-step (generate samples from a fixed policy, filter with reward) from the M-step (fine-tune on the filtered dataset). This decoupling has two major advantages. First, it is computationally efficient: all generation happens in a batch phase before any training, so the expensive forward passes through the large model are not interleaved with gradient updates. Second, it provides training stability: the data distribution is stationary within each iteration (since the generating policy is fixed), which simplifies optimization and avoids the distribution-shift pathologies of online methods.

The paper formalizes this in Section 3, showing that with non-negative rewards and the identity function for ff, the M-step objective becomes a reward-weighted maximum likelihood estimation:

θt+1:=arg⁡max⁡θEx∼D[Ey∼pθt(y∣x)[r(x,y)log⁡pθ(y∣x)]]\theta_{t+1} := \arg\max_\theta \mathbb{E}_{x \sim \mathcal{D}} \left[ \mathbb{E}_{\mathbf{y} \sim p_{\theta_t}(\mathbf{y}|\mathbf{x})} \left[ r(\mathbf{x}, \mathbf{y}) \log p_\theta(\mathbf{y} | \mathbf{x}) \right] \right]

For binary rewards (correct/incorrect), this reduces to standard SFT on the subset of generated solutions that pass the correctness check — the simplest possible instantiation of the framework.

Always fine-tuning from the base model. Unlike the original ReST (Gulcehre et al., 2023), which fine-tunes the model from the previous iteration (creating a chain: base → model₁ → model₂ → model₃), ReST^EM always starts each M-step from the original pretrained model. This is a seemingly minor implementation detail with major consequences: it prevents the model from drifting too far from its pretrained distribution across iterations, preserving generalization to held-out tasks. The paper demonstrates this explicitly in Figure 7, where ReST^EM matches ReST on the training task (APPS) but substantially outperforms it on transfer to HumanEval. This design choice is a direct response to the task-specific overfitting concerns.

Temperature sampling for diverse exploration. Unlike STaR's greedy decoding or rationalization, ReST^EM uses top-K sampling (K=40, temperature 0.7) during the E-step to generate multiple diverse solutions per problem. This is critical: for self-training to improve pass@1 performance, the training data must contain correct solutions that the model would not have generated under greedy decoding. If greedy decoding always produces the correct answer already, there's nothing new for the model to learn. Temperature sampling surfaces alternative reasoning paths that are correct but not maximally probable under the current policy, and fine-tuning on these paths shifts the probability mass toward them, improving greedy pass@1.

Cut-off threshold for per-problem solution count. The paper introduces a practical refinement: a maximum of 10 correct solutions per problem are included in the fine-tuning dataset, even if more than 10 are generated. This prevents the dataset from being dominated by easy problems — which will have many more correct solutions — and ensures diversity in the training data. Without this threshold, the model would see an overwhelming number of easy-problem solutions and relatively few hard-problem solutions, potentially skewing its learning toward easy patterns at the expense of harder ones.

The Gap This Paper Fills

Synthesizing the above: prior to this work, the field had evidence that self-training could improve small LMs on simple reasoning tasks (GSM8K, semantic parsing), but did not know whether these benefits would transfer to (a) large models (100B+ parameters), (b) complex reasoning tasks (competition math, code generation), or (c) multi-iteration regimes where the model learns from its own improved outputs. Moreover, prior work had not established whether self-generated data could outperform human data for fine-tuning on the same tasks — a question with enormous practical implications for how fine-tuning pipelines should be constructed.

The paper fills this gap by providing the first systematic demonstration that a simple EM-based self-training loop — with careful design choices around base-model reset, temperature sampling, and per-problem solution caps — yields larger performance gains than human-data SFT on MATH and APPS, that these gains scale favorably with model size (contradicting the RFT scaling results on GSM8K), and that multiple iterations provide additional benefit until overfitting sets in on the smaller APPS dataset. The theoretical grounding in the EM-for-RL framework provides a principled explanation for why decoupling generation from optimization works, and the ablation studies (on dataset size, number of iterations, model-generated vs. human data, ReST vs. ReST^EM) establish the practical boundaries of the approach.

Real-World Significance

The paper's findings, if generalizable beyond MATH and APPS, suggest a fundamental shift in how organizations should approach fine-tuning for problem-solving tasks. Instead of investing heavily in human annotation — recruiting subject-matter experts, designing annotation protocols, managing quality control — practitioners can invest in (a) collecting a set of input problems (which is typically much cheaper than collecting full solution demonstrations) and (b) defining an automated reward function (which, for math, code, and many other domains, can be done with unit tests or answer matching). The model then generates its own training data at scale, filtered by the reward function, and iteratively improves.

This decouples the cost of fine-tuning from the cost of human expertise, potentially enabling specialization for niche domains where human annotation would be prohibitively expensive. It also opens the door to continuous improvement loops: as the base pretrained model improves (through better pretraining recipes), the same set of problems and reward functions can be reused to fine-tune the new model without any additional human annotation cost. The fine-tuning data becomes a function of the model and the reward, not a static human-curated artifact.

However, the paper is careful to note (Section 6) that this vision requires a moderately-sized set of input problems (at least ~1,000 based on Figure 8 left) and a reliable automated reward function — both of which require human effort to establish initially. The method reduces dependence on human data but does not eliminate it entirely.

3. Technical Approach

3.1 Reader Orientation

ReST^EM is a self-training algorithm that turns a language model into a data-generation engine for its own improvement: the model repeatedly generates multiple candidate solutions for each problem in a training set, keeps only the solutions that pass an automated correctness check, and fine-tunes itself on these surviving high-quality samples — then repeats the cycle. The core idea is that the model's own successful outputs, when diverse enough and filtered by a reliable reward signal, constitute better training data than the single human-written solution typically available per problem, because they are more in-distribution (matching the model's generation patterns), more numerous (one problem can generate multiple correct solution paths), and increasingly higher-quality as the model improves across iterations.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that alternate in a loop:

  1. A pretrained language model (PaLM 2-S, PaLM 2-S*, or PaLM 2-L) parameterized by $\theta$ — serves as both the solution generator and the student being fine-tuned. During generation (E-step), it samples multiple candidate outputs per input problem using temperature sampling. During improvement (M-step), it is fine-tuned via standard supervised learning on the filtered outputs.

  2. A training dataset of input problems (MATH with 7,500 questions, APPS Introductory with 2,342 questions) — provides the input contexts $\mathbf{x}$ for which the model generates solutions. Only the problem statements are needed from this dataset; the human-written reference solutions are used solely for baseline comparison, not for ReST^EM itself.

  3. A binary reward function $r(\mathbf{x}, \mathbf{y}) \in \{0, 1\}$ — an automated, external correctness checker. For MATH, it compares the model's extracted final answer against the ground-truth answer using string matching. For APPS, it executes the generated code against test cases and rewards solutions that pass all tests. This reward function is the only supervision signal ReST^EM requires.

  4. A fine-tuning loop (the Improve / M-step) — takes the filtered dataset from the E-step (input problems paired with correct model-generated solutions), resets to the original pretrained base model, and performs supervised fine-tuning by minimizing the negative log-likelihood of the correct solution tokens given the problem and a few-shot prompt.

Information flow: For each iteration $i = 1, \ldots, I$ (where $I$ is typically 2–3), the system (a) samples $N$ solutions per problem from the current policy using temperature sampling (E-step), (b) scores each solution with the binary reward, (c) filters to keep only correct solutions, up to a maximum of 10 per problem, (d) fine-tunes the base pretrained model on this filtered dataset (M-step), and (e) uses the fine-tuned model as the generating policy for the next E-step. A held-out validation set monitors reward improvement during the M-step to determine when to stop fine-tuning.

3.3 Roadmap for the Deep Dive

  • First, the formal EM-for-RL framework (Equation 2) that underpins ReST^EM — what the ELBO objective represents, how the E-step and M-step are derived, and why decoupling generation from optimization matters for large models. This provides the theoretical justification for the algorithm's design.

  • Second, the E-step mechanics — how solutions are generated (temperature, top-K, number of samples per problem), how the binary reward is computed, how the per-problem cutoff threshold prevents dataset imbalance, and what design choices distinguish ReST^EM's E-step from alternatives like greedy decoding (STaR) or rationalization.

  • Third, the M-step mechanics — the reward-weighted objective, why it collapses to standard SFT on correct solutions under binary rewards, why the base model is always the starting point for each M-step (not the previous iteration's model), and how the validation set controls stopping.

  • Fourth, the multi-iteration loop — what changes between iterations, why multiple iterations help (the model generates from progressively stronger policies, producing higher-quality training data each round), and the mechanisms that prevent runaway distribution shift (always resetting to base model, per-problem cap).

  • Fifth, the key design choices distilled — temperature sampling over greedy decoding, base-model reset over iterative fine-tuning, per-problem cutoff over using all correct solutions, and no human-data augmentation — each with the explicit justification the paper provides.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological paper with strong empirical validation whose core idea is that iteratively generating, filtering, and training on a language model's own successful outputs — using an EM-inspired decoupling of data collection and policy optimization — yields better specialized performance than fine-tuning on human data, and that this approach scales favorably with model size when applied to sufficiently challenging tasks.

The Expectation-Maximization Framework for RL with Language Models

The paper grounds ReST^EM in the EM-for-RL formulation originally proposed by Dayan and Hinton (1997) and later refined by Peters and Schaal (2007) for reward-weighted regression. This framework treats the RL problem — maximizing expected reward — as a maximum-likelihood estimation problem over a latent optimality variable, which enables the clean decoupling that makes ReST^EM computationally tractable for large models.

Step 1: Defining the optimality variable. The paper introduces a binary random variable $O$ (for "optimality") such that $O = 1$ indicates that a solution $\mathbf{y}$ to a problem $\mathbf{x}$ has achieved high reward. Specifically, the probability that a solution is optimal is proportional to some non-decreasing function $f$ of the reward:

$$p(O = 1 \mid \mathbf{x}, \mathbf{y}) \propto f(r(\mathbf{x}, \mathbf{y}))$$

where $r(\mathbf{x}, \mathbf{y})$ is the scalar reward (0 or 1 in the paper's setting), $f: \mathbb{R} \to \mathbb{R}^+$ is any non-decreasing non-negative function (e.g., the identity function when rewards are non-negative), and the proportionality constant ensures this is a valid probability distribution over $O$.

What this defines: a probabilistic model where higher reward implies higher probability of optimality. For binary rewards with $f$ as the identity, this becomes a simple threshold: correct solutions ($r = 1$) have $p(O = 1) \propto 1$ and incorrect solutions ($r = 0$) have $p(O = 1) = 0$. The optimality variable $O$ is a latent bridge between the observable reward and the unobservable "goodness" of a solution.

Why this form: introducing a probabilistic optimality variable converts the RL problem (maximize reward) into a probabilistic inference problem (maximize the likelihood of observing optimality). This probabilistic framing is what enables the application of expectation-maximization — a workhorse algorithm for latent-variable models — to reinforcement learning.

Step 2: The marginal likelihood objective. The goal becomes maximizing the log-probability of observing $O = 1$ (i.e., generating a high-reward solution) given a problem $\mathbf{x}$:

$$\log p(O = 1 \mid \mathbf{x}) := \log \sum_{\mathbf{y}} p_\theta(\mathbf{y} \mid \mathbf{x}) \, p(O = 1 \mid \mathbf{x}, \mathbf{y})$$

where $p_\theta(\mathbf{y} \mid \mathbf{x})$ is the language model's distribution over output sequences $\mathbf{y}$ given input $\mathbf{x}$, parameterized by $\theta$, and the sum is over all possible sequences.

What this equation says: the probability that the model produces an optimal solution for problem $\mathbf{x}$ is the sum, over all possible output sequences, of the probability that the model generates that sequence multiplied by the probability that sequence is optimal. In operational terms: you generate every possible answer, weight each by how likely it is to be optimal, and sum. The model parameters $\theta$ appear only in the generation term $p_\theta(\mathbf{y} \mid \mathbf{x})$, so maximizing this marginal likelihood with respect to $\theta$ pushes the model to assign higher probability to sequences that are likely to be optimal.

Why this objective: it is the natural probabilistic formulation of "make the model produce good outputs." Unlike the standard RL objective, which takes an expectation over the model's own distribution (making it an on-policy objective that changes as the model changes), this marginal likelihood formulation separates the data-generating process from the optimization target — a separation that EM will exploit.

Step 3: The ELBO and the EM decomposition. The sum over all possible sequences $\mathbf{y}$ is intractable (there are exponentially many). The standard variational inference trick is to introduce a variational distribution $q(\mathbf{y} \mid \mathbf{x})$ — a tractable distribution over sequences — and maximize the Evidence Lower Bound (ELBO):

$$\log p(O = 1 \mid \mathbf{x}) \geq \mathbb{E}_{q(\mathbf{y} \mid \mathbf{x})} \left[ \log \frac{p(O = 1 \mid \mathbf{x}, \mathbf{y}) \, p_\theta(\mathbf{y} \mid \mathbf{x})}{q(\mathbf{y} \mid \mathbf{x})} \right] =: L(p_\theta, q)$$

where $L(p_\theta, q)$ is the ELBO, and the inequality follows from Jensen's inequality applied to the concave log function.

What this computes: the right-hand side is the expectation under $q$ of the log-ratio between the joint probability $p(O = 1 \mid \mathbf{x}, \mathbf{y}) p_\theta(\mathbf{y} \mid \mathbf{x})$ (how likely the sequence is to be both generated and optimal) and the variational distribution $q(\mathbf{y} \mid \mathbf{x})$ (how likely the sequence is under our approximate sampling distribution). This is a lower bound on the true log-marginal $\log p(O = 1 \mid \mathbf{x})$ — maximizing the ELBO improves our guarantee on the true objective.

Why this form: the ELBO decomposes into two interpretable terms (shown in the paper's Equation 2):

$$L(p_\theta, q) = \mathbb{E}_{q(\mathbf{y}|\mathbf{x})} [\log p(O = 1 \mid \mathbf{x}, \mathbf{y})] - \text{KL}[q(\mathbf{y} \mid \mathbf{x}) \,||\, p_\theta(\mathbf{y} \mid \mathbf{x})]$$

The first term $\mathbb{E}_q[\log p(O = 1)]$ encourages $q$ to put mass on sequences with high optimality probability (exploitation: sample where rewards are high). The second term $-\text{KL}[q \,||\, p_\theta]$ penalizes $q$ for deviating from the model's current distribution $p_\theta$ (regularization: don't drift too far from what the model can actually generate). This explore-exploit tradeoff is baked into the objective structure.

Step 4: The E-step and M-step explicitly. The EM algorithm for this ELBO alternates between two optimization subproblems at each iteration $t$:

E-step (Expectation): Fix the model parameters $\theta_t$ and solve for the optimal variational distribution $q_{t+1}$:

$$q_{t+1} = \arg\max_q L(p_{\theta_t}, q)$$

The paper shows this has a closed-form solution: $q_{t+1}(\mathbf{y} \mid \mathbf{x}) \propto p(O = 1 \mid \mathbf{x}, \mathbf{y}) \, p_{\theta_t}(\mathbf{y} \mid \mathbf{x})$. In words: the optimal $q$ is the model's current distribution $p_{\theta_t}$ reweighted by the probability of optimality. Sequences that are both likely under the current model AND likely to be optimal get more weight.

What happens physically: you sample many sequences from the model $p_{\theta_t}$, score each with the reward function to compute $p(O = 1 \mid \mathbf{x}, \mathbf{y})$, and retain them with weights proportional to this optimality probability. For binary rewards with the identity function, this means you simply keep the correct solutions (reward = 1) and discard the incorrect ones (reward = 0). The E-step produces a dataset of problems paired with weighted (or filtered) model-generated solutions.

M-step (Maximization): Fix the variational distribution $q_{t+1}$ (i.e., fix the filtered dataset from the E-step) and optimize the model parameters:

$$\theta_{t+1} = \arg\max_\theta L(p_\theta, q_{t+1}) = \arg\min_\theta \sum_{\mathbf{y}} -q_{t+1}(\mathbf{y} \mid \mathbf{x}) \log p_\theta(\mathbf{y} \mid \mathbf{x})$$

What happens physically: this is a reward-weighted supervised fine-tuning step. The model $p_\theta$ is trained to maximize the log-likelihood of sequences in the filtered dataset, where each sequence is weighted by its reward-derived optimality probability. For binary rewards with the identity function, this simplifies further: you perform standard supervised fine-tuning on the set of correct model-generated solutions, with uniform weight (since all correct solutions have the same $p(O = 1)$ weight, and incorrect solutions have weight 0 and are excluded entirely).

Why this decomposition works for large models: the EM framework decouples data collection (E-step) from policy optimization (M-step). During the E-step, the generating policy $p_{\theta_t}$ is frozen — no gradients flow through the generation process. The expensive forward passes (generating $N$ solutions per problem for thousands of problems) happen in a batch, completely offline from training. During the M-step, the training data is fixed (it doesn't change as the model updates), making optimization straightforward with standard supervised learning infrastructure. This contrasts with online RL methods like policy gradients, which require interleaved sampling and gradient updates — each requiring forward passes through a model whose parameters are continuously changing, making the computation inherently sequential and hard to scale to 100B+ parameter models.

The monotonic improvement guarantee. The paper notes that alternating E-steps and M-steps guarantees $L(p_{\theta_{t+1}}, q_{t+1}) \geq L(p_{\theta_t}, q_{t+1}) \geq L(p_{\theta_t}, q_t)$ — the ELBO never decreases. The first inequality holds because the M-step maximizes $L$ with respect to $\theta$ (so the new parameters are at least as good), and the second holds because the E-step maximizes $L$ with respect to $q$. This provides a theoretical assurance that the process does not degrade — though it does not guarantee convergence to the global optimum, and in practice the finite-sample approximations and the reset-to-base-model design choice mean the guarantee is approximate.

The E-Step: Generating and Filtering Solutions

The E-step is the data-generation phase where the language model produces candidate solutions and the reward function filters them into a training dataset. The paper implements this as a batch sampling procedure with specific hyperparameter choices that balance diversity, quality, and computational cost.

Sampling configuration. For each problem $\mathbf{x}$ in the training set $\mathcal{D}$, the current policy $p_{\theta}$ generates $N$ candidate solutions:

  • $N = 32$ for the MATH dataset.
  • $N = 64$ for the APPS dataset.

These values represent a deliberate tradeoff. Too few samples per problem, and the model may not generate any correct solutions for hard problems, yielding no training signal for those problems. Too many samples, and the computational cost of the E-step dominates the pipeline, leaving less budget for iterations or downstream training. The paper uses more samples for APPS than MATH (64 vs. 32), likely because code generation is higher-variance — a correct program must satisfy all test cases, not just produce a matching final answer — so more sampling is needed to reliably surface correct solutions.

Decoding strategy. Solutions are generated using top-K sampling with $K = 40$ and a temperature of 0.7. Specifically, at each token generation step, the model's logits are divided by the temperature $\gamma = 0.7$ before softmax, then the probability distribution is truncated to the top 40 tokens, with the remaining tokens assigned zero probability. The token is then sampled from this truncated distribution.

The choice of temperature 0.7 and top-K 40 over alternatives like greedy decoding (temperature 0, or argmax) is motivated by the need for exploration. If the model were to use greedy decoding, it would produce only the single most likely solution per problem. If that solution happens to be correct, the model learns nothing new (it already assigns high probability to the correct answer). If it is incorrect, the problem contributes no training data at all. Temperature sampling surfaces alternative solutions that may be correct but not maximally probable — the model's "second choice" might be right even when its first choice is wrong. Training on these alternative correct solutions shifts probability mass toward them, improving the model's greedy (pass@1) performance in subsequent iterations.

The specific values — temperature 0.7 and top-K 40 — are standard defaults in the LLM literature for balancing diversity and coherence. Temperature 0.7 is warm enough to produce meaningful variation without degenerating into random output; top-K 40 is broad enough to include diverse options without including very low-probability (and likely nonsensical) tokens.

Few-shot prompting. The model is conditioned using a few-shot prompt that includes examples of the task format. For MATH, the prompt includes step-by-step solutions to example math problems. For APPS, the prompt includes example programs that solve similar coding challenges. This in-context learning steers the model toward producing solutions in the expected format (reasoning chains for math, executable code for APPS) without requiring task-specific fine-tuning before the first E-step. The paper does not specify the exact number of few-shot examples, but the practice is standard for both domains.

Reward computation. Each generated solution $\mathbf{y}$ is evaluated by the binary reward function $r(\mathbf{x}, \mathbf{y})$:

  • MATH: The model's solution $\mathbf{y}$ contains both a reasoning chain and a final answer. The final answer is extracted (using a grading function that parses the output and identifies the answer string) and compared against the ground-truth answer from the MATH dataset. If they match, $r = 1$; otherwise, $r = 0$. The paper uses the grading function from the original MATH benchmark release (Hendrycks et al., 2021b) for answer extraction.

  • APPS: The model's solution $\mathbf{y}$ is a program (or function). This program is executed against the test cases provided in the APPS dataset. If the program passes all test cases, $r = 1$; otherwise (including compilation errors, runtime errors, or wrong outputs for any test case), $r = 0$. APPS test cases include both example inputs (visible in the problem description) and hidden test cases (used only for evaluation). The paper evaluates against all test cases to determine correctness.

A critical property of both reward functions: they are deterministic (the same solution always gets the same reward) and automated (no human judgment is needed to evaluate a solution). This is what makes ReST^EM scalable — the E-step can generate and evaluate hundreds of thousands of solutions without human intervention.

Filtering and dataset construction. After generating $N$ solutions per problem and scoring them, the E-step constructs the dataset $\mathcal{D}_i$ for the $i$-th iteration as follows:

  1. Keep only solutions with $r(\mathbf{x}, \mathbf{y}) = 1$ (correct solutions). Incorrect solutions are discarded — they contribute zero weight in the reward-weighted objective.

  2. Per-problem cutoff: For each problem, keep at most 10 correct solutions. If the model generates more than 10 correct solutions for a problem, randomly select 10 to include in $\mathcal{D}_i$. If it generates fewer than 10, include all correct solutions.

The per-problem cutoff is a critical practical refinement that the paper explicitly justifies. Without it, the training dataset would be highly imbalanced: easy problems (for which the model generates many correct solutions) would dominate the dataset, while hard problems (for which the model generates few or zero correct solutions) would be underrepresented. This imbalance would skew the M-step fine-tuning toward patterns that are already easy for the model, providing little learning signal for improvement. The cutoff of 10 ensures that each problem contributes at most 10 training examples, regardless of how easy it is, which means hard problems — once the model starts generating even a few correct solutions for them — are not drowned out.

The value of 10 is not ablated in the paper, but it represents a balance: large enough to provide diversity (multiple reasoning paths per problem), small enough to prevent easy-problem dominance. Zelikman et al. (2022) used a similar cutoff in STaR, suggesting it is an established heuristic in the self-training literature.

Dataset size after filtering. The paper does not report the exact size of $\mathcal{D}_i$ after filtering, but we can reason about it. With 7,500 MATH training problems and $N = 32$ samples per problem, the raw E-step generates 240,000 candidate solutions. If the base model's pass@1 on MATH is approximately 20% (a reasonable estimate based on the figures), then roughly 48,000 solutions would be correct before the per-problem cutoff. The cutoff of 10 per problem would further reduce this — easy problems might contribute 10 solutions each, medium problems might contribute 3–5, and hard problems might contribute 0–2. The final dataset likely contains on the order of 15,000–30,000 (problem, solution) pairs, representing correct solutions for a sizeable fraction of the training problems.

No human-data augmentation. Unlike the original ReST algorithm (Gulcehre et al., 2023), which augmented the model-generated dataset with human-written solutions, ReST^EM uses only model-generated data in $\mathcal{D}_i$. The paper argues that human data "may not always be optimal for learning or it might not be easily available." This is a philosophical choice: ReST^EM is designed to work in settings where human data is scarce, expensive, or nonexistent, so including it would obscure the evaluation of whether self-generated data alone suffices.

Multiple iterations and the improving E-step. After the first iteration, the E-step uses the fine-tuned model from the previous M-step (not the original base model) as the generating policy. This means that in iteration 2, the model generating solutions is already better at the task than the base model — it has been fine-tuned on correct solutions from iteration 1. Consequently, the solutions generated in iteration 2's E-step should be higher quality on average: more solutions are correct, and the correct solutions may involve more sophisticated reasoning. The E-step thus produces a progressively better training dataset across iterations, which is the mechanism by which multiple iterations provide additional gains beyond a single round of self-training.

The M-Step: Reward-Weighted Fine-Tuning from Base Model

The M-step takes the filtered dataset $\mathcal{D}_i$ produced by the E-step and uses it to fine-tune the language model. The objective, derived from the EM framework with binary rewards and the identity function $f(r) = r$, is:

$$\theta_{t+1} := \arg\max_\theta \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \mathcal{D}_i} \left[ r(\mathbf{x}, \mathbf{y}) \, \log p_\theta(\mathbf{y} \mid \mathbf{x}) \right]$$

where $\mathcal{D}_i$ is the dataset from the $i$-th E-step, $r(\mathbf{x}, \mathbf{y}) \in \{0, 1\}$ is the binary reward, and $\log p_\theta(\mathbf{y} \mid \mathbf{x})$ is the log-likelihood of the solution sequence under the model parameters $\theta$.

What this objective computes: for each (problem, solution) pair in $\mathcal{D}_i$, multiply the reward by the log-probability of the solution, and sum (or average) over the dataset. Since the dataset only contains correct solutions ($r = 1$) after filtering, and incorrect solutions ($r = 0$) are excluded entirely, this reduces to standard supervised fine-tuning on the correct model-generated solutions:

$$\theta_{t+1} := \arg\max_\theta \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \mathcal{D}_i} \left[ \log p_\theta(\mathbf{y} \mid \mathbf{x}) \right]$$

In practice, this is implemented as minimizing the negative log-likelihood loss, which is the cross-entropy between the model's predicted token distribution and the one-hot target tokens from the correct solution:

$$\mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \mathcal{D}_i} \left[ \sum_{t=1}^{T} \log p_\theta(y_t \mid \mathbf{y}_{<t}, \mathbf{x}) \right]$$

where $y_t$ is the $t$-th token of the correct solution $\mathbf{y}$, $\mathbf{y}_{<t}$ are the preceding tokens, and $T$ is the solution length.

Why the objective simplifies: the binary reward (0 or 1) acts as a hard filter rather than a soft weight. Incorrect solutions contribute zero weight (they are effectively removed from the expectation), and correct solutions contribute unit weight. This is a special case of reward-weighted regression where the reward function is an indicator. If the reward were real-valued (e.g., a continuous score between 0 and 1), the objective would weight solutions proportionally to their reward, giving more influence to higher-quality but imperfect solutions. The paper's setting — binary correctness — is the simplest instantiation of the framework, and it avoids the question of how to calibrate continuous reward weights.

The critical design choice: always fine-tune from the base model. This is arguably the most important implementation detail distinguishing ReST^EM from both the original ReST (Gulcehre et al., 2023) and from naive iterative fine-tuning. At each M-step, regardless of how many iterations have been completed, the model is initialized from the original pretrained base model — not from the model produced by the previous iteration's M-step. Table 1 in the paper makes this explicit: "Finetunes from base model in each iteration: ReST^EM (yes), ReST (no), STaR (yes), RFT (N/A)."

What this means operationally: in iteration 1, the base model $p_{\theta_0}$ generates solutions, and a fine-tuned model $p_{\theta_1}$ is produced by training the base model on those solutions. In iteration 2, $p_{\theta_1}$ generates solutions (a better generator), and the base model $p_{\theta_0}$ is trained again, this time on the solutions from $p_{\theta_1}$, producing $p_{\theta_2}$. The model $p_{\theta_1}$ is discarded — it is used only as a data generator for the next E-step. In iteration 3, $p_{\theta_2}$ generates solutions, and the base model $p_{\theta_0}$ is trained a third time on these solutions.

Why this design choice matters — the transfer performance argument. The paper demonstrates in Figure 7 that ReST^EM matches ReST on the training task (APPS) but substantially outperforms ReST on transfer to HumanEval (coding problems the model was not fine-tuned on). The interpretation is that iterative fine-tuning (ReST's approach) compounds distribution shift: each M-step fine-tunes an already-fine-tuned model, causing it to drift progressively further from the original pretrained distribution. While this may not hurt (and may slightly help) performance on the training task, it degrades general capabilities — the model becomes a narrow specialist. By resetting to the base model each iteration, ReST^EM ensures that the fine-tuned model is always only one fine-tuning step away from the pretrained distribution, regardless of how many iterations have been run. This acts as an implicit regularizer against catastrophic forgetting.

A subtler benefit is that the data quality improves across iterations without the model quality degrading on holdout data. The generating model $p_{\theta_i}$ gets progressively better (since it was fine-tuned on better data), producing better training data each round. The training target is always the base model, which has strong general capabilities. The resulting fine-tuned model inherits the improved data quality without inheriting the accumulated distribution shift that would come from chained fine-tuning.

The loss is computed only on solution tokens, not on the prompt. The paper specifies: "We only apply the next token prediction loss (Equation 1) on the targets." The input to the model during training is the few-shot prompt concatenated with the problem statement (the "context"), followed by the model-generated solution (the "target"). The loss is computed only on the tokens of the solution — the model is not penalized for its predictions of the prompt tokens. This is standard practice in supervised fine-tuning of language models: you want the model to learn the conditional distribution of solutions given problems, not to memorize the problem statements themselves.

Validation-based early stopping. The M-step uses a held-out validation set $\mathcal{D}_{\text{val}}$ to determine when to stop fine-tuning. The paper states: "while reward improves on $\mathcal{D}_{\text{val}}$ do / Optimise $\theta$ to maximize objective." This means that during the M-step, the model is periodically evaluated on the validation set (by generating solutions and checking their correctness), and training continues as long as validation reward is improving. Once validation reward plateaus or declines, training stops. This is a guard against overfitting to the training set — since $\mathcal{D}_i$ contains only correct solutions, the model could overfit by memorizing these specific solutions rather than learning generalizable problem-solving patterns. Early stopping on validation performance mitigates this risk.

Fine-tuning hyperparameters. The paper does not report the optimizer, learning rate, batch size, or number of epochs used for the M-step fine-tuning. This is a notable omission — the training configuration is a critical determinant of how much the model adapts to the self-generated data. Based on the PaLM 2 technical report (Google et al., 2023) and standard practices, the fine-tuning likely uses the AdamW optimizer with a learning rate in the range $10^{-5}$ to $10^{-4}$, a batch size of 32–128, and training for 1–3 epochs over the filtered dataset. The validation-based early stopping means the exact number of training steps varies per iteration and per task.

The Multi-Iteration Loop and Convergence Behavior

Algorithm 1 in the paper formalizes the ReST^EM loop with $I$ iterations, where each iteration consists of one Generate (E-step) and one Improve (M-step). The paper experiments with $I = 1, 2, 3$ for MATH and $I = 1, 2$ for APPS.

Iteration dynamics. At iteration 0 (the base model, before any ReST^EM), the model has some baseline pass@1 accuracy on the test set — approximately 35–37% for PaLM 2-L on MATH (reading from Figure 2) and approximately 18–20% for PaLM 2-S* on APPS (reading from Figure 3).

Iteration 1: The base model generates $N = 32$ (MATH) or $N = 64$ (APPS) solutions per training problem. Correct solutions are collected (up to 10 per problem). The base model is fine-tuned on this dataset, producing model $p_{\theta_1}$. Performance jumps substantially — for PaLM 2-L on MATH, from approximately 37% to roughly 41% (a gain of ~4 percentage points); for PaLM 2-L on APPS, from approximately 20% to roughly 26% (a gain of ~6 percentage points).

Iteration 2: Model $p_{\theta_1}$ (the fine-tuned model from iteration 1) is now the generator. Because $p_{\theta_1}$ is better than the base model at the task, it generates more correct solutions per problem, and the correct solutions it generates may be of higher quality. The base model is fine-tuned on this new, higher-quality dataset, producing model $p_{\theta_2}$. On MATH, performance improves further — to approximately 42% for PaLM 2-L, a smaller incremental gain than iteration 1 but still positive. On APPS, performance regresses — dropping to roughly 25%, below the iteration 1 peak.

Iteration 3 (MATH only): Model $p_{\theta_2}$ generates solutions, the base model is fine-tuned again, producing model $p_{\theta_3}$. Performance reaches approximately 42% (PaLM 2-L) — essentially flat or marginally improved over iteration 2.

Why multiple iterations help (when they do). The mechanism is bootstrapping: each fine-tuned model is a better solution generator than the base model, so each successive E-step produces a dataset with a higher fraction of correct solutions and potentially more sophisticated correct solutions. The base model, when fine-tuned on this improved data, learns from solutions it could not have generated itself. This is the "reinforcement" in reinforced self-training — the model's own improved outputs become the training signal for further improvement.

Why multiple iterations stop helping (and sometimes hurt). Figure 4 shows the train-test gap widening with iterations: training performance (pass@1 on the training set) continues to improve monotonically, but test performance plateaus (MATH) or declines (APPS). This is classic overfitting. The training set is finite — 7,500 problems for MATH, 2,342 for APPS — and each iteration reuses the same problems. The model increasingly memorizes problem-specific patterns rather than learning generalizable reasoning. APPS, with a training set one-third the size of MATH, overfits faster and more severely — performance regresses by the second iteration.

The paper also notes that on MATH, using 3× more data in a single iteration (equivalent sample budget to three iterations) underperforms three iterations, indicating that the iterative refinement of the data-generating policy — not just the total volume of data — contributes to the gains. A single E-step with the base model cannot produce the higher-quality solutions that iteration 2's E-step (using the fine-tuned model) can produce.

Key Design Choices and Their Justifications (Summary)

Temperature sampling (top-K = 40, temperature = 0.7) over greedy decoding. Greedy decoding produces only the single most probable solution per problem, providing no exploration. Temperature sampling surfaces diverse reasoning paths, including correct solutions that are not the model's top choice under greedy decoding. Training on these shifts probability mass toward them, improving pass@1. The paper explicitly contrasts this with STaR's greedy decoding and rationalization, noting that rationalization introduces false positives (correct answers via incorrect reasoning).

Always fine-tuning from the base model over iterative fine-tuning. Reset to the base model at each M-step prevents distribution drift and preserves generalization to held-out tasks, as demonstrated in Figure 7 (ReST^EM vs. ReST transfer to HumanEval). This is the key mechanism for avoiding task-specific overfitting.

Per-problem cap of 10 correct solutions over using all correct solutions. Prevents easy problems from dominating the training dataset, ensuring that hard problems — once they start yielding correct solutions — contribute proportionally to the learning signal. Maintains diversity in the fine-tuning data.

No human-data augmentation over mixed human-model datasets. Simplifies the pipeline, tests the pure self-training hypothesis, and avoids the complication of mixing out-of-distribution human solutions with in-distribution model-generated solutions.

Multiple iterations with validation-based early stopping. Captures the benefit of learning from progressively better model outputs while using validation performance as a guard against overfitting. The validation reward signal (not training loss) determines when the M-step should stop and when the overall iteration loop should terminate.

4. Key Insights and Innovations

Innovation 1: Model-Generated Data Can Be Better Than Human Data — Not Just a Cheaper Substitute

The most intellectually distinctive contribution of this paper is a demonstration that upends the default assumption about the relationship between human and synthetic training data. Before this work, the prevailing narrative — implicit in most of the self-training and distillation literature — was that model-generated data is a substitute of last resort: useful when human data is scarce or expensive, but fundamentally a degraded signal compared to expert-written demonstrations. The goal was to close the gap, not to surpass it. ReST^EM challenges this framing directly by showing that on MATH and APPS, fine-tuning on self-generated solutions outperforms fine-tuning on human-written solutions — not by a marginal amount, but by a gap that widens with model scale (Figures 2 and 3).

This finding is not merely about cost reduction. It suggests that model-generated data possesses qualitative properties that human data lacks for the purpose of fine-tuning. The paper hypothesizes that model-generated solutions are "more in-distribution compared to human-written solutions" (Section 5.3), and this hypothesis carries a deeper implication: when a language model generates a solution, it does so in its own "voice" — using the reasoning patterns, lexical choices, and structural conventions it naturally produces. Human solutions, by contrast, are an out-of-distribution target: they reflect human reasoning styles, human error patterns, and human organizational habits that may not align with how the model naturally processes the task. Fine-tuning on human data forces the model to imitate an alien style; fine-tuning on self-generated data reinforces and amplifies the model's own successful behaviors. The latter is arguably a more efficient learning signal because there is no "translation" cost — the model isn't learning to mimic an external style, it's learning to do more of what it already does well.

The paper strengthens this claim with an apples-to-apples comparison (Figure 6, left) that controls for the number of solutions per problem: even when limited to one model-generated solution per problem (ReST^*), matching the human-data scenario (one solution per problem), the model-generated data outperforms human data. This eliminates the confound that ReST^EM benefits from having multiple solutions per problem while SFT only has one. The fact that a single correct model-generated solution is a better training example than a single human-written solution is a striking result that demands explanation beyond "more data is better."

Prior work largely operated under the implicit assumption that human data was the gold standard. RFT (Yuan et al., 2023) showed that self-generated data could improve models on GSM8K but did not claim superiority over human data — and in fact observed diminishing returns with model scale, suggesting that self-generated data might be most useful for weaker models. STaR (Zelikman et al., 2022) used self-generated rationales but also incorporated rationalization (providing hints to the model) when generation alone failed, implicitly treating model-generated data as inferior and in need of human assistance. ReST^EM's demonstration that self-generated data surpasses human data — and that this advantage grows with model capability — reframes synthetic data from a fallback option to a potentially preferred training signal for capable models on tasks with verifiable rewards.

The significance of this reframing extends beyond the specific benchmarks. If model-generated data is genuinely superior for fine-tuning on any task where correctness can be automatically verified, then the bottleneck shifts from "how do we collect more human demonstrations?" to "how do we design better reward functions and generation strategies?" The expensive, slow, expertise-limited resource (human annotation) is replaced by a scalable, fast, model-driven resource (generation + automated verification). This is a fundamental shift in the economics of specialization for large language models.

Innovation 2: EM as a Organizing Principle, Not Just an Optimization Algorithm

The paper's second conceptual contribution is the elevation of the expectation-maximization framework from a technical detail to a design philosophy for self-training. Prior self-training methods — IML, RFT, STaR, RAFT — can all be retroactively described as implementing some form of E-step (generate) and M-step (train), but they were not designed from first principles within the EM-for-RL framework. The paper's formal derivation (Section 3) does more than provide theoretical window-dressing; it makes explicit why decoupling generation from optimization is beneficial, what objective is being optimized, and what guarantees (monotonic improvement of the ELBO) the procedure provides.

This theoretical grounding yields two practical insights that would be difficult to arrive at through pure empirical tuning. First, it explains why fine-tuning on self-generated correct solutions works at all: the M-step is maximizing a reward-weighted log-likelihood, which under binary rewards reduces to standard SFT on successful samples. This is not an ad-hoc trick — it is the natural consequence of applying EM to the marginal optimality likelihood. Second, it provides a principled account of why decoupling matters for scale: the EM framework separates the data-generating process (E-step, where the expensive forward passes happen on a frozen model) from the optimization process (M-step, where gradients are computed on a fixed dataset). This decoupling is what makes ReST^EM computationally tractable for 100B+ parameter models, in contrast to online methods like IML where sampling and gradient updates are interleaved, making the computation inherently sequential and expensive.

The paper also uses the EM framework to organize the field. Section 4 shows how Expert Iteration, STaR, RFT, IML, RWR, and RAFT can all be understood as special cases or variants within the EM-for-RL template, differing in how they implement the E-step (greedy decoding vs. sampling vs. search), how they handle the M-step (mini-batch vs. full-dataset, iterative vs. base-model reset), and how they define the reward function (binary vs. real-valued, explicit vs. implicit). This taxonomic contribution may be as valuable as the empirical results: it gives researchers a shared language and a conceptual framework for reasoning about self-training methods, rather than treating each as an isolated algorithmic recipe.

The significance here is not that EM is new — Dayan and Hinton published their formulation in 1997, and Peters and Schaal's reward-weighted regression dates to 2007. The innovation is the deliberate, end-to-end application of EM as a design principle for large-scale language model self-training, with each implementation choice (temperature sampling, per-problem cap, base-model reset) justified in terms of how it serves the E-step or M-step objectives. This transforms EM from a retroactive explanation into a proactive design tool.

Innovation 3: Self-Training Scaling Laws That Reverse the Prior Trend

The paper's third distinctive contribution is an empirical correction to the emerging narrative that self-training benefits diminish with model scale. Yuan et al. (2023), studying RFT on GSM8K, had reported that larger models showed smaller improvements from self-generated data — a finding that, if general, would severely limit the practical relevance of self-training for state-of-the-art LLMs. ReST^EM demonstrates the opposite scaling trend on harder benchmarks: on MATH, the absolute improvement is 5.94% for PaLM 2-S vs. 6.34% for PaLM 2-L; on APPS, 5.6% for PaLM 2-S* vs. 6.4% for PaLM 2-L. Larger models benefit more, not less, from self-training.

This reversal is not just a "our numbers are bigger" claim — it carries a specific, testable implication about when self-training will be most effective. GSM8K is an elementary math reasoning benchmark where strong models already achieve high accuracy. In that regime, the base model's pass@1 may be high enough that the correct solutions it generates are not substantially different from what it would produce under greedy decoding — there's little new information to extract from self-generated data. MATH and APPS are harder benchmarks where even strong models have substantial room for improvement, and the correct solutions surfaced by temperature sampling represent reasoning paths that the model does not already assign high probability. Self-training is therefore most impactful when the task is challenging enough that the model's exploratory samples include correct solutions it wouldn't discover greedily, but not so hard that the model produces no correct solutions at all.

This reframes the question from "does self-training work for large models?" to "on what kinds of tasks does self-training benefit large models?" The answer, per these results, is tasks in the "sweet spot" of difficulty where the model has non-trivial but imperfect capability — exactly the regime where fine-tuning is most practically valuable. Very easy tasks don't need fine-tuning; impossibly hard tasks can't be improved by any amount of self-training (the model generates no correct solutions to learn from). The middle ground — competition math, competitive programming — is where self-training provides outsized returns, and those returns increase with model capability because stronger models generate more diverse and sophisticated correct solutions in that middle ground.

Figure 8 (right) provides direct evidence for this difficulty-dependence: when problems are binned by the base model's success rate, ReST^EM improves performance across all difficulty levels, but the largest gains come for questions categorized as "medium" (50–75% base success rate) and "hard" (25–50%), not for "easy" (75–100%) or "very hard" (below 25%). This maps cleanly onto the sweet-spot hypothesis: medium and hard problems are challenging enough that the model needs improvement but tractable enough that temperature sampling produces correct solutions to learn from.

This insight is significant because it provides a diagnostic for when to deploy self-training. Practitioners can estimate their base model's success rate on a target task, and if a substantial fraction of problems fall in the 25–75% success range, self-training is likely to yield meaningful gains. If problems are all very easy or all very hard, self-training will provide minimal benefit — a different approach (prompt engineering for easy problems, pretraining or architectural improvements for hard problems) would be more appropriate.

Innovation 4: Base-Model Reset as a Mechanism for Preserving Generality

One of the paper's most understated but practically important innovations is the demonstration that always fine-tuning from the base model — rather than iteratively fine-tuning the model from the previous iteration — largely solves the task-specific overfitting problem that has plagued self-training methods. This is not framed as the paper's headline contribution, but it may be the design choice with the broadest implications for practitioners.

The mechanism is simple: at each M-step, discard the fine-tuned model from the previous iteration (except as a data generator for the next E-step) and retrain the base model from scratch on the latest filtered dataset. The paper justifies this as preventing distribution drift and preserving generalization, and it demonstrates the benefit concretely in Figure 7: ReST (which fine-tunes iteratively) and ReST^EM (which resets to base) achieve similar performance on the training task (APPS), but ReST^EM substantially outperforms ReST on transfer to HumanEval — a held-out coding benchmark the model was never fine-tuned on.

What makes this conceptually interesting is that it decouples data quality from model drift. In the iterative fine-tuning approach (ReST), the model gets better at the training task because it's trained on better data AND because it's been fine-tuned multiple times — but the latter effect compounds distribution shift, pulling the model away from its general pretrained capabilities. In the base-model reset approach (ReST^EM), the model benefits from better data (each iteration's E-step uses a stronger generator) without accumulating multiple rounds of fine-tuning drift. The model is always exactly one fine-tuning step away from the pretrained distribution, regardless of how many iterations have been run.

This finding has implications beyond self-training. It suggests a general principle for specialization of large language models: use improved models as data generators, not as initialization points for further training. The data improves across iterations even if the model resets each time, because the generating policy gets stronger. This principle could apply to distillation, to multi-task fine-tuning, and to any setting where one wants to improve task-specific performance without sacrificing generality.

The paper's ablation on dataset size (Figure 8, left) adds nuance: with only 1,000 MATH questions, a single iteration of ReST^EM already yields significant gains. This suggests the base-model reset approach is not only effective for preserving generality but also data-efficient — it extracts more learning signal per input problem than methods that chain fine-tuning steps.

Innovation 5: Overfitting as a First-Class Phenomenon in Self-Training — With a Clear Diagnostic

The paper's treatment of overfitting in self-training is distinctive not because it observes overfitting (that's expected), but because it characterizes it precisely and provides a diagnostic for when it will occur. Figure 4 shows the train-test gap widening with iterations: training performance improves monotonically (the model gets better and better at the specific problems it was trained on), but test performance plateaus (MATH) or regresses (APPS). The paper attributes this to the finite size of the training problem set — 7,500 for MATH, 2,342 for APPS — and notes that the smaller dataset overfits faster and more severely.

This is not merely "smaller datasets overfit more." The paper's contribution is to identify the number of unique training problems, not the number of training examples (problem-solution pairs), as the binding constraint. Each iteration of ReST^EM generates new solutions for the same set of problems. The model sees additional diversity in reasoning paths, but it never sees new problem types, new difficulty categories, or new mathematical concepts beyond those represented in the original problem set. Eventually, the model exhausts the useful variation in the problem set and begins to memorize problem-specific patterns that don't generalize. The solution diversity provided by temperature sampling and multiple iterations delays this saturation but does not eliminate it — eventually, the model has learned everything generalizable from the fixed problem distribution, and further training on the same problems only reinforces spurious patterns.

This insight has direct practical implications: the size of the problem set, not the per-problem sampling budget, is the primary determinant of how many ReST^EM iterations are useful. For MATH (7,500 problems), 2–3 iterations provide benefit before saturating. For APPS (2,342 problems), even 2 iterations overshoot. The paper's data-size ablation (Figure 8, left) quantifies this: 1,000 problems yield substantial gains, 2,000 yield more, but the marginal benefit per additional problem diminishes. This suggests practitioners should invest in collecting more distinct problems rather than generating more solutions per problem when scaling up self-training.

The paper also identifies the validation reward signal (not training loss) as the correct early-stopping criterion for the M-step, and the cross-iteration comparison of validation performance as the correct stopping criterion for the overall ReST^EM loop. This is a methodological contribution: it provides a principled way to decide when to stop iterating without access to test-set labels, using only the same automated reward function that drives the self-training process itself.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary benchmarks are Hendrycks' MATH dataset (7,500 training problems, 500 test problems) for mathematical reasoning and the APPS Introductory dataset (2,342 training problems) for code generation. Held-out transfer tasks include GSM8K (grade-school math), the Hungarian HS Finals Exam (a 2023 national exam for stress-testing real-world math capabilities), HumanEval (code generation), and Big-Bench Hard (BBH, 23 challenging tasks probing general reasoning capabilities). The MATH and APPS datasets are chosen because both provide automated binary correctness signals — ground-truth answers for MATH that can be string-matched, and test cases for APPS that determine whether generated code is executable and correct — enabling the scalar feedback that ReST^EM requires without any human-in-the-loop evaluation.

  • Base model(s). All experiments use the PaLM 2 model family (Google et al., 2023), with three variants tested: PaLM 2-S (Bison), PaLM 2-S* (Codey, a code-specialized variant), and PaLM 2-L (Unicorn, the largest variant). The paper states these were chosen because they are "representative of the capabilities of many contemporary LLMs" and cover a range of scales, enabling the core scaling analysis — whether ReST^EM benefits larger models more, less, or equally compared to smaller ones. The public cloud API versions of these models are used for all experiments.

  • Metrics. The primary metric throughout is pass@1 test accuracy: the fraction of test-set problems for which the model's single greedy-decoded solution (temperature 0) is correct. For MATH, correctness is determined by extracting the final answer from the model's output and comparing it to the ground-truth answer using the grading function from Hendrycks et al. (2021b). For APPS, correctness requires the generated program to pass all test cases (both example and hidden). Secondary metrics include pass@K (the probability that at least one of K temperature-sampled solutions is correct, evaluated with temperature 1.0 and nucleus sampling p=0.95), majority voting accuracy (the most common answer among K sampled solutions), and training-set pass@1 (used to diagnose overfitting by comparing against test pass@1). Transfer performance is measured using each benchmark's standard metric: accuracy for GSM8K and Hungarian HS Finals, pass@1 for HumanEval, and few-shot accuracy (with and without chain-of-thought prompting) for BBH tasks.

  • Baselines. The paper compares against three categories of baselines: (1) The pretrained base model with no fine-tuning (iteration 0 in all ReST^EM plots), serving as the lower bound for what self-training must surpass. (2) Supervised fine-tuning on human-written data (labeled "SFT" in figures), where the model is fine-tuned using the reference solutions from the MATH or APPS training sets — one human-written solution per problem. This is the critical baseline for the paper's central claim that model-generated data can outperform human data. (3) The original ReST algorithm (Gulcehre et al., 2023), which differs from ReST^EM in two ways: it augments the self-generated dataset with human data during the E-step, and it fine-tunes iteratively (each M-step starts from the previous iteration's model rather than resetting to the base model). This baseline tests whether the base-model reset and pure self-generation design choices in ReST^EM are beneficial. Additional comparisons in the literature review (Table 1) position ReST^EM relative to STaR (Zelikman et al., 2022), RFT (Yuan et al., 2023), and Expert Iteration (Anthony et al., 2017), though these are not re-implemented as direct experimental baselines. The paper also includes baseline results from other publicly reported models (GPT-4, Minerva, LLaMA variants, etc.) in Figure 1, but these are presented for contextual performance levels rather than as controlled comparisons.

  • Generation budget. The primary compute budget in the E-step is the number of solutions sampled per training problem per iteration: N=32 for MATH and N=64 for APPS. The per-problem cutoff limits the M-step training dataset to at most 10 correct solutions per problem regardless of how many were generated. For a fair comparison between single-iteration and multi-iteration ReST^EM, the paper uses a 3× data variant: a single E-step generating 3× as many solutions per problem (e.g., 96 instead of 32 for MATH) to match the total generation budget of three iterations. The M-step fine-tuning cost is not explicitly quantified in FLOPs but is standard supervised fine-tuning on the filtered dataset, always starting from the original base model. For the distillation experiments, the generation budget is determined by the teacher model (PaLM 2-L) generating solutions that are then used to fine-tune the student (PaLM 2-S).

  • Statistical protocol / Cross-validation. The paper does not employ formal cross-validation for model selection or hyperparameter tuning. Instead, a held-out validation set is used for early stopping during each M-step: "while reward improves on D_val do / Optimise θ" (Algorithm 1). The specific size or construction of this validation split is not reported. For evaluation, all test-set metrics are computed on the standard test splits of MATH (500 problems) and APPS (Introductory split). For the BBH evaluation (Figure 9), the paper notes that "Evaluations were conducted across multiple checkpoints, and the vertical black lines denote standard deviation," indicating some replicate measurements to assess variance, though the number of checkpoints is not specified. For the Hungarian HS Finals exam, the paper manually grades outputs using the official rubric, providing a qualitative accuracy check beyond automated metrics. The paper acknowledges (Section 5.3, discussing Figure 8 left) that "conducting this experiment multiple times would help quantify this variance, but this is prohibitively resource-intensive," explicitly flagging the single-run nature of most experiments as a limitation due to computational constraints.

Main Quantitative Results

ReST^EM vs. Human-Data SFT on MATH and APPS

The headline finding across both benchmarks is that fine-tuning on model-generated synthetic data via ReST^EM substantially outperforms supervised fine-tuning on human-written solutions, and this advantage grows with model scale.

MATH results (Figure 2): For PaLM 2-S, the base model achieves a baseline pass@1 test accuracy (iteration 0, reading from the figure) of approximately 15%. After three iterations of ReST^EM, accuracy reaches roughly 21% — an absolute improvement of approximately 6 percentage points. The human-data SFT baseline (PaLM-2-S-SFT, flat dashed line) achieves roughly 18%, meaning ReST^EM outperforms human-data fine-tuning by approximately 3 percentage points.

For the larger PaLM 2-L model, the base accuracy is approximately 35%. After three iterations of ReST^EM, accuracy reaches approximately 42% — an improvement of roughly 7 percentage points. The human-data SFT baseline (PaLM-2-L-SFT) reaches approximately 38%, giving ReST^EM a roughly 4 percentage point advantage. The paper reports these gains as 5.94% for PaLM 2-S and 6.34% for PaLM 2-L, confirming that the absolute improvement from ReST^EM is larger for the larger model — a trend directly contrary to the diminishing-returns finding reported by Yuan et al. (2023) for RFT on GSM8K.

The GSM8K transfer results (Figure 2, right panel) show a similar pattern: ReST^EM iterations improve transfer performance monotonically, and the ReST^EM-fine-tuned models substantially outperform the SFT baselines. For PaLM 2-L, GSM8K accuracy improves from roughly 62% (base, iteration 0) to approximately 73% after three ReST^EM iterations, while the SFT baseline achieves roughly 66%.

APPS results (Figure 3): For PaLM 2-S*, the base pass@1 on APPS Introductory is approximately 18%. After one ReST^EM iteration, accuracy jumps to roughly 24% — a gain of approximately 6 percentage points. The SFT baseline reaches roughly 22%. After a second iteration, however, performance regresses to approximately 23%, below the iteration-1 peak.

For PaLM 2-L, the base accuracy is approximately 20%, improving to roughly 26% after one iteration — a gain of roughly 6 percentage points, with the SFT baseline reaching approximately 23%. Again, a second iteration causes regression. The paper reports the improvements as 5.6% for PaLM 2-S* and 6.4% for PaLM 2-L. HumanEval transfer (Figure 3, right panel) shows that for PaLM 2-L, one ReST^EM iteration improves HumanEval pass@1 from roughly 46% to roughly 50%, while SFT reaches approximately 48%. Performance on HumanEval remains stable or slightly declines after the second ReST^EM iteration, mirroring the overfitting pattern observed on APPS itself.

The critical scaling observation: Across both benchmarks, larger models not only start from a higher baseline but also gain more absolute improvement from ReST^EM — MATH: 5.94% (PaLM 2-S) vs. 6.34% (PaLM 2-L); APPS: 5.6% (PaLM 2-S*) vs. 6.4% (PaLM 2-L). This is explicitly contrasted with Yuan et al. (2023), who found that RFT on GSM8K showed diminishing returns with model scale. The paper attributes the reversal to task difficulty: on harder benchmarks (MATH, APPS), larger models have more room to improve and generate more diverse correct solutions during the E-step, providing richer training data for subsequent iterations.

Iteration Dynamics and Overfitting

The multi-iteration behavior reveals a clear data-size-dependent saturation point (Figures 2 and 3), with the smaller APPS dataset overfitting much faster than MATH.

For MATH (7,500 training problems, Figure 2 left): pass@1 on the test set improves substantially from iteration 0 to iteration 1 (+4–5 percentage points), improves modestly from iteration 1 to iteration 2 (+1–2 points), and is approximately flat or marginally positive from iteration 2 to iteration 3. Training-set performance (Figure 4 left), however, continues to increase monotonically across all three iterations — from roughly 42% at iteration 1 to roughly 52% at iteration 3 for PaLM 2-L. The widening gap between training and test accuracy (roughly 42% vs. 41% at iteration 1 growing to roughly 52% vs. 42% at iteration 3) is the signature of overfitting: the model increasingly memorizes problem-specific patterns from the fixed training set rather than learning generalizable reasoning.

For APPS (2,342 training problems, Figure 3 left): pass@1 improves from iteration 0 to iteration 1 (+5–6 points) but then regresses at iteration 2, dropping below the iteration-1 peak. Training-set performance (Figure 4 right) climbs from roughly 26% at iteration 1 to roughly 34% at iteration 2, while test performance drops — an even starker train-test divergence than MATH, consistent with the smaller training set providing less protection against overfitting. GSM8K transfer (Figure 2 right) also shows positive transfer improvement from ReST^EM on MATH for both model sizes, with monotonic improvement across iterations for PaLM 2-S and PaLM 2-L alike, suggesting the overfitting on MATH is task-specific rather than catastrophic.

Single-iteration with more data vs. multiple iterations: The paper's 3× data ablation (Section 5.3) addresses whether the benefit of multiple iterations is simply a matter of total data volume. Fine-tuning PaLM 2-L on a single E-step dataset with 3× the solutions per problem (matching the total generation budget of three iterations) yields pass@1 of 40.3%. This is lower than both iteration 2 (41%) and iteration 3 (41.9%), demonstrating that the iterative refinement of the data-generating policy — not just the total volume of data — contributes to the gains. A single E-step from the base model cannot produce the higher-quality solutions that the iteration-2 E-step (using the fine-tuned model as generator) can produce.

Apples-to-Apples Comparison: Model-Generated vs. Human Data with Equal Solutions Per Problem

A potential confound in the main ReST^EM vs. SFT comparison is that ReST^EM provides multiple correct solutions per problem (up to 10) while SFT uses only a single human-written solution. The paper addresses this with a controlled experiment (Figure 6, left) on the subset of MATH questions (approximately 5,000) for which the base model generates at least one correct solution:

  • SFT (5K): Standard fine-tuning on one human-written solution per problem — accuracy approximately 39%.
  • ReST* (5K): One iteration of ReST^EM using only one model-generated correct solution per problem (randomly selected) — accuracy approximately 40%, already outperforming human-data SFT by roughly 1 percentage point despite using the same number of solutions per problem.
  • ReST^EM (5K): The full three-iteration ReST^EM pipeline with up to 10 correct solutions per problem — accuracy approximately 41.5%, showing that multiple solutions and multiple iterations provide additional gains beyond the single-solution parity point.

This result eliminates the "more data" confound: even at exactly one solution per problem, model-generated data outperforms human-written data. The paper hypothesizes this is because model-generated solutions are more in-distribution — they match the model's own generation patterns — making them a more natural target for fine-tuning than human-written solutions, which may use different reasoning styles, notation conventions, or organizational structures that the model must learn to imitate.

Distillation: Teacher-Generated Data Outperforms Student Self-Generation

The distillation experiments (Figure 6, right) test whether ReST^EM-generated data from a larger model can improve a smaller model, and whether this is more or less effective than the smaller model's own self-generated data:

  • SFT (Human): PaLM 2-S fine-tuned on human-written MATH solutions — accuracy approximately 18%.
  • ReST^EM (2-S): PaLM 2-S fine-tuned with three iterations of ReST^EM on its own self-generated data — accuracy approximately 21%.
  • Distill* (2-L): PaLM 2-S fine-tuned on PaLM 2-L-generated solutions, using exactly one correct solution per problem (matching the data-per-problem of SFT) — accuracy roughly 22%, already outperforming the student's own self-training.
  • Distill (2-L): PaLM 2-S fine-tuned on multiple PaLM 2-L-generated solutions per problem from the final ReST^EM iteration — accuracy roughly 24–25%, substantially outperforming both human-data SFT and the student model's own ReST^EM.

The implication is that a stronger model's outputs are better training data than a weaker model's own successful outputs, and that distillation from a larger teacher (with or without multiple solutions per problem) can be more effective than the student model's self-training — at least when the teacher is substantially more capable on the task. This also validates that model-generated data is beneficial across model scales, not just for the model that generated it.

Pass@K and Majority Voting Improvements

To assess whether ReST^EM fine-tuning improves the diversity of correct solutions or merely sharpens the model's most likely output, the paper evaluates pass@K and majority voting (Figure 5, Section 5.2):

  • MATH (PaLM 2-L, Figure 5 left): The ReST^EM-fine-tuned model achieves higher pass@K than the base model across all values of K from 1 to 64. The gap is largest at K=1 (roughly 42% vs. 37%) and narrows but remains positive at K=64 (roughly 78% vs. 70%).
  • APPS (PaLM 2-L, Figure 5 middle): Similar pattern — the fine-tuned model outperforms the base model at all K from 1 to 10, with the largest relative gap at K=1.
  • HumanEval (PaLM 2-L, Figure 5 right): The fine-tuned model outperforms the base model across K values ranging from 1 to 64, with the gap at K=64 being roughly 88% vs. 80%.

For majority voting on MATH with 64 samples per question, the paper reports: "the PaLM 2-L fine-tuned with ReST^EM obtains a test accuracy of 48.82, while the base model gets 44.02." This roughly 4.8 percentage point improvement in majority-voting performance indicates that ReST^EM improves both the most likely answer (pass@1) and the ensemble of diverse correct reasoning paths (pass@K / majority vote).

Which Questions Benefit Most from ReST^EM

The paper stratifies MATH test questions by the base model's success rate at temperature 1.0 into four difficulty categories (Figure 8, right): "easy" (75–100% success), "medium" (50–75%), "hard" (25–50%), and "very hard" (below 25%). ReST^EM improves the average success rate across all categories, but with a pronounced inverted-U pattern: the largest gains occur for "medium" and "hard" questions, with smaller gains for "easy" questions (which the base model already solves most of the time) and "very hard" questions (for which the model generates few or no correct training examples). This difficulty-dependence is consistent across both MATH and APPS.

Transfer and General Capabilities

The paper evaluates ReST^EM-fine-tuned models on several held-out benchmarks to test whether task-specific self-training degrades or improves general capabilities:

  • GSM8K (Figure 2 right): Both PaLM 2-S and PaLM 2-L models fine-tuned with ReST^EM on MATH show positive transfer to this grade-school math benchmark, with accuracy improving monotonically across iterations. For PaLM 2-L, GSM8K accuracy moves from approximately 62% (base) to roughly 73% after three iterations. This is an example of beneficial transfer: learning to solve harder math problems (MATH) improves performance on easier math problems (GSM8K).

  • HumanEval (Figure 3 right): Models fine-tuned on APPS show positive transfer to HumanEval after one iteration (e.g., PaLM 2-L improving from roughly 46% to roughly 50%), with stable or slightly declining performance after a second iteration. This is consistent with the APPS overfitting pattern — once the model begins to overfit on the APPS training problems, its general coding ability plateaus or slightly degrades.

  • Hungarian HS Finals Exam (Figure 10): The PaLM 2-L model fine-tuned with ReST^EM on MATH achieves an exam score of approximately 54% (estimated from Figure 10), outperforming many specialized math models that score well on GSM8K but poorly on this harder, real-world exam. This includes models like MetaMath Mistral 7B, Llemma 34B, and Code Llama 34B, which achieve GSM8K scores in the 80–90% range but exam scores of 30–40%. The paper highlights this as evidence of robust transfer: ReST^EM on MATH does not simply teach the model to solve MATH-style problems but improves generalized mathematical reasoning that transfers to novel, out-of-distribution exam formats.

  • Big-Bench Hard (Figure 9): Both the MATH-fine-tuned and APPS-fine-tuned PaLM 2-L models show no significant degradation on any of the 23 BBH tasks compared to the base model, when evaluated using chain-of-thought prompting. The MATH-fine-tuned model actually outperforms the base model on average BBH performance with chain-of-thought (roughly 72% vs. 68%), while the APPS-fine-tuned model shows slightly positive gains (roughly 70% vs. 68%). Under direct prompting (no chain-of-thought), all three models — base, MATH-fine-tuned, APPS-fine-tuned — perform similarly (roughly 63–65%). The vertical black lines in Figure 9 denote standard deviation across multiple checkpoints, and the paper states there is "no major degradation on any of the BBH tasks."

This transfer profile — positive transfer to related tasks (GSM8K, HumanEval, Hungarian exam), no degradation on a broad suite of 23 diverse reasoning tasks — supports the claim that the base-model reset in ReST^EM preserves general capabilities while improving task-specific performance. The contrast with the original ReST (Figure 7), where iterative fine-tuning degrades HumanEval transfer, reinforces this interpretation.

Ablation Studies and Robustness Checks

ReST^EM vs. ReST (iterative fine-tuning vs. base-model reset): Figure 7 compares the two approaches using PaLM 2-S* on APPS. On the APPS test set, ReST^EM and ReST achieve nearly identical performance (roughly 21.7% vs. 21.6% after multiple iterations). However, on HumanEval transfer, ReST^EM substantially outperforms ReST — reaching roughly 44% after multiple iterations versus roughly 39–40% for ReST. This demonstrates that the base-model reset preserves general coding ability while iterative fine-tuning progressively degrades it, even though both achieve similar task-specific performance. The human-data SFT baseline on HumanEval (flat line) achieves roughly 40–41%, meaning ReST overfits to APPS at the expense of falling below the human-data baseline on transfer, while ReST^EM remains above it.

Number of ReST^EM iterations and overfitting: Figure 4 shows the train-test gap widening with iterations for both MATH (PaLM 2-L) and APPS (PaLM 2-S*). On MATH, training accuracy increases from roughly 42% (iteration 1) to roughly 52% (iteration 3), while test accuracy moves from approximately 40% to approximately 42%. On APPS, training accuracy climbs from roughly 24% (iteration 1) to roughly 35% (iteration 2), while test accuracy drops from approximately 24% to approximately 22%. The takeaway: more iterations always improve training performance, but test performance saturates (MATH) or regresses (APPS) depending on training set size, with the smaller APPS dataset (2,342 problems) overfitting faster than MATH (7,500 problems).

Dataset size ablation (Figure 8, left): Using a single iteration of ReST^EM on MATH with varying numbers of training problems: 1,000 problems yields pass@1 of roughly 38%, 2,000 yields roughly 40%, 4,000 yields roughly 39.5% (note the slight dip, which the paper attributes to "potential variance in the fine-tuning process"), and 7,000 yields roughly 41%. The finding: ReST^EM is sample-efficient — substantial gains are achievable with only 1,000 training problems — but additional problems continue to provide incremental benefit up to the full dataset size. The anomalous dip at 4,000 highlights the variance inherent in single-run fine-tuning experiments that the paper acknowledges it cannot fully quantify due to computational constraints.

Solutions per problem threshold: The paper uses a per-problem cutoff of 10 correct solutions in the E-step. No formal ablation of this specific threshold is reported. The justification provided is that without it, the dataset becomes "imbalanced" toward easy problems, but the sensitivity of results to alternative cutoff values (e.g., 5, 20, or no cutoff) is not empirically tested. The paper notes this design choice was "also used by Zelikman et al. (2022)" but does not independently validate it.

Sampling temperature and strategy: The paper uses top-K sampling with K=40 and temperature 0.7 for all E-step generations. No ablation of alternative temperatures, alternative K values, or alternative decoding strategies (nucleus sampling, beam search during generation) is reported. The paper contrasts temperature sampling with greedy decoding in the literature review and with STaR's rationalization approach but does not provide empirical evidence within its own experiments for why these specific values were chosen.

Distillation with teacher-generated vs. self-generated data (Figure 6, right): This ablation tests four conditions for fine-tuning PaLM 2-S on MATH: human data (SFT, ~18%), self-generated ReST^EM data (~21%), teacher-generated data with one solution per problem (Distill*, ~22%), and teacher-generated data with multiple solutions from the full ReST^EM pipeline (Distill, ~24–25%). The finding: teacher-generated data from PaLM 2-L is more effective than PaLM 2-S's own self-generated data, and multiple solutions per problem provide additional benefit beyond single-solution distillation. This does not ablate ReST^EM itself but clarifies the relative value of different data sources.

Rationalization vs. temperature sampling (Section 4, qualitative): The paper reports a negative result from preliminary experiments: STaR-style rationalization (providing the correct answer as input and asking the model to generate reasoning for it) "leads to substantial increase in false positive solutions that result in correct answer but with incorrect reasoning." No quantitative data is presented for this claim — it is reported as a qualitative observation that motivated the choice to use pure temperature sampling rather than rationalization for hard problems.

Critical Assessment

Claim 1: ReST^EM substantially outperforms fine-tuning on human-generated data

What the experiments demonstrate: On MATH, ReST^EM with three iterations achieves higher pass@1 than SFT on human data for both PaLM 2-S and PaLM 2-L (Figure 2). On APPS, the same holds for one iteration (Figure 3). The controlled apples-to-apples comparison (Figure 6 left) shows that even with exactly one model-generated solution per problem, ReST^* outperforms SFT on the same subset of questions.

What the experiments do NOT demonstrate: The comparison is specific to PaLM 2 models on two benchmarks (MATH and APPS). The paper does not test whether model-generated data outperforms human data for models from other families (GPT, LLaMA, Claude), for tasks beyond math and code, or for tasks where the reward signal is noisy, continuous, or learned rather than binary and deterministic. Additionally, the human data consists of the original reference solutions included in the MATH and APPS datasets — these are single solutions of unknown quality and diversity. It is conceivable that a more carefully curated human dataset (multiple diverse solutions per problem, expert-verified reasoning quality) would close or eliminate the gap. The paper demonstrates that model-generated data can outperform the specific human data available in these benchmarks, not that it universally outperforms any possible human data.

Boundary conditions identified in the paper: The advantage over human data is most pronounced for larger models (Figure 2, 3) and for problems in the medium-to-hard difficulty range (Figure 8 right). On very easy or very hard problems, the gap narrows. On APPS, the advantage disappears by the second iteration due to overfitting, highlighting that the superiority of model-generated data is contingent on having a sufficiently large training problem set.

Missing experiments: The paper does not test whether human data combined with model-generated data (as in original ReST) outperforms either alone — the ReST^EM design deliberately excludes human data. A hybrid condition would clarify whether the gains are additive or whether model-generated data alone is genuinely sufficient. The paper also does not compare against a baseline where human-written solutions are augmented (e.g., by asking a human to write two or three diverse solutions per problem) to match the per-problem data volume of ReST^EM.

Claim 2: ReST^EM scales favorably with model size — larger models benefit more

What the experiments demonstrate: On MATH, PaLM 2-L gains 6.34% from ReST^EM while PaLM 2-S gains 5.94%. On APPS, PaLM 2-L gains 6.4% while PaLM 2-S* gains 5.6%. The absolute improvement is larger for larger models, and this holds despite the larger model starting from a substantially higher baseline (making further improvement harder in absolute terms). This directly contradicts Yuan et al. (2023)'s RFT scaling results on GSM8K.

What the experiments do NOT demonstrate: Only two scale points are compared — PaLM 2-S (~tens of billions of parameters) and PaLM 2-L (~hundreds of billions). Two points do not establish a scaling trend, only a directional observation. A genuine scaling analysis would require at least 3–4 model sizes spanning an order of magnitude in parameters, ideally with a fitted scaling law. The paper's conclusion that "larger models benefit more" is based on comparing exactly two models per task.

The GSM8K reversal is attributed to task difficulty, which is a plausible hypothesis but is not experimentally verified: the paper does not run ReST^EM on GSM8K to confirm that it shows the same diminishing-returns pattern as RFT on that easier benchmark. Without this direct replication, the attribution to task difficulty remains speculative.

Claim 3: Model-generated data can be more effective than human data for distillation

What the experiments demonstrate: Distillation from PaLM 2-L to PaLM 2-S using model-generated data (Figure 6 right) outperforms both human-data SFT and PaLM 2-S's own self-training. The Distill* condition (one teacher solution per problem) already beats SFT.

What the experiments do NOT demonstrate: The distillation comparison is limited to a single teacher-student pair (PaLM 2-L → PaLM 2-S). The generalization to other scale gaps (e.g., PaLM 2-L → PaLM 2-S*), other model families, or other tasks is untested. The paper also does not compare distillation from a teacher fine-tuned on human data vs. a teacher fine-tuned with ReST^EM — the distillation uses the raw PaLM 2-L as the teacher, not a ReST^EM-improved PaLM 2-L. This leaves open whether the distillation gains would compound if both teacher and student underwent ReST^EM.

Claim 4: ReST^EM preserves generalization unlike iterative fine-tuning (ReST)

What the experiments demonstrate: Figure 7 shows that ReST^EM matches ReST on APPS while substantially outperforming it on HumanEval transfer. Figure 9 shows no degradation on BBH tasks and even slight improvements for the MATH-fine-tuned model under chain-of-thought prompting. The Hungarian exam result (Figure 10) and GSM8K transfer (Figure 2 right) show positive transfer.

What the experiments do NOT demonstrate: The BBH evaluation (Figure 9) tests 23 specific reasoning tasks and shows no degradation, but this is not a comprehensive evaluation of all capabilities. The paper does not test for degradation on tasks far from the fine-tuning domain — e.g., creative writing, translation, summarization, or dialogue — where distribution shift from math/code fine-tuning might manifest differently. The APPS-fine-tuned model does show slight regression on HumanEval after the second ReST^EM iteration (Figure 3 right), indicating that preservation of generalization is not absolute and depends on stopping before overfitting.

Methodological concern: The transfer results (GSM8K, HumanEval, Hungarian exam) are evaluated on the final fine-tuned model, not on intermediate models. It is therefore unclear whether the positive transfer is due to ReST^EM's design (base-model reset) or simply due to the beneficial effects of improved reasoning capabilities from MATH/APPS training that would occur with any fine-tuning method. The one direct comparison — ReST^EM vs. ReST on HumanEval (Figure 7) — supports the claim that base-model reset is better for transfer, but this comparison is made only for APPS → HumanEval with one model.

Overall Assessment

The experiments provide strong but narrow evidence for the paper's central thesis: on competition-level math and introductory code generation with PaLM 2 models, iteratively training on self-generated correct solutions outperforms training on the provided human-written solutions. The evidence is strong within this scope because it is replicated across two tasks, two model scales, multiple iterations, and several controlled ablations.

The narrowness is the primary limitation: everything is tested on exactly two benchmarks from two domains, with exactly one model family, using exactly one type of reward function (binary, deterministic, automated). Whether these findings transfer to other tasks (scientific reasoning, planning, open-ended generation), other reward structures (learned reward models, noisy human preferences, continuous scores), or other model families (GPT, LLaMA, Claude) is entirely unexamined. The paper itself is appropriately modest about this scope (Section 6), acknowledging that the method requires "a moderately-sized training set of problems or prompts" and "access to a manually-designed or learned reward function" — resources that must be provided by humans initially.

The single-run nature of all fine-tuning experiments, acknowledged explicitly in the discussion of Figure 8 (left), is a genuine weakness for quantitative claims about exact accuracy numbers. The anomalous dip at 4,000 training questions in the dataset size ablation demonstrates that single-run variance is non-trivial. The paper's accuracy claims should be understood as nominal values from single training runs rather than statistically reliable estimates with quantified uncertainty.

Finally, while the paper's comparison against human data is its strongest empirical claim and most attention-grabbing result, the "human data" in question is simply the reference solutions packaged with the MATH and APPS benchmarks. These are not necessarily high-quality pedagogical demonstrations — they are whatever solutions the dataset creators chose to include. The paper would be strengthened by a comparison against a more carefully constructed human baseline: multiple diverse solutions per problem, verified for reasoning quality, or written by domain experts specifically for the purpose of fine-tuning. The current comparison, while fair as a benchmark comparison, may overstate the advantage of synthetic data relative to what a well-resourced human annotation effort could produce.

6. Limitations and Trade-offs

6.1 The Method Requires a Moderately-Sized Training Set of Problems With Known Ground-Truth Answers

The assumption or constraint. ReST^EM depends on two resources that must be provided by humans before the self-training loop can begin: a training dataset of input problems (the paper uses 7,500 MATH questions and 2,342 APPS questions) and a deterministic, automated binary reward function that can evaluate the correctness of any model-generated solution. The paper is explicit about both requirements in Section 6:

"First, this method requires a moderately-sized training set of problems or prompts, which would need to be collected (from humans) for any new task of interest. Second, ReST^EM also requires access to a manually-designed or learned reward function, ideally one that can be computed automatically."

For MATH, the reward function is answer-matching against ground-truth answers that come bundled with the dataset. For APPS, it is execution against test cases — including hidden test cases not visible in the problem description. In both cases, the reward function is not just binary and deterministic, but oracle-level: it has access to the ground-truth answer or the full test suite that defines correctness. This is not a minor implementation detail — it means the reward function is perfectly reliable for the tasks studied, with zero false positives and zero false negatives.

The consequence. For any task where such a perfect automated reward does not exist — which is the vast majority of real-world problem-solving tasks — ReST^EM cannot be applied without either (a) building a learned reward model (which introduces its own errors, distribution shift, and potential for reward hacking), (b) using human evaluation as the reward (which eliminates the scalability and cost advantages that motivate the method), or (c) accepting an imperfect reward signal whose errors would propagate through the self-training loop, potentially amplifying false positives into the training data.

The paper's results fundamentally rely on the reward function being perfectly reliable. A single false positive — a solution that passes the reward check but contains incorrect reasoning — becomes a training example that the model learns to imitate. In a multi-iteration loop, these errors can compound: the model learns flawed patterns from iteration 1's false positives, then generates more sophisticated flawed solutions in iteration 2's E-step, some of which pass the reward check again, further reinforcing the errors. The paper does not study this failure mode because the oracle reward functions used for MATH and APPS preclude it, but it is the central risk for any deployment where the reward is imperfect.

The requirement for a "moderately-sized training set" of problems is also binding. Figure 8 (left) shows that 1,000 MATH questions yield substantial gains, but this is still 1,000 competition-level math problems with verified answers — a non-trivial curation effort. For a novel domain where no such problem set exists, the human effort required to create it (writing diverse problems, verifying answers, designing test cases) may rival or exceed the effort of writing full solution demonstrations. The paper's framing as "reducing dependence on human data" is accurate for eliminating the need for human-written solutions, but the dependence on human-written problems with verified answers remains and is not benchmarked in cost terms.

What evidence exists in the paper. Figure 8 (left) quantifies the sensitivity to training problem count: pass@1 drops from approximately 41% with 7,000 questions to approximately 38% with 1,000 questions. The overfitting analysis (Figure 4) demonstrates that the binding constraint is the number of distinct problems, not the number of generated solutions — APPS with 2,342 problems overfits by iteration 2, while MATH with 7,500 problems tolerates 3 iterations. This indirectly confirms that the problem set size, which must come from human curation, is the bottleneck resource. The paper provides no experiment where the reward function is noisy, learned, or imperfect.

Mitigation status. The paper acknowledges these requirements in Section 6 as "limitations" and suggests "future research in self-improvement in language models should focus on automating manual parts of the pipeline (likely through language models as well)." No concrete proposal is developed. The suggestion to use language models as automated reward functions is mentioned but not tested — and would reintroduce the verifier reliability problem that ReST^EM currently avoids by using oracle rewards. The paper does not discuss the failure modes that would arise from imperfect rewards, nor does it provide guidance on how reliable a reward function must be for ReST^EM to be beneficial rather than harmful.

6.2 Overfitting on the Training Problem Set Is the Hard Ceiling — And the Paper Provides No Mechanism to Detect It Without a Test Set

The assumption or constraint. The paper demonstrates that ReST^EM's performance on held-out test data saturates and eventually declines with additional iterations, while training-set performance continues to improve monotonically (Figure 4). This overfitting occurs because each iteration reuses the same finite set of training problems — the model sees more diverse solutions to the same problems but never encounters new problem types. The paper identifies the size of the training problem set as the primary factor determining how many iterations are useful: MATH (7,500 problems) supports 2–3 iterations, while APPS (2,342 problems) overfits by the second iteration.

Critically, the paper uses test-set performance to diagnose this overfitting and to decide how many iterations to run. In a real deployment, the practitioner does not have access to the test set — if they did, they would not need to fine-tune the model; they would already have the answers. The paper's Algorithm 1 specifies using a validation set ("while reward improves on D_val") for early stopping within each M-step, but this validation set consists of held-out problems from the same distribution — problems for which the ground-truth answers are known and can be used to compute the reward. For the overfitting across iterations (deciding whether to run iteration 2, 3, etc.), the paper relies on test-set performance (Figures 2 and 3) to conclude when to stop.

The consequence. A practitioner deploying ReST^EM on a new task faces a version of the overfitting problem with no clear solution. They can monitor training-set accuracy and validation-set accuracy (if they hold out some of their problems with known answers), but as Figure 4 shows, validation accuracy may continue to improve or stay flat while test accuracy degrades, especially when the training set is small relative to the diversity of the test distribution. On APPS (Figure 3), the validation signal would need to detect the iteration-1-to-iteration-2 regression, but this regression is modest (roughly 1–2 percentage points) and may be within the noise of validation-set evaluation on a small held-out set.

More fundamentally, the overfitting is to the specific types of problems in the training set. If the training problems are not representative of the deployment distribution — a common scenario when training data is easier to collect than real-world examples — the model may appear to improve on validation (which is drawn from the same biased distribution) while degrading on the actual target distribution. The paper's transfer experiments (GSM8K, HumanEval, Hungarian exam) show positive transfer from MATH/APPS training, but these are cases where the training distribution (harder problems) transfers well to the test distribution (easier or different-format problems). The opposite scenario — where the training problems are easier or narrower than the deployment problems — would likely show negative transfer, but this is not tested.

What evidence exists in the paper. Figure 4 directly shows the train-test divergence for both MATH and APPS. For MATH (PaLM 2-L), training accuracy climbs from approximately 42% (iteration 1) to approximately 52% (iteration 3), while test accuracy moves from approximately 40% to approximately 42% — the gap widens from 2 points to 10 points. For APPS (PaLM 2-S*), training accuracy climbs from approximately 24% to approximately 35% while test accuracy drops from approximately 24% to approximately 22% — a complete decoupling where training improvement predicts test degradation. The paper's dataset-size ablation (Figure 8, left) shows that more training problems monotonically improve test performance (modulo the anomalous dip at 4,000), confirming that problem set size is the active constraint. The paper does not report validation-set performance across iterations, so it is unknown whether the validation signal would have correctly identified the overfitting point — this is a missing diagnostic that a practitioner would need.

Mitigation status. The paper does not address the problem of detecting overfitting without test-set access. The validation set mentioned in Algorithm 1 is used only for within-iteration early stopping, not for cross-iteration stopping decisions. The paper's suggestion (Section 6) that "future research... should focus on automating manual parts of the pipeline" does not specifically address overfitting detection. A practitioner would need to hold out a portion of their training problems (with known answers) as a validation set and monitor its performance across iterations, but the paper provides no evidence that this would reliably detect the iteration-1-to-iteration-2 regression observed on APPS.

6.3 The Experiments Cover Exactly Two Benchmarks, One Model Family, and Two Model Scales — Generalization to Other Domains, Models, and Reward Structures Is Unexamined

The assumption or constraint. Every experiment in the paper uses PaLM 2 models (S, S*, and L variants) on exactly two benchmarks: MATH (competition-level mathematics) and APPS Introductory (code generation from problem descriptions). Both benchmarks share a specific structure: the input is a problem statement, the output is a solution (reasoning chain + answer for MATH, program code for APPS), and correctness is determined by a deterministic, automated, oracle-level binary reward (answer-matching for MATH, test-case execution for APPS). The paper does not test on any other model family (GPT, LLaMA, Claude, Mistral), any other task domain (scientific reasoning, planning, open-ended generation, dialogue), or any other reward type (learned reward models, continuous scores, noisy human labels, partial-credit rewards).

All claims about scaling behavior — "larger models benefit more," "self-training with feedback can reduce dependence on human-generated data" — are supported by comparing exactly two model sizes per task: PaLM 2-S and PaLM 2-L for MATH, PaLM 2-S* and PaLM 2-L for APPS. Two data points do not establish a scaling trend; they establish a directional observation at two specific scales.

The consequence. There are several plausible failure modes that the current experimental scope cannot rule out:

  • Model family dependence. PaLM 2 models may have properties that make them particularly amenable to self-training — specific calibration characteristics, specific patterns of generation diversity, or specific base-model capabilities that affect the quality of self-generated solutions. Models from other families (e.g., GPT-4, LLaMA-2) might respond differently to ReST^EM, potentially showing weaker gains, faster overfitting, or even performance degradation if their self-generated solutions are less diverse or contain more subtle errors that pass the reward check.

  • Task domain dependence. MATH and APPS are both symbolic reasoning tasks with unambiguous correctness criteria. ReST^EM's effectiveness may depend on this structure: the binary reward is perfectly reliable, and the task requires multi-step reasoning where diverse correct solution paths exist. For tasks where correctness is subjective (summarization, creative writing), where the reward signal is noisy (learned reward models), or where the task is more about factual recall than reasoning (trivia, knowledge-intensive QA), the self-training dynamics could be qualitatively different — potentially amplifying biases in the reward model, reinforcing shallow patterns rather than reasoning, or providing no benefit because temperature sampling does not surface alternative correct solutions for factual questions.

  • Scale monotonicity. The paper observes that PaLM 2-L benefits more from ReST^EM than PaLM 2-S, and attributes this to the larger model's greater capacity to generate diverse correct solutions. However, this trend could reverse at even larger scales. A hypothetical PaLM 2-XL might have such high baseline performance that the remaining errors are on problems where even temperature sampling produces no correct solutions — the hard-ceiling regime where the model's capability gap cannot be bridged by self-training. In that case, the benefit of ReST^EM would plateau or decline at the largest scales, making the "larger models benefit more" trend U-shaped rather than monotonic.

  • Reward structure dependence. The binary oracle reward used for MATH and APPS is the simplest and most reliable reward type. If the reward were continuous (e.g., a 0–1 score from a learned verifier), the reward-weighted M-step objective (Equation 3) would weight solutions proportionally to their reward rather than applying a hard filter. This introduces a hyperparameter (the reward-weighting scheme) and a new failure mode (low-reward but correct solutions being downweighted, high-reward but subtly flawed solutions being upweighted) that the paper's experiments do not explore.

What evidence exists in the paper. The paper provides zero experiments outside the PaLM 2 / MATH / APPS combination. The transfer experiments (GSM8K, HumanEval, Hungarian exam, BBH) evaluate the fine-tuned models on held-out tasks but do not test whether ReST^EM itself works when applied to those tasks as the training objective — they test generalization of a MATH-trained model, not replication of the self-training pipeline on a different domain. The paper cites other works that applied related methods to other domains (Gulcehre et al., 2023 for machine translation, Agarwal et al., 2019 for semantic parsing, Yuan et al., 2023 for GSM8K) but does not replicate these experiments with ReST^EM or compare against their specific results.

Mitigation status. The paper does not claim generalization beyond its experimental scope. Section 6 acknowledges that ReST^EM requires "a moderately-sized training set of problems or prompts" and "access to a manually-designed or learned reward function" for "any new task of interest," implicitly recognizing that the method must be re-validated per domain. However, the title ("Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models") and abstract suggest broader applicability than the experiments support. The paper does not propose a research program for testing ReST^EM across diverse domains, reward types, or model families, leaving the generalization question entirely to future work.

6.4 The Compute Cost of Generating and Filtering Solutions Is Not Compared to the Cost of Obtaining Human Data — Nor Is It Amortized in the Efficiency Claims

The assumption or constraint. The paper measures computational cost only implicitly, through the number of solutions generated per problem per iteration (32 for MATH, 64 for APPS). The total generation budget for three iterations of ReST^EM on MATH with 7,500 training problems is 7,500 × 32 × 3 = 720,000 forward passes through a PaLM 2-L model. Each forward pass generates a full solution (potentially hundreds of tokens), and each must be scored by the reward function (answer extraction + string matching for MATH, code execution against test cases for APPS). The M-step then fine-tunes the base model on the filtered dataset (tens of thousands of problem-solution pairs), requiring additional backward passes.

The paper does not translate these generation counts into FLOPs, wall-clock time, or dollar cost, and it does not compare this compute cost to the cost of alternative approaches — specifically, the cost of hiring human experts to write additional training solutions or the cost of pretraining a larger model that might match the ReST^EM-improved performance without self-training.

The consequence. A practitioner evaluating whether to adopt ReST^EM faces a cost-benefit decision that the paper does not inform. The headline claim — "self-training with feedback can reduce dependence on human-generated data" — is a claim about substituting one resource (human annotation effort) for another (compute). But the paper provides no basis for comparing these resources. How many GPU-hours of PaLM 2-L inference does it cost to run three ReST^EM iterations on MATH? How does that compare to the cost of hiring mathematicians to write additional solution demonstrations? How does it compare to simply using a larger pretrained model with greedy decoding? Without these comparisons, the "reduce dependence on human data" framing is a qualitative statement, not a quantitative efficiency claim.

The cost picture becomes more complex when considering that the E-step generation budget (32 or 64 samples per problem) is a hyperparameter that the paper does not ablate. Would 16 samples per problem yield nearly the same gains at half the cost? Would 128 samples yield substantially better gains? The paper's 3× data ablation (Section 5.3) suggests that more samples in a single iteration underperform fewer samples across multiple iterations, indicating that the generation budget per iteration matters — but the sensitivity of final performance to this budget is unexplored. A practitioner cannot determine the cost-optimal generation budget from the paper's experiments.

The reusability of the self-training compute is also unexamined. The solutions generated during ReST^EM are used once (for one M-step) and then discarded — only the fine-tuned model is retained. If a new base model is released (e.g., PaLM 3), the entire ReST^EM pipeline must be rerun from scratch because the self-generated solutions are model-specific (they are in-distribution for PaLM 2, not necessarily for PaLM 3). In contrast, human-written solutions are model-agnostic and can be reused across model generations. The paper's approach amortizes none of the generation cost across training runs.

What evidence exists in the paper. The paper reports generation counts (32 for MATH, 64 for APPS) and the 3× data comparison (Section 5.3) but provides no FLOP counts, no timing measurements, and no cost estimates. The paper does not compare the cost of ReST^EM to the cost of pretraining (unlike the Singh et al. 2024 companion paper, which includes a FLOPs-matched pretraining-vs-inference comparison — but that is a different work and not part of this paper's analysis). The per-problem cutoff of 10 solutions is a cost-relevant design choice (it limits the M-step dataset size) but is not ablated, so its cost impact relative to no cutoff is unknown.

Mitigation status. The paper does not address compute cost as a limitation or propose efficiency improvements. Section 6 mentions that the method requires "a moderately-sized training set" and an "automated reward function," but does not discuss the computational resources needed to run the self-training loop. The remark that "future research... should focus on automating manual parts of the pipeline" does not address compute efficiency.

6.5 The 38% Correct-to-Incorrect Reversion Rate Observed in the Companion Paper Raises Questions About Solution Quality — But This Paper Provides No Mechanism to Assess It

The assumption or constraint. ReST^EM treats all solutions that pass the binary reward check as equally valid training examples. For MATH, a solution is correct if the extracted final answer matches the ground-truth answer. For APPS, a solution is correct if the generated code passes all test cases. In neither case does the reward function assess the quality of the reasoning that leads to the correct answer. A solution can receive a reward of 1 while containing flawed logic, irrelevant steps, or even contradictory statements, as long as the final answer is correct.

The paper is aware of this issue. Section 4 reports a negative result from preliminary experiments with STaR-style rationalization (providing the correct answer as a hint): "rationalization leads to substantial increase in false positive solutions that result in correct answer but with incorrect reasoning." This observation motivated the choice to use temperature sampling rather than rationalization. However, temperature sampling does not eliminate false positives — it only avoids the specific failure mode where the model is explicitly prompted to reason backward from the answer.

The companion Singh et al. (2024) paper (cited in this work's Section 8 and analyzed extensively in this summary's Section 6.1) reports that revision models trained on self-generated correct solutions have approximately a 38% rate of converting correct answers back to incorrect ones during revision chains — a phenomenon attributed to the training data containing only correct solutions (so the model never learns that some solutions are already correct and should not be revised). While that specific failure mode applies to revision models, the underlying concern — that self-generated "correct" solutions may contain subtle errors or undesirable properties that propagate through self-training — applies equally to ReST^EM.

The consequence. If a non-trivial fraction of the "correct" solutions in the E-step dataset contain flawed reasoning, the M-step fine-tunes the model to imitate those flawed patterns. The model learns that certain reasoning errors are acceptable as long as they lead to the correct answer on the training problems. When deployed on new problems, these flawed reasoning patterns may lead to incorrect answers — but the model has been trained to produce them confidently because they were rewarded during self-training.

This is a particularly insidious failure mode because it is invisible to the reward function. The validation-set accuracy (which uses the same reward function) would not detect the degradation in reasoning quality — validation accuracy might even improve if the model gets better at producing the correct answer through flawed shortcuts. The BBH evaluation (Figure 9) shows no degradation on 23 reasoning tasks, but these tasks are diverse and may not specifically probe the kinds of reasoning errors that MATH self-training might amplify. The Hungarian exam result (Figure 10) is manually graded and might catch reasoning errors, but the paper reports only an aggregate score, not a qualitative analysis of reasoning quality.

What evidence exists in the paper. The paper provides no direct measurement of false positive rates in the E-step datasets — it does not manually inspect the generated solutions to estimate what fraction of "correct" solutions contain flawed reasoning. The preliminary finding about rationalization introducing false positives (Section 4) is reported qualitatively without quantitative data. The train-test gap analysis (Figure 4) is consistent with models learning spurious patterns (training accuracy rises while test accuracy plateaus), but this is indirect evidence — the gap could also be explained by memorization of problem-specific patterns that are genuinely correct, not by learning flawed reasoning.

Mitigation status. The paper does not address solution quality as a limitation or propose mechanisms for filtering solutions based on reasoning quality rather than just final-answer correctness. The per-problem cutoff of 10 solutions provides some implicit quality control (if a problem generates many correct solutions, only 10 are used, reducing the chance that all 10 are false positives), but this is a weak filter and is not justified in quality terms. The paper's suggestion that future work should "explore algorithmic improvements that reduce the gap to pass@K performance" (Section 6) does not address reasoning quality. A practitioner concerned about this issue would need to implement additional quality filters — such as requiring solutions to pass both answer-matching and a learned verifier, or manually inspecting a sample of generated solutions — at additional cost.

6.6 The Method Has No Mechanism for Improving on Problems Where the Base Model Generates Zero Correct Solutions — The "Very Hard" Problem Regime Is a Hard Ceiling

The assumption or constraint. ReST^EM can only learn from problems for which at least one of the $N$ temperature-sampled solutions in the E-step passes the binary reward check. If the base model generates zero correct solutions for a problem — regardless of how many samples are drawn — that problem contributes nothing to the training dataset for that iteration. If the fine-tuned model from the previous iteration also generates zero correct solutions for that problem in the next E-step, it remains absent from the training data. There is no mechanism in ReST^EM for the model to learn to solve problems that are initially beyond its capability: no rationalization (providing hints), no curriculum (starting with easier variants), no demonstration bootstrapping (seeding with human-written solutions for hard problems), and no reward shaping (giving partial credit for partially correct solutions).

The consequence. For any problem category where the base model's pass@1 is near zero — the "very hard" bin in Figure 8 (right), corresponding to questions where the base model's success rate at temperature 1.0 is below 25% — ReST^EM provides minimal or zero improvement. Figure 8 (right) confirms this: the "very hard" category shows the smallest relative gain from ReST^EM among all four difficulty levels. This is not because the model saturates (as with "easy" problems, where the base model already succeeds most of the time) but because the E-step generates few or no correct training examples for these problems.

This hard ceiling has two important implications. First, ReST^EM cannot expand the frontier of the model's capability — it can only amplify capabilities the model already possesses at some non-zero level. If the base model fundamentally cannot solve a class of problems (e.g., it lacks the mathematical knowledge for a particular proof technique, or it cannot reason about a specific type of code structure), no amount of self-training will enable it to do so. Second, the distribution of problems in the training set determines which capabilities get amplified. If certain problem types are underrepresented among the "solvable" problems (because the model fails on them), those capabilities will not improve, and may even degrade relative to other capabilities that are being amplified.

What evidence exists in the paper. Figure 8 (right) directly shows the difficulty-dependent improvement from ReST^EM. The "very hard" category (below 25% base success rate) shows an improvement from approximately 5% average success rate to approximately 10% — a 5 percentage point gain that is smaller in absolute terms than the gains for "medium" (50–75% base success, improving from roughly 62% to roughly 75%, a 13-point gain) and "hard" (25–50% base success, improving from roughly 37% to roughly 52%, a 15-point gain). The paper does not report how many training problems fall into each difficulty category, so it is unclear what fraction of the training set is in the "very hard" bin and thus contributes few or no training examples. The APPS overfitting results (Figure 3) are consistent with the model failing to improve on the hardest problems in the APPS training set — after the easy and medium problems are learned (iteration 1), the remaining headroom comes from hard problems that the model cannot generate correct solutions for, leading to stagnation and eventual regression.

Mitigation status. The paper does not propose any mechanism for addressing this limitation. Section 4 notes that STaR's rationalization approach (providing the correct answer as a hint) was tested in preliminary experiments and rejected because it introduced false positives. The paper does not explore alternative approaches: curriculum learning (training on easier problems first, then progressively including harder ones as the model improves), iterative refinement of near-correct solutions (giving partial credit), or seeding the initial E-step with a small number of human-written solutions for the hardest problems. Section 6 suggests that "future research in self-improvement in language models should focus on... algorithmic improvements that reduce the gap to pass@K performance," which gestures at the capability-ceiling problem without proposing a specific solution.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood not as introducing a fundamentally new paradigm — expectation-maximization for RL dates to Dayan and Hinton (1997), and self-training with model-generated data has a long lineage — but as resolving a specific, consequential empirical contradiction that had been blocking the field's confidence in synthetic data for large-model fine-tuning. Before this work, the evidence pointed in opposite directions: Yuan et al. (2023) showed that self-generated data provided diminishing returns with model scale on GSM8K, implying that larger models benefit less from self-training. This threatened to relegate synthetic data to a technique for small models on easy tasks — useful for academic benchmarks but irrelevant for frontier LLMs. ReST^EM demonstrates the opposite scaling trend on harder benchmarks (MATH, APPS): larger models benefit more, not less, and the resulting models outperform human-data fine-tuning by margins that grow with scale.

Resolving this contradiction reframes the conversation around synthetic data from "is it a viable substitute when human data is scarce?" to "under what conditions is it actually better than human data?" The paper's answer — model-generated data is better when the task is hard enough that the model's exploratory samples contain correct solutions it wouldn't generate greedily, but not so hard that it generates no correct solutions at all — provides both a diagnostic (check the base model's success rate on the task; if substantial mass falls in the 25–75% range, self-training will help) and a boundary condition (very hard problems where the base model produces zero correct solutions represent a hard ceiling that self-training alone cannot breach). This transforms self-training from a blind empirical gamble into a decision with testable preconditions.

The paper also provides a unifying conceptual framework that subsumes prior methods. Section 4's taxonomy — showing how Expert Iteration, STaR, RFT, IML, RWR, and RAFT are all special cases of the EM-for-RL template differing in their E-step implementation (greedy vs. sampling vs. search), M-step implementation (mini-batch vs. full-dataset, iterative vs. base-model reset), and reward definition (binary vs. real-valued) — is a genuinely useful contribution. It gives the field a shared language for reasoning about self-training algorithms rather than treating each as an isolated recipe. This taxonomic contribution means future self-training papers can position themselves relative to a known space of design choices (E-step strategy, M-step initialization, reward structure) rather than reinventing the conceptual wheel.

A more subtle contribution is the decoupling of data quality from model drift via the base-model reset. The paper demonstrates (Figure 7) that always fine-tuning from the base model — discarding the previous iteration's fine-tuned model except as a data generator — preserves generalization to held-out tasks while still benefiting from progressively better self-generated data. This finding has implications beyond self-training: it suggests a general principle that improved models should be used as data generators, not as initialization points for further fine-tuning, whenever the goal is to improve task performance without sacrificing generality. This principle could influence how practitioners approach multi-stage fine-tuning, distillation, and continual learning more broadly.

The paper also establishes overfitting to the problem set — not to the solution distribution — as the primary failure mode for iterative self-training. Figure 4's train-test divergence, with training accuracy continuing to climb while test accuracy plateaus (MATH) or regresses (APPS), is a crisp diagnostic: the binding constraint is the number of distinct training problems, not the volume of self-generated solutions. The smaller APPS dataset (2,342 problems) overfits by iteration 2, while MATH (7,500 problems) tolerates 3 iterations. This shifts the optimization target for practitioners: invest in collecting more diverse problems, not in generating more solutions per problem. The paper's finding that a single E-step with 3× more data underperforms three iterations with the standard budget (Section 5.3) adds nuance — iterative refinement of the data-generating policy matters, but only up to the point where the problem set is exhausted.

A notable gap in the paper's landscape impact is that it does not directly engage with the reward reliability question. The oracle-level binary rewards used for MATH and APPS (ground-truth answer matching, full test suite execution) are perfectly reliable — zero false positives, zero false negatives. This is not an accident; it is what makes ReST^EM work cleanly. But it also means the paper does not provide evidence about what happens when the reward is imperfect — a learned verifier, a noisy human label, a partial-credit scoring function. This leaves a crucial open question for the field: does ReST^EM amplify reward errors across iterations, or is it robust to moderate reward noise? Without answering this, the practical applicability of ReST^EM to domains beyond math and code — where perfect oracle rewards don't exist — remains speculative.

Follow-Up Research This Work Enables

Stress-testing ReST^EM with imperfect, learned reward models. The paper's cleanest result — that ReST^EM outperforms human-data SFT — relies on oracle binary rewards. The most important follow-up is to systematically degrade the reward signal and measure how ReST^EM's performance changes. A concrete experiment: train a learned verifier (process reward model or outcome reward model) on the MATH training set, use it as the reward function for ReST^EM (replacing ground-truth answer matching), and compare final pass@1 against both the oracle-reward ReST^EM and human-data SFT. Vary the verifier's accuracy (by training on different fractions of the data, or by adding calibrated label noise) to establish the relationship between reward reliability and self-training benefit. The key question: is there a threshold below which imperfect rewards cause ReST^EM to degrade performance (by amplifying false positives across iterations), and if so, what is that threshold? This experiment would determine whether ReST^EM can be applied to the large class of reasoning tasks where only learned verifiers exist.

Combining ReST^EM with search-based E-steps for harder problems. The paper identifies a hard ceiling: problems where the base model generates zero correct solutions contribute nothing to the training data, and ReST^EM cannot improve on them. An obvious extension is to replace temperature sampling in the E-step with a search procedure — beam search against a process reward model, tree-of-thought exploration, or best-of-N with majority voting serving as a pseudo-reward — to surface correct solutions that the model cannot generate with independent temperature samples alone. The EM framework already accommodates this: Expert Iteration (Anthony et al., 2017) uses Monte Carlo tree search as the E-step, and the paper explicitly notes this connection in Section 4. A concrete experiment: on MATH, bin problems by base-model pass@1, and for the "very hard" bin (below 25% success), use PRM-guided beam search in the E-step to generate correct solutions that temperature sampling misses. Measure whether ReST^EM iterations with this augmented E-step close the gap on very hard problems that the base ReST^EM cannot touch. A negative result — search does not help because even search cannot find correct solutions for genuinely out-of-capability problems — would clarify the fundamental limits of test-time compute for capability expansion.

ReST^EM as a data-generation engine for self-improvement loops. The paper's distillation experiment (Figure 6, right) hints at a broader use case: a large model running ReST^EM generates high-quality training data that can improve other models, including future versions of itself. A natural experiment: take a base model (Model v1), run ReST^EM to produce a fine-tuned model (Model v1+ReST^EM), use v1+ReST^EM to generate a large corpus of correct solutions on the training problems, and then fine-tune Model v2 (a future, better pretrained model) on this corpus. Compare against fine-tuning Model v2 on human data and on v2's own self-generated data. The hypothesis: data from a ReST^EM-improved model is higher quality than data from either the base model or from human annotators, and this quality benefit persists across model generations — making ReST^EM-generated data a reusable, model-agnostic training asset. If confirmed, this would change how organizations think about fine-tuning data: invest once in running ReST^EM with the best available model, then reuse the generated solutions across future model releases.

Quantifying when self-generated data outperforms high-quality human data, not just benchmark reference solutions. The paper's comparison against human data uses the reference solutions bundled with MATH and APPS — single solutions per problem of unknown pedagogical quality. A fairer test: commission multiple expert-written solutions per problem (at least 3–5, from mathematicians or competitive programmers), ensure they represent diverse reasoning approaches, and compare ReST^EM against fine-tuning on this enriched human dataset. The question is whether the advantage of model-generated data over human data persists when the human data is produced with comparable diversity and volume, or whether the paper's result is primarily a demonstration that multiple solutions (whether human or model) outperform single solutions. If expert-written diverse solutions match or exceed ReST^EM, the "model-generated data is inherently better" narrative weakens; if ReST^EM still outperforms, the in-distribution hypothesis (model-generated solutions match the model's own generation patterns) gains strong evidence.

Scaling ReST^EM to an order-of-magnitude more model scales to establish genuine scaling laws. The paper compares exactly two model sizes per task — a directional observation, not a scaling law. A proper scaling analysis would run ReST^EM on at least 4–5 model sizes spanning 1–2 orders of magnitude in parameter count (e.g., 1B, 7B, 30B, 100B, 400B), fit a curve to the absolute improvement vs. model size, and test whether the trend is monotonic, logarithmic, or U-shaped. The key hypothesis to test: does the "larger models benefit more" trend continue indefinitely, or does it saturate or reverse when the base model's capability becomes so high that the remaining errors are on problems where even temperature sampling produces zero correct solutions? The paper's own difficulty-bin analysis (Figure 8, right) predicts a reversal at very high baseline performance (the "easy" bin shows smaller gains than "medium" because the model already solves most problems), which would imply an inverted-U relationship between model capability and ReST^EM benefit. A large-scale scaling study would confirm or falsify this prediction.

Applying ReST^EM to code generation with execution-based reward shaping. The current APPS experiments use a binary reward: pass all test cases or not. For code generation, a richer reward signal is available: the number of test cases passed, the type of error (compilation vs. runtime vs. wrong output), or execution traces. A natural extension of ReST^EM would use this finer-grained signal in the M-step: solutions that pass 8/10 test cases get higher weight than solutions that pass 2/10, even if neither is "correct" under the strict binary reward. This is a straightforward instantiation of the reward-weighted regression objective (Equation 3) with non-binary rewards — the framework already supports it, but the paper only tests binary rewards. The experiment: on APPS, use the fraction of test cases passed as the reward weight in the M-step, and measure whether this (a) improves final pass@1 compared to binary filtering, (b) reduces overfitting by providing more training signal per problem, and (c) enables learning on hard problems where the model never generates a fully correct solution but does generate partially correct ones. A positive result would substantially expand the applicability of ReST^EM beyond tasks with perfect oracle rewards.

Practical Applications and Downstream Use Cases

Automated fine-tuning pipelines for coding assistants. Organizations deploying coding assistants (GitHub Copilot, internal code generation tools) face a perpetual data problem: they want the model to specialize on their internal codebase, coding conventions, and common problem patterns, but writing high-quality training demonstrations for every API, framework, and coding pattern is prohibitively expensive. The APPS results suggest a scalable alternative: collect a set of coding problems representative of the deployment domain (this requires human effort, but writing problem statements is far cheaper than writing full solutions with test cases), define automated correctness checks (unit tests, compilation checks, runtime behavior assertions), and run ReST^EM to generate diverse correct solutions for fine-tuning. The paper shows that on APPS, one ReST^EM iteration improves pass@1 by ~6 percentage points — a substantial gain from a process that requires no human-written solutions beyond the initial problem set. The HumanEval transfer result (Figure 3 right) suggests the fine-tuned model's general coding ability is preserved or improved, not degraded.

Math education and tutoring systems. MATH-trained ReST^EM models show strong transfer to both easier math benchmarks (GSM8K, ~73% for PaLM 2-L after ReST^EM, Figure 2 right) and real-world exam settings (Hungarian HS Finals, ~54% and outperforming specialized math models, Figure 10). For an educational technology company building a math tutoring system, this transfer profile is valuable: fine-tune on competition-level MATH problems (where ground-truth answers are available for automated reward), and the resulting model generalizes to grade-school word problems and national exam questions — exactly the range of difficulty a tutoring system encounters. The positive transfer from harder to easier problems (MATH → GSM8K) suggests a curriculum design principle: collect problems at the upper bound of difficulty, and self-training will pull up performance on easier variants as well. The BBH results (Figure 9) further suggest the model does not become a narrow math specialist — general reasoning capabilities are preserved or slightly improved.

Distillation from large proprietary models to smaller deployable models. The distillation experiment (Figure 6, right) demonstrates that PaLM 2-L-generated solutions, when used to fine-tune PaLM 2-S, outperform both the smaller model's own self-generated data and human-written solutions. For organizations with access to a very large proprietary model (e.g., GPT-4, Claude, Gemini) but whose deployment constraints (latency, cost, on-device requirements) demand a smaller model, this recipe is directly actionable: (1) use the large model to generate many solutions per problem on a task-specific problem set, (2) filter with an automated reward function, (3) fine-tune the small model on this filtered dataset. The Distill* result — one solution per problem already beats human-data SFT — means the approach works even when generation budget is limited. The gap between Distill* (~22%) and full Distill (~24–25%) quantifies the additional value of multi-solution diversity, giving practitioners a concrete tradeoff: more generation budget buys further improvement, but most of the gain comes from the first few correct solutions per problem.

Efficient specialization for niche domains where human annotation is the primary cost. Consider a domain like patent law, organic chemistry synthesis planning, or hardware description language programming — tasks where the pool of qualified human annotators is small, their time is expensive, and the demand for training data outstrips supply. ReST^EM's sample efficiency (Figure 8 left: 1,000 MATH questions already yield substantial gains) means the initial human investment can be modest: collect ~1,000 representative problems with verifiable correctness criteria. The model then generates its own training data at scale, with no further human annotation needed. The per-problem generation cost (32 forward passes per iteration) is trivial compared to the cost of hiring domain experts to write equivalent solution diversity. The paper's finding that model-generated solutions are more in-distribution and therefore potentially better training targets than human-written solutions means the synthetic data may not just be cheaper — it may produce a better final model for the specific deployment distribution.

When to Prefer This Method

The paper provides a clear set of preconditions and tradeoffs that enable a decision rule for when ReST^EM is likely to be the right choice. It does not frame this as an explicit decision matrix, but the experimental results and identified limitations collectively define the applicability boundaries:

ReST^EM is the preferred approach when:

  • The task has an automated, reliable binary reward function — ideally oracle-level (ground-truth answers, comprehensive test suites), but at minimum with a false positive rate low enough that incorrect solutions don't contaminate the training data across iterations. Math with answer-matching, code with test suites, and any domain with formal verification meet this bar. Tasks where correctness is subjective (summarization, creative writing) or where any automated reward is noisy (learned verifiers with unknown error rates) are risky without additional reward-quality validation.
  • The base model has non-trivial but imperfect capability on the task — specifically, a substantial fraction of problems fall in the 25–75% success rate range at temperature sampling (the "medium" and "hard" bins in Figure 8 right, which show the largest ReST^EM gains). If the model's baseline is near 0% (the "very hard" regime), ReST^EM provides minimal benefit because the E-step generates no correct training data. If the baseline is near 100% (the "easy" regime), there is little headroom for improvement and gains are small.
  • Human-written solutions are scarce, expensive, or nonexistent for the target task — which is the regime ReST^EM was designed for. If high-quality, diverse human demonstrations are already available in volume, the paper does not demonstrate that ReST^EM would outperform fine-tuning on that enriched human dataset (the comparison is against single-reference solutions, not curated multi-demonstration sets).
  • The training problem set is reasonably sized (at least ~1,000–2,000 distinct problems based on Figure 8 left and the APPS overfitting result) and representative of the deployment distribution. Smaller problem sets risk overfitting within 1–2 iterations, and problem sets that don't cover the deployment distribution will produce models that appear to improve on validation (drawn from the same biased distribution) while degrading on real-world inputs.
  • Preserving general capabilities is important — the base-model reset in ReST^EM (vs. iterative fine-tuning in ReST) provides strong evidence of generalization preservation (Figure 7, Figure 9), making it suitable when the fine-tuned model must remain useful across a broad range of tasks, not just the training task.

ReST^EM should be deprioritized (or augmented) when:

  • The task lacks any automated correctness signal, and building a learned reward model would require substantial human annotation effort — at which point the human effort might be better spent writing solution demonstrations directly. The paper provides no evidence on how ReST^EM performs with imperfect learned rewards.
  • The base model's capability on the task is near zero — ReST^EM cannot create capability from nothing. Alternative approaches (further pretraining, architectural improvements, retrieval augmentation, or human-demonstration seeding) are necessary to establish a non-zero baseline before self-training can amplify it.
  • The training problem set is very small (fewer than ~1,000 problems) and cannot be expanded — single-iteration overfitting will limit gains, and the multi-iteration benefit that distinguishes ReST^EM from RFT will be unavailable.
  • The deployment requires very fast iteration or the training problem set changes frequently — each ReST^EM iteration requires generating solutions from a fine-tuned model (which itself requires a prior fine-tuning step), making the full pipeline sequential and relatively slow compared to single-shot SFT on available human or synthetic data.