ArXiv: 2403.04642

🎯 Pitch

Expert Iteration, a simple rejection-sampling method, matches or beats PPO for improving LLM math reasoning while using the same number of samples—but RL fine-tuning doesn't teach models to solve new problems beyond what supervised fine-tuning already enables in a best-of-96 setting.

1. Executive Summary

This paper studies how reinforcement learning algorithms improve the reasoning capabilities of large language models, evaluating Proximal Policy Optimization (PPO), Expert Iteration (EI), and Return-Conditioned RL (RCRL) on GSM8K and SVAMP math word problem benchmarks using instruction-tuned Llama-2 7B and 13B models. The central finding is that all three algorithms perform comparably, but Expert Iteration — which constructs an optimal policy approximation by rejection sampling the student policy — attains the best performance in most configurations while requiring at most on the order of 10^6 samples to converge, roughly matching PPO's sample complexity. The paper demonstrates that RL fine-tuning simultaneously improves both maj@1 and pass@96 metrics (unlike supervised fine-tuning, where continued training increases maj@1 at the expense of pass@96), establishing that online RL's self-generated training data prevents the output diversity collapse observed with static SFT datasets. Crucially, the authors conclude that models fail to explore significantly beyond solutions already produced by SFT models during RL training — meaning that while RL fine-tuning reliably boosts greedy decoding accuracy, it does not expand the fundamental range of solvable problems beyond what light supervised fine-tuning already achieves under a best-of-96 sampling budget.

2. Context and Motivation

The Core Problem: We Don't Know Which RL Algorithm to Use for LLM Reasoning

The fundamental question this paper tackles is deceptively simple: if you want to use reinforcement learning to improve an LLM's reasoning capabilities, which RL algorithm should you choose, and why? The paper opens (Section 1) by observing that the reasoning abilities of LLMs are rapidly improving on math, science, and code benchmarks, while simultaneously, RLHF has become the dominant paradigm for aligning LLM outputs with human preferences. These two trends naturally point toward a synthesis: use RL to improve reasoning. But the field lacks a principled understanding of how to do this — which algorithms work, under what conditions, and what bottlenecks limit further progress.

This gap matters for several reasons that the paper develops:

The RL-for-reasoning space is combinatorially large. Any attempt to apply RL to LLM reasoning must make choices across multiple axes: the RL algorithm itself (PPO, Expert Iteration, decision transformers, etc.), the reward structure (sparse vs. dense, heuristic vs. learned), the model initialization (pretrained vs. SFT, model size), and the training data regime (with or without supervised fine-tuning data). Prior work explored individual points in this space — for example, Yuan et al. (2023) applied a single round of Expert Iteration on GSM8K, Luo et al. (2023) used PRM rewards with PPO, and Zelikman et al. (2022) used a STaR-like approach — but each study varied multiple factors simultaneously, making it impossible to attribute success or failure to any single choice. The paper explicitly names this problem:

"Despite all the above work, it remains unclear exactly what factors account for the biggest impact during RL fine-tuning due to wide variance in tasks, pretraining data, supervised fine-tuning data, RL algorithm used, and the reward source."

Without a controlled comparison holding most factors constant and varying only the dimension of interest, the literature offers a collection of data points without a coherent picture.

The pretraining prior changes everything. Classical RL successes — AlphaGo, AlphaStar, OpenAI Dota 2 — trained agents from scratch in environments where exploration in high-dimensional action spaces required sophisticated algorithms and millions of environment interactions. LLM fine-tuning is fundamentally different: the model starts from a pretrained checkpoint that already encodes substantial knowledge about language, reasoning patterns, and problem structure. This pretraining prior acts as an extremely strong inductive bias on the kind of exploration the model will engage in. The paper hypothesizes — and later provides evidence — that this prior dominates the training dynamics to such an extent that the choice of RL algorithm may matter far less than in traditional RL settings. Understanding whether this hypothesis is correct has direct practical implications: if simpler algorithms work just as well, why pay the implementation and computational overhead of PPO?

The SFT tradeoff is a known but unexplained problem. Cobbe et al. (2021) previously identified a troubling phenomenon during supervised fine-tuning on math reasoning: continued training on GSM8K's SFT dataset improved greedy decoding accuracy (maj@1) but reduced the model's pass@96 performance — that is, the model got better at producing the right answer on the first try, but worse at generating correct solutions when sampled many times. The paper identifies the root cause as dataset diversity collapse: SFT trains on a static dataset, so as training continues, the model overfits to the specific solution patterns in that dataset, losing the ability to generate diverse alternative solution paths. This matters because real-world deployments often use test-time compute strategies (majority voting, verifier reranking) that depend on generating multiple diverse candidate solutions. An algorithm that improves maj@1 while preserving or improving pass@96 would be strictly superior — but prior work provided no such algorithm nor any understanding of why RL might help.

The role of exploration in LLM fine-tuning is poorly understood. Classical RL theory emphasizes exploration as the mechanism by which agents discover novel, high-reward behaviors. But what does "exploration" mean for a pretrained LLM fine-tuning on reasoning tasks? The model already has a strong prior over reasonable solution approaches. Does RL cause the model to genuinely explore novel reasoning strategies it wouldn't have discovered through simple SFT? Or does it merely reinforce existing capabilities, making the model more reliable at producing solutions it already "knows"? The answer has profound implications: if exploration is limited to the model's pretraining neighborhood, then RL fine-tuning can at best amplify existing reasoning capability — it cannot teach genuinely new reasoning patterns. This would place a fundamental ceiling on what RL can achieve for LLM reasoning, redirecting research toward better pretraining or better prompting rather than more sophisticated RL algorithms.

Conflicting Signals from Prior Work

The paper is motivated by a genuine tension in the literature regarding what works for LLM fine-tuning.

PPO is the dominant RLHF algorithm, but is it necessary? RLHF as practiced by Ouyang et al. (2022) and Bai et al. (2022) uses PPO as the default policy optimization algorithm. PPO's clipped objective and KL-constraint provide theoretical protection against policy collapse and reward over-optimization. However, a growing body of work — ReST (Gulcehre et al., 2023), Reward-Ranked Fine-tuning (Dong et al., 2023), AlpacaFarm (Dubois et al., 2023) — demonstrates that simpler approaches involving fine-tuning on high-reward responses with standard cross-entropy loss can achieve comparable performance on RLHF tasks. The paper groups these under the umbrella term Expert Iteration. This raises a critical question: if Expert Iteration suffices for RLHF, does it also suffice for reasoning? And conversely, does PPO's protective machinery provide any advantage when the reward signal is a clean ground-truth correctness check rather than a noisy learned reward model?

Expert Iteration works well in isolated studies, but its limits are unknown. Yuan et al. (2023) applied a single round of Expert Iteration across multiple model sizes on GSM8K, observing sizeable gains for smaller models that diminished for larger ones. Shen et al. (2023) reported a dramatic jump in StarCoder's maj@1 performance after one round of Expert Iteration (from ~30% to ~60%). These results are impressive but raise questions the paper seeks to answer: How many rounds of Expert Iteration are beneficial before performance saturates? Does Expert Iteration improve pass@96 or only maj@1? How does its sample complexity compare to PPO's? Is its performance sensitive to the quality of the initial policy? The answers determine whether Expert Iteration is a one-shot trick or a general-purpose algorithm for reasoning improvement.

Return-conditioned RL is theoretically appealing but empirically uncertain. Decision Transformers (Chen et al., 2021) and related return-conditioned approaches offer an elegant formulation: instead of optimizing a policy through trial-and-error RL, simply train a model to generate actions conditioned on desired returns. At test time, condition on the highest return to get optimal behavior. Brandfonbrener et al. (2022) showed this works well in deterministic environments — which reasoning tasks are — but prior work had not evaluated whether return-conditioned RL could match or exceed PPO and Expert Iteration for LLM reasoning, or whether the conditioning signal would be reliably interpretable by the model.

ORM and dense rewards: intuitively helpful, but are they? Cobbe et al. (2021) demonstrated that training Outcome-Based Reward Models (ORMs) to verify solution correctness and using them to rerank candidate solutions significantly boosts performance. It seems natural that ORM scores could also serve as dense reward signals during RL training, providing learning signal at intermediate steps rather than only at the final answer. Similarly, if ground-truth intermediate steps are available (as in the GSM8K dataset), heuristic dense rewards could guide the model step-by-step. Prior work had not systematically tested whether these intuitively helpful augmentations actually improve final performance, or whether the sparse ground-truth correctness signal alone is sufficient.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

No controlled head-to-head comparison. The single most important gap the paper identifies is the absence of a study that varies only the RL algorithm while holding constant the model architecture, dataset, reward source, and evaluation protocol. Without such a comparison, statements like "PPO outperforms Expert Iteration on RLHF" (or vice versa) are confounded by differences in prompt distributions, reward model quality, KL penalty tuning, and dozens of other implementation details. The paper positions itself as filling this gap by running all algorithms on identical hardware with identical base models and identical evaluation metrics.

The exploration question is unasked. Most prior work on RL for LLMs focuses on final task performance — does the model get better at solving GSM8K problems? — without asking why the improvement occurs. Is the model learning new reasoning strategies, or is it simply becoming more consistent at strategies it already possesses? The paper argues that answering this question requires looking at multiple metrics simultaneously (maj@1, pass@96, rerank@96, maj@96) and tracking solution diversity throughout training. Prior work typically reports only one or two of these metrics, making it impossible to distinguish between "genuine capability improvement" and "reliability improvement on existing capability."

Sample complexity is rarely measured or compared. In classical RL, sample complexity is a primary axis of comparison between algorithms. In LLM fine-tuning, most papers report final performance after a fixed amount of training without measuring how many environment interactions (model rollouts) were needed to reach that performance. This matters enormously for practical deployment: if Expert Iteration requires 10^7 samples to match PPO's performance at 10^5 samples, then PPO is preferable despite its implementation complexity. The paper explicitly designs experiments to measure and compare sample complexity across algorithms, providing the first systematic data on this dimension.

The SFT initialization question is underexplored. When applying RL to LLMs, one can start from a pretrained checkpoint with only a few-shot prompt (no SFT data), from a lightly SFT checkpoint, or from a heavily SFT checkpoint. Each choice affects the initial policy's quality and diversity, which in turn affects exploration. Prior work typically picks one starting point and reports results, without studying how the choice of initialization interacts with the choice of algorithm. The paper runs all algorithms from both SFT and no-SFT initializations to map out this interaction.

How This Paper Positions Itself

The paper frames its contribution not as proposing a new algorithm, but as providing the first comprehensive, controlled study of RL algorithms for LLM reasoning — an empirical investigation into what works, what doesn't, and why (Section 1). It positions itself at the intersection of three research traditions:

  1. The RLHF literature (Ouyang et al., 2022; Bai et al., 2022; Stiennon et al., 2020): The paper adopts PPO, Expert Iteration, and reward model training from this tradition but applies them to reasoning tasks with ground-truth rewards rather than learned human preference models. This allows the paper to study algorithm behavior without the confounding variable of reward model quality.

  2. The LLM reasoning literature (Cobbe et al., 2021; Lewkowycz et al., 2022; Zelikman et al., 2022): The paper uses GSM8K and SVAMP as standard benchmarks and adopts the verifier training methodology from Cobbe et al. (2021), positioning its results within the established math reasoning evaluation framework.

  3. The classical RL literature (Schulman et al., 2017; Anthony et al., 2017; Brandfonbrener et al., 2022): The paper imports not just algorithms but conceptual frameworks — particularly the distinction between online/offline RL and the role of deterministic vs. stochastic dynamics — using them to explain why certain algorithms perform similarly on reasoning tasks despite differing dramatically in traditional RL benchmarks.

A key part of the paper's positioning is its agnosticism about which algorithm is "best" — it explicitly sets out not to advocate for any single method but to understand the landscape. This is reflected in the study design, which systematically sweeps across algorithms, reward types, model sizes, and initializations. The paper's central thesis is that the answer to "which RL algorithm should I use for LLM reasoning?" is "it matters less than you think" — that the pretraining prior and the fundamental limits on exploration during fine-tuning dominate the choice of algorithm, making simple approaches like Expert Iteration competitive with or superior to more complex ones like PPO.

The paper also positions its exploration-related findings as having implications beyond reasoning tasks. The observation that models fail to explore significantly beyond their SFT-trained behavior suggests a fundamental limitation of current RL fine-tuning approaches: they can amplify existing capabilities but may not teach models fundamentally new reasoning patterns. This connects the paper's contributions to broader debates about whether scaling laws for pretraining versus fine-tuning follow similar patterns, and whether RL fine-tuning can ever substitute for larger-scale pretraining on hard reasoning problems.

3. Technical Approach

3.1 Reader Orientation

The paper constructs a controlled experimental framework — not a single deployable system — that evaluates multiple reinforcement learning algorithms (PPO, Expert Iteration, and Return-Conditioned RL) on math reasoning tasks while systematically varying the reward structure, model size, and initialization to isolate which factors drive improvements. It solves the problem of comparative algorithm evaluation under confounded variables by running all algorithms on identical hardware with identical base models (Llama-2 7B and 13B), identical evaluation metrics (maj@1, maj@96, rerank@96, pass@96), and identical reward sources (sparse ground-truth, ORM, dense heuristic), so that differences in performance can be attributed to the algorithm and reward choices rather than implementation artifacts.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework has five major components:

  1. Base LLMs — instruction-tuned Llama-2 7B and 13B models that serve as the starting point for all algorithms. These can be initialized from pretrained checkpoints with few-shot prompts or from supervised fine-tuned (SFT) checkpoints trained on GSM8K data.

  2. RL Algorithms — three distinct policy improvement procedures (PPO, Expert Iteration, Return-Conditioned RL) that consume rollout data from the student policy and produce updated model parameters. Each algorithm defines its own exploration strategy, data collection protocol, and parameter update rule.

  3. Reward Sources — four types of reward signals provided to the algorithms: sparse ground-truth correctness (+1 for correct final answer, 0 otherwise), ORM-predicted correctness (a learned classifier's probability estimate), dense heuristic rewards (comparing intermediate steps to reference solutions), and dense ORM rewards (the ORM's predictions at intermediate steps).

  4. ORM Verifier — a separately trained outcome-based reward model that predicts the probability of reaching a correct final answer given a question and a prefix of solution steps. Used both as a reward source during RL training and as a reranker at evaluation time.

  5. Evaluation Suite — a protocol computing four metrics on a held-out test set: maj@1 (greedy single-sample accuracy), maj@96 (majority vote over 96 samples), rerank@96 (ORM-ranked best of 96 samples), and pass@96 (oracle best of 96 samples). Together these distinguish "reliability improvements" from "capability expansions."

Information flows as follows: a prompt enters from the GSM8K or SVAMP training set → the active RL algorithm samples the student policy (using algorithm-specific temperature and sampling parameters) to generate solution rollouts → the chosen reward source scores each rollout (per-step or final-answer) → the RL algorithm updates the student parameters using its specific loss function (PPO's clipped surrogate objective, EI's cross-entropy on filtered correct solutions, or RCRL's return-conditioned cross-entropy) → the updated student is evaluated on the test set periodically → the best checkpoint is selected via validation performance.

3.3 Roadmap for the Deep Dive

  • First, the MDP formulation of reasoning tasks — because every RL algorithm presupposes a formal decision process, understanding how token generation maps to states, actions, and rewards is foundational.
  • Second, Proximal Policy Optimization (PPO) — the most widely used RLHF algorithm and the paper's primary online RL baseline, whose architecture, objective function, and hyperparameters need thorough exposition.
  • Third, Expert Iteration (EI) — the simplest algorithm that performs best in most settings; understanding its rejection-sampling-based data construction and cross-entropy distillation explains why it achieves competitive sample complexity despite its algorithmic simplicity.
  • Fourth, Return-Conditioned RL (RCRL) — the offline algorithm that conditions on desired return tokens; understanding its training data construction and why it underperforms reveals constraints on what information models can usefully extract from negative demonstrations.
  • Fifth, the Outcome-Based Reward Model (ORM) — the shared verifier component used across all algorithms as both a reward source and an evaluation tool; understanding its training procedure and score aggregation is necessary for interpreting rerank@96 results.
  • Sixth, the reward sources — sparse, ORM, and dense rewards — because each algorithm can be paired with any reward, and understanding which combinations work (and why dense rewards sometimes hurt) explains key results.
  • Seventh, implementation details, hyperparameter choices, and their justifications — because RL training is notoriously sensitive to these choices, and the paper's ablations of architecture, sampling parameters, and KL constraints provide essential context for interpreting comparative performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical comparative analysis paper whose core idea is that in the pretrained-LLM fine-tuning regime, the specific choice of RL algorithm matters less than the quality and diversity of the exploration data, because the pretraining prior constrains exploration to a neighborhood of solutions the model already "knows."

Reasoning as an MDP: The Token-Level Decision Process

The paper frames math word problem solving as a Markov Decision Process (MDP) (Section 3, "Reasoning as an RL problem"). This formalization is necessary because RL algorithms operate over states, actions, transitions, and rewards — and mapping natural language generation onto this framework requires explicit definitions.

State space $\mathcal{S}$: The state at time $t$ is the concatenation of the question $Q$ and all tokens generated so far, $s_t = (Q, a_1, a_2, \ldots, a_{t-1})$. The state is the entire accumulated context that the model conditions on when predicting the next token.

Action space $\mathcal{A}$: An action $a_t$ is a single token from the model's vocabulary $\mathcal{V}$. At each step, the model samples a token from its policy distribution $\pi(a_t | s_t)$.

Transition dynamics $P_a$: The environment is deterministic — applying action $a_t$ in state $s_t$ deterministically produces state $s_{t+1} = s_t \oplus a_t$, where $\oplus$ denotes concatenation. There is no stochasticity in how the environment processes actions; the only randomness comes from the policy's sampling.

Reward function $R_a$: By default, the reward is sparse and based solely on the final answer:

R(sT)={+1if the final answer in sT is correct0otherwiseR(s_T) = \begin{cases} +1 & \text{if the final answer in } s_T \text{ is correct} \\ 0 & \text{otherwise} \end{cases}

where $s_T$ is the terminal state (the complete generated solution). All intermediate states receive zero reward. The paper also experiments with dense rewards (discussed in a later subsection) that assign values at intermediate steps.

The optimization objective: All RL algorithms aim to maximize the expected return of a student policy $\pi$ on the task distribution $\tau$:

EA∼π(Q),(Q,⋅)∈τ[R(A)]\mathbb{E}_{A \sim \pi(Q), (Q, \cdot) \in \tau} \left[ R(A) \right]

where $A = (a_1, a_2, \ldots, a_T)$ is a complete generated solution (action sequence), $\pi(Q)$ is the policy's output distribution conditioned on question $Q$, and $R(A)$ is the reward assigned to the complete solution.

Why this formulation matters: The deterministic transition dynamics are a crucial property that the paper later uses to explain why simpler algorithms (EI, RCRL) perform comparably to PPO. In deterministic environments, the optimal policy given the ground-truth reward can be learned effectively through filtering correct rollouts (as EI does) without needing the advantage estimation and clipped updates that PPO uses to stabilize learning in stochastic environments. Brandfonbrener et al. (2022) theoretically showed that return-conditioned supervised learning works well under deterministic dynamics — the paper empirically confirms this holds for LLM reasoning.

A subtle but important detail: while the MDP framing is standard in RL, directly applying it to token-level generation in LLMs creates a state space of size $|\mathcal{V}|^T$ — astronomically large. The fact that any learning occurs at all is entirely due to the pretrained model's strong prior over token sequences. Without this prior, exploration in such a space would be impossible; RL fine-tuning works because it operates in the much smaller subspace of "reasonable continuations" that the pretrained model assigns non-trivial probability.

Proximal Policy Optimization (PPO) for LLM Fine-Tuning

PPO is the paper's representative online, on-policy RL algorithm (Section 3, "Reasoning as an RL problem"). At each iteration, PPO samples rollouts from the current student policy $\pi_\theta$, computes advantages using a learned value function, and updates the policy parameters $\theta$ by maximizing a clipped surrogate objective. The training loop alternates between an exploration phase (data collection) and a policy improvement phase (gradient updates on collected data).

Exploration phase (data collection): The student policy $\pi_\theta$ is sampled on questions from the training set. The paper samples $N = 4$ rollouts per question, with a total of 1024 rollouts collected per iteration. The sampling temperature depends on the initialization: $T = 0.7$ when starting from an SFT checkpoint (to encourage diversity without collapse), and $T = 0.2$ when starting from a pretrained prompted model (to prevent degenerate outputs since pretrained models on math tasks produce nonsensical solutions at high temperatures). The paper also experiments with best K of N (KoN) sampling: generating $N$ rollouts per prompt, keeping only the $K$ highest-reward rollouts for training, and discarding the rest. In practice, $K = 4, N = 4$ works best — that is, keep all four rollouts, which is equivalent to standard sampling.

Value function architecture: PPO requires a value function $V_\phi(s)$ to estimate expected returns for advantage computation. The paper uses a joint architecture where the policy and value heads share the transformer trunk but have separate output branches. Three architectural choices proved critical:

  1. Large value branch: The value head consists of $L = 4$ transformer layers (rather than a simple linear projection). A smaller value branch caused instability because it couldn't accurately estimate returns, leading to noisy advantages.
  2. Detached value gradients: Gradients from the value loss are not allowed to flow back into the shared trunk. Without this detachment, value gradients interfered with policy gradients — a large value update could destabilize the policy representations because both heads competed over the same trunk parameters. The paper explicitly notes this observation matches Stiennon et al. (2020)'s finding in summarization RLHF.
  3. LoRA for stability: Low-Rank Adaptation (Hu et al., 2021) with rank $r = 128$ is applied to all weight matrices. Full-layer fine-tuning without LoRA proved unstable; LoRA constrains the parameter updates to a low-dimensional subspace, acting as implicit regularization.

Policy improvement phase (the PPO objective): After collecting 1024 rollouts, the paper runs $K = 4$ PPO epochs on this fixed batch, updating with mini-batches of size 256. The clipped surrogate objective is:

J(θ)=Et[min⁡(πθ(at∣st)πold(at∣st)A^t,    clip(1−ϵ,1+ϵ,πθ(at∣st)πold(at∣st))A^t)]J(\theta) = \mathbb{E}_t \left[ \min\left( \frac{\pi_\theta(a_t | s_t)}{\pi_{\text{old}}(a_t | s_t)} \hat{A}_t, \;\; \text{clip}\left(1 - \epsilon, 1 + \epsilon, \frac{\pi_\theta(a_t | s_t)}{\pi_{\text{old}}(a_t | s_t)}\right) \hat{A}_t \right) \right]

where:

  • $\pi_\theta(a_t | s_t)$ is the current policy's probability of action $a_t$ in state $s_t$
  • $\pi_{\text{old}}(a_t | s_t)$ is the policy's probability at the time the rollout was collected (the behavior policy)
  • $\hat{A}_t$ is the estimated advantage — the difference between the empirical return and the value function's prediction $V(s_t)$, computed using Generalized Advantage Estimation (GAE)
  • $\epsilon$ is the clipping threshold (typically 0.2 in standard PPO; the paper does not specify an explicit value but the standard default is used)
  • $\text{clip}(1 - \epsilon, 1 + \epsilon, r)$ clamps the probability ratio $r$ to the interval $[1 - \epsilon, 1 + \epsilon]$

What it computes: For each token $a_t$ in each rollout, the objective computes the probability ratio $r_t = \pi_\theta(a_t|s_t) / \pi_{\text{old}}(a_t|s_t)$ — how much more (or less) likely the current policy makes that token compared to when it was sampled. It then computes the standard policy gradient objective $r_t \hat{A}_t$ (which would be an unbiased estimator of the policy gradient if $r_t = 1$), but takes the minimum of this and a clipped version $\text{clip}(r_t) \hat{A}_t$. The result is a scalar per token that is maximized via gradient ascent on $\theta$.

Why this form: The minimum between the unclipped and clipped objective means the policy is only rewarded for probability changes in the direction of the advantage when the ratio stays within $[1 - \epsilon, 1 + \epsilon]$. If the ratio moves outside this interval, the clipped term becomes active and the gradient is zero (when the advantage is positive and $r_t > 1 + \epsilon$, meaning an already-increased probability would be increased further; or when the advantage is negative and $r_t < 1 - \epsilon$). This prevents catastrophic policy collapses where a single update makes the policy too different from the behavior policy, which would invalidate the importance sampling correction underlying the objective. In the LLM context, this is critical because the token distribution is over a vocabulary of tens of thousands — without clipping, the model could rapidly shift probability mass away from tokens it previously generated, causing it to produce nonsensical completions.

KL constraint: In addition to the clipped objective, the paper applies an explicit KL divergence penalty with coefficient $0.05$ between the current policy and the initial policy (the SFT or pretrained checkpoint). This penalizes the policy for deviating from the starting distribution:

Ltotal=J(θ)−β⋅KL(πθ∥πinitial)\mathcal{L}_{\text{total}} = J(\theta) - \beta \cdot \text{KL}(\pi_\theta \| \pi_{\text{initial}})

where $\beta = 0.05$. The paper notes this is "critical for preventing model collapse after more than a hundred gradient updates." Interestingly, they contrast this with Bai et al. (2022), who found KL constraints unnecessary for Constitutional AI dialogue tasks. The paper hypothesizes the difference arises because math reasoning involves "somewhat unnatural distribution of text" — broken natural language and computation tags (<<x+y=z>>) that are far from the pretraining distribution, making the model more prone to drift.

Training duration and other hyperparameters: Training runs for 4000 gradient steps with learning rate $1 \times 10^{-6}$ and batch size 256. The best checkpoint is selected based on validation set performance. When using an ORM for rewards (rather than sparse ground-truth), the ORM's probability estimate replaces the binary reward, providing a continuous signal.

Computational characteristics: PPO's main bottleneck is memory — maintaining the value head, storing rollout data, and computing the KL penalty all consume GPU memory, resulting in lower rollout throughput compared to Expert Iteration. The paper notes that despite requiring fewer total rollouts to converge (approximately 60,000), PPO training takes "about a day" — roughly the same wall-clock time as Expert Iteration — because the memory constraints force smaller mini-batches and slower per-sample processing.

Expert Iteration (EI)

Expert Iteration is the paper's representative online, off-policy RL algorithm (Section 3, "Reasoning as an RL problem"). Unlike PPO, which updates the policy using a surrogate RL objective, EI constructs an approximation to the optimal policy by repeatedly: (1) sampling the current student policy many times per question, (2) filtering out incorrect and duplicate solutions, and (3) fine-tuning the base pretrained model on the filtered dataset using standard cross-entropy loss. This process iterates with the newly fine-tuned model becoming the student for the next round.

The iterative procedure in detail:

Round 0 (initial exploration): If an SFT checkpoint is available, the first round uses the SFT model as the exploration policy. If starting from scratch, a few-shot prompt (two examples from the GSM8K validation set) is prepended to each question, and the pretrained model generates solutions. Regardless of the exploration source, the base pretrained model — not the SFT checkpoint — is used as the starting point for fine-tuning. The paper explicitly notes that resetting to the pretrained base model at each iteration (following Zelikman et al., 2022) is "crucial for achieving best performance." This prevents the model from compounding errors from previous iterations' approximations.

Exploration phase: For each question in the training set, the student policy $\pi_{i-1}$ (the model from the previous iteration) samples $K = 96$ complete solutions with temperature $T = 1.0$. The high temperature maximizes solution diversity within the constraint of not producing too many degenerate (nonsensical) outputs.

Dataset construction: The $K \times \text{num\_train}$ generated solutions are filtered to construct the training dataset $D_i$ for the next iteration:

  1. Reward filtering: Solutions with reward below a threshold $T$ are discarded. With sparse ground-truth reward, this means only correct solutions are kept (since $T > 0$ and only $+1$ and $0$ are possible rewards). The paper calls this "rejection sampling."
  2. De-duplication: Duplicate solutions (exact string matches) are removed to prevent the model from memorizing repeated outputs.
  3. Accumulation: The new filtered data $R_i$ is combined with all previous iterations' data: $D_i = R_i \cup D_{i-1}$. This means the training set grows over iterations, and the model continues to see correct solutions from earlier rounds.

Policy improvement phase (distillation): The pretrained base model is fine-tuned on $D_i$ using the standard autoregressive cross-entropy loss:

LEI=∑(Q,A)∈Di∑t=1∣A∣−log⁡πθ(at∣Q,a1,…,at−1)\mathcal{L}_{\text{EI}} = \sum_{(Q, A) \in D_i} \sum_{t=1}^{|A|} -\log \pi_\theta(a_t | Q, a_1, \ldots, a_{t-1})

where $(Q, A)$ is a question–answer pair in the filtered dataset and $|A|$ is the number of tokens in the solution.

What it computes: For each token in each correct solution in the dataset, the model predicts the next token's probability distribution and receives negative log-likelihood loss proportional to how much probability mass it assigned to the actual observed token. The loss is summed over all tokens and all (question, solution) pairs, then minimized via gradient descent on $\theta$.

Why this form: Cross-entropy on correct solutions is behavior cloning of the optimal policy (restricted to states visited by the current exploration policy). In the MDP framework, this is equivalent to maximizing the likelihood of trajectories generated by the expert policy. The key difference from standard SFT is that the "expert" here is the model's own previous iteration filtered to keep only high-reward outputs — the model is bootstrapping from its own successes. This works because the environment is deterministic: if the model can produce a correct solution once, that solution is a valid demonstration of optimal behavior for that question.

Fine-tuning hyperparameters: The same hyperparameters are used as in SFT training: 4 epochs, global batch size 128, initial learning rate $2 \times 10^{-5}$ decayed to $2 \times 10^{-7}$ with a cosine warmup schedule.

Iteration control: The process repeats until performance on a validation set saturates. For SFT-initialized models, this occurs after $n = 2$ iterations for both 7B and 13B models (Figure 2). For pretrained models without SFT data, convergence takes $n = 4$ iterations on GSM8K and $n = 5$ on SVAMP. After the first iteration with pretrained models, the few-shot prompt is removed — the generated SFT data from round 1 provides sufficient formatting signal.

Sample complexity analysis: The paper explores whether $K = 96$ samples per question is necessary. Reducing to $K = 4$ samples per question requires more iterations to converge (since each iteration provides less data) but uses far fewer total rollouts. With $K = 4$, EI has the same sample complexity as PPO on GSM8K, converging in accuracy "only a few percentage points lower" than $K = 96$. This suggests the large $K$ used in the main experiments oversamples each prompt and could be significantly reduced without major performance loss.

Why reset to the pretrained model each iteration? This is a non-obvious design choice with important consequences. If the model were fine-tuned sequentially — each iteration building on the previous fine-tuned model — errors from early filtering (e.g., correct solutions that happen to have stylistic quirks) would compound. By resetting to the pretrained base model and fine-tuning on the accumulated dataset of all correct solutions discovered so far, each iteration starts from a clean slate with a growing set of high-quality demonstrations. This is effectively an iterative data expansion strategy: the model discovers correct solutions, which expand the training set, which enables better fine-tuning, which enables discovering even more correct solutions. The process stops when the model can no longer discover new correct solutions beyond what is already in the dataset.

The ORM-guided Expert Iteration variant: When an ORM is available, the filter threshold $T$ can be set to a continuous value (e.g., $T = 0.5$), and solutions are filtered by ORM-predicted correctness probability rather than ground-truth binary reward. The paper finds this "improves sample efficiency but not performance" (Figure 3) — ORM-guided EI converges with fewer total rollouts but reaches the same final accuracy. This makes sense: the ORM provides a slightly noisier signal than ground-truth, but the filtering-and-distillation process is robust to moderate noise as long as high-scoring solutions are genuinely correct most of the time.

Return-Conditioned RL (RCRL)

Return-Conditioned RL is the paper's representative offline RL algorithm (Section 3, "Return Conditioned RL"). Unlike PPO and EI, which alternate between data collection and policy improvement, RCRL trains entirely on a pre-collected static dataset. The core idea is to train a policy $\pi_\theta(a_t | s_t, g_t)$ that conditions on both the current state and a return token $g_t$ indicating the desired outcome. At test time, the best performance is obtained by conditioning on the highest possible return token.

The training objective:

LRCRL=∑τ∈D∑t=1H−log⁡πθ(at∣st,gt)\mathcal{L}_{\text{RCRL}} = \sum_{\tau \in D} \sum_{t=1}^{H} -\log \pi_\theta(a_t | s_t, g_t)

where $\tau = ((s_t, a_t, g_t))_{t=1}^{H}$ is a trajectory of state-action-return tuples drawn from the training dataset $D$, and $H$ is the trajectory length.

What it computes: For each token in each trajectory in the dataset, the model predicts the next token's probability distribution conditioned on the current state AND the return token. The loss is the negative log-probability of the actual action, summed over the trajectory. The return token remains constant throughout a trajectory and is prepended to the context — in practice, it is represented as a natural language token: "[GOOD]" for high-return solutions and "[BAD]" for low-return solutions.

Why this form: This is behavior cloning augmented with a conditioning variable. The hypothesis is that the model can learn to associate the "[GOOD]" token with solution patterns that lead to correct answers and the "[BAD]" token with patterns that lead to incorrect answers. At test time, conditioning on "[GOOD]" should bias generation toward correct solutions. The approach is closely related to Decision Transformers (Chen et al., 2021) but simplified — there is no return-to-go calculation, just a binary good/bad label.

Training data construction (the critical design challenge): The paper constructs $D$ by sampling solutions from the best EI-trained policy $\pi_{\text{EI}}$. For each question $Q$, the EI policy generates complete solutions. To produce per-step return labels, the paper uses a Monte Carlo estimation procedure:

  1. For each intermediate prefix $P_i = (S_1, \ldots, S_i)$ (the first $i$ steps of a solution), sample the EI policy $K$ times to generate completions from $P_i$ onward.
  2. For each completion, evaluate the binary correctness label $l_k \in \{0, 1\}$ based on whether the final answer matches the ground truth.
  3. Compute the average return for this prefix: $\frac{1}{K} \sum_{k=1}^{K} l_k$.
  4. Label the step $S_i$ as "[GOOD]" if the average return exceeds a threshold $T = 0.5$ and "[BAD]" otherwise.

This procedure estimates the value function for each prefix — the probability that the EI policy can reach a correct answer starting from that prefix. The paper calls these models "stepwise ORMs" or SORMs in Appendix F, distinguishing them from Process Reward Models (PRMs) because they estimate future success probability rather than step-level independent correctness.

A crucial property: A prefix with a mistake at step $j < i$ but a valid step $i$ will have $V^*(Q, P_i) = 0$ (the mistake makes correct completion impossible) but a PRM would rate step $i$ as correct in isolation (since the mistake was at step $j$, not step $i$). The paper's SORM captures this distinction — it labels such steps "[BAD]" because subsequent rollouts cannot recover from the earlier mistake.

Label balance experiments: By default, the training dataset is balanced between "[GOOD]" and "[BAD]" labeled steps. An ablation (Appendix A, Table 5) shows that balanced labels yield the worst performance. Increasing the proportion of positive data improves results, with best performance when positive samples greatly outweigh negative samples. This suggests the 7B model does not effectively learn from negative demonstrations — it cannot usefully incorporate information about what constitutes a "[BAD]" step. The paper hypothesizes that either a larger model or an easier task would be needed for negative examples to provide meaningful learning signal.

An interesting failure mode: When sampling the RCRL model unconditionally (without specifying "[GOOD]" or "[BAD]"), the model often generates perfectly valid steps following a "[BAD]" label that still result in a correct final answer. This indicates the model has not truly learned to distinguish good from bad steps — it has learned that "[BAD]" is just another token to be followed by plausible completions. This is a form of conditioning collapse: the model treats the return token as part of the natural language context but does not internalize its semantic meaning (that "[BAD]" should correlate with incorrect solution patterns).

Outcome-Based Reward Model (ORM)

The ORM is a shared component used across all algorithms as both a training-time reward source and an evaluation-time reranker (Section 3, "Outcome Based Reward Modeling"). It is a classifier that takes a question $Q$ and a partial solution prefix $P_i$ and predicts the probability that the complete solution will be correct.

ORM training objective:

LORM=−(ylog⁡(p(is_correct(A)∣Q,Pi))+(1−y)log⁡(1−p(is_correct(A)∣Q,Pi)))\mathcal{L}_{\text{ORM}} = -\left( y \log(p(\text{is\_correct(A)} | Q, P_i)) + (1 - y) \log(1 - p(\text{is\_correct(A)} | Q, P_i)) \right)

where $y \in \{0, 1\}$ is the binary ground-truth label (whether the complete solution $A$ containing prefix $P_i$ is correct) and $p(\text{is\_correct(A)} | Q, P_i)$ is the ORM's predicted probability.

What it computes: Standard binary cross-entropy between the ORM's predicted correctness probability and the actual observed correctness. The model is trained on prefixes from both correct and incorrect solutions, learning to distinguish solution paths that will ultimately succeed from those that will fail. The predictions are made from prefixes at all possible truncation points in the solutions, not just final answers — this forces the ORM to learn intermediate indicators of correctness.

Training data generation: The ORM is trained on data generated by the student policy $\pi$. For each question $Q$ in the training set, the student samples $K$ solutions. Each solution $A = (S_1, \ldots, S_L)$ is then decomposed into all possible prefixes $P_i = (S_1, \ldots, S_i)$ for $i = 1, \ldots, L$. Each prefix is labeled with the binary ground-truth correctness of the complete solution $A$. The pool of all such (question, prefix, label) tuples across all questions forms the ORM training dataset.

A subtle property of ORM labels: If a solution is incorrect, all its prefixes receive a label of 0 — even if the first several steps were correct and the mistake occurs only at the final step. This means the ORM's prediction for a "correct so far" prefix from an ultimately-incorrect solution should be low, because the ORM learns that solutions with this prefix tend to fail. The ORM is thus trained to estimate the conditional probability of success given a prefix, marginalizing over the student policy's completion distribution, rather than the independent correctness of the prefix itself.

Usage as reranker (rerank@96): At evaluation time, the ORM scores 96 candidate solutions per question and selects the one with the highest predicted correctness probability. This is analogous to the verifier reranking in Cobbe et al. (2021), but trained specifically on the model's own output distribution.

Usage as reward source during RL: When an ORM is used as the reward during PPO or EI training, its predicted probability replaces the binary ground-truth reward. This provides a continuous, dense signal — even incorrect solutions receive a probability estimate, allowing the model to distinguish between "almost correct" and "completely wrong" solutions. However, the paper finds this ORM reward signal introduces noise: the ORM is imperfect and can assign high probability to solutions that are actually incorrect (false positives), or low probability to solutions that happen to be correct but use an unusual reasoning pattern (false negatives). The paper notes that ORM-guided PPO and EI sometimes match or slightly underperform sparse ground-truth reward counterparts, suggesting that at the model scales studied (7B-13B), ORM noise outweighs the benefit of a continuous signal.

Potential overfit concern: When evaluating rerank@96, the paper uses an ORM trained on samples from the best EI model. This creates a potential confound: the ORM may be overfit to the EI model's output distribution, inflating rerank@96 scores for EI models relative to PPO models. The paper acknowledges this (Section 4.1): "we believe a non-trivial percentage of this gap is due to overfit of the ORM to the EI model which was used to generate its training data." The rerank@96 gap between EI and other models (approximately 5%) should therefore be interpreted as an upper bound on genuine reranking quality difference.

Reward Sources and Structures

The paper experiments with four qualitatively different reward sources, testing whether the algorithm performance rankings are robust to reward type (Section 4.1, "Extra guidance from ORMs or dense rewards provides little benefit").

1. Sparse ground-truth reward: The default. Reward is $+1$ if the final extracted answer matches the ground-truth answer, and $0$ otherwise. All intermediate tokens receive zero reward. This is the cleanest signal (no false positives or negatives) and is available because the training set includes ground-truth answers. The paper finds this simple reward is sufficient for all algorithms to achieve their best performance.

2. ORM reward: The ORM's predicted correctness probability $p(\text{is\_correct(A)} | Q, P_i)$ at the final step $L$ (or at all steps for dense ORM reward). This provides a continuous value in $[0, 1]$ rather than a binary. When used as a sparse reward, only the final step's ORM score is used. When used as a dense reward, each step $i$ receives the ORM's prediction from prefix $P_i$ as its reward.

For PPO, the ORM reward is integrated into the advantage estimation: the value function is trained to predict the ORM's assessments, and advantages are computed using ORM-derived returns. For EI, ORM scores replace binary correctness in the filtering step — solutions with ORM scores above some threshold are kept for training. For RCRL, ORM scores could be used instead of Monte Carlo return estimates (though the paper does not do this; RCRL uses EI rollouts with Monte Carlo labels).

3. Dense heuristic reward: When SFT data is available, the training dataset includes reference solutions with explicit intermediate steps. The paper extracts these reference steps and provides a reward of $+1$ at each step whose generated content matches the reference step, and $0$ otherwise. The paper finds this slightly harms performance relative to sparse reward (Section 4.1). They attribute this to overfitting: "Giving intermediate reward by comparing intermediate model generated steps to the reference solution... likely encourages more overfit to exact solutions in the train set, limiting solution diversity." The model learns to copy reference solutions rather than developing its own reasoning patterns.

4. Dense ORM reward: The ORM provides a score at every step (using the prefix up to that step). This can be used as a dense reward, providing learning signal at intermediate timesteps. The paper finds this provides no benefit to final converged performance compared to sparse ORM reward at the final step only (Figures 4, 14, 15, 16, 17). In some cases, dense ORM reward slightly reduces final accuracy compared to sparse ORM reward. The paper speculates that the intermediate ORM predictions are too noisy to provide useful guidance, and that the final-step signal is the dominant term in any case.

Why sparse ground-truth is sufficient: The paper's key finding on rewards is that the sparse ground-truth signal — despite providing no intermediate guidance — is sufficient for these tasks. This is explained by two factors: (1) the pretrained model already generates reasonable solution structures, so intermediate rewards are unnecessary for shaping; the model just needs to know which solutions are correct overall. (2) The trust region enforced by PPO's KL constraint and EI's filter-then-distill approach is narrow enough that the model does not need dense guidance to stay on track — it stays close to its initialization by construction.

Implementation Details and Design Choices

Beyond the algorithm-specific architectures described above, the paper documents several implementation choices that proved critical for stable training (Section 4.3, "Implementation Details").

PPO architecture and stability (Section 4.3, "PPO model architecture and training parameters"):

  • Large value branch ($L = 4$ transformer layers): Prevents value function underfitting, which would produce noisy advantages and unstable policy updates (Figure 7).
  • Detached value gradients: Prevents gradient interference between policy and value objectives in the shared trunk. Without detachment, "value gradients interfere with policy gradients... causing instability with a big update to either branch" (Section 4.3). This is a concrete architectural insight transferable to other PPO-for-LLM implementations.
  • LoRA rank $r = 128$: Full fine-tuning without LoRA proved unstable; LoRA constrains updates to a low-dimensional subspace, acting as implicit regularization. The paper notes this choice was informed by Sun et al. (2023).
  • Batch size 256, learning rate $1 \times 10^{-6}$: Larger batch sizes stabilize gradient estimates; the small learning rate prevents catastrophic updates.
  • KL penalty coefficient $\beta = 0.05$: "Critical for preventing model collapse after more than a hundred gradient updates." The paper contrasts this with Bai et al. (2022), who did not need KL constraints for dialogue RLHF. The difference is attributed to the "unnatural distribution of text" in math reasoning tasks (computation tags, broken natural language), which creates a larger gap between the pretraining and fine-tuning distributions, making the model more susceptible to drift.
  • Partial fine-tuning ablation: Fine-tuning only the top $M$ transformer layers (rather than all layers with LoRA) saves memory but costs "a few percentage points of performance," confirming that full-model fine-tuning is beneficial.
  • PPO epochs $K = 4$: Four epochs of PPO updates on the same rollout batch. More epochs risk overfitting to the batch (reducing diversity); fewer epochs leave optimization on the table.

Sampling parameters (Section 4.3, "Sampling parameters affect exploration"):

  • Temperature for PPO: $T = 0.7$ from SFT initialization; $T = 0.2$ from pretrained initialization. The paper explicitly notes that higher temperatures cause collapse with pretrained prompted models on math tasks, likely because the pretrained model has not been fine-tuned to produce coherent math solutions and high temperature sampling produces nonsensical outputs.
  • Temperature for EI: $T = 1.0$ during exploration to maximize solution diversity. The paper notes this avoids "too many degenerate solutions" — at $T = 1.0$, the model produces diverse but mostly coherent outputs, which is the sweet spot for discovery.
  • Best K of N (KoN) sampling: The paper sweeps $K \in \{1, 2, 4\}$ and $N \in \{1, 4, 8\}$. $K = 4, N = 4$ works best (Figure 8). Small $K/N$ ratios (e.g., $K = 1, N = 8$) discard most generated data, dramatically increasing sample complexity without improving performance. The paper concludes that "most configurations yielded decreased performance over $K = 1, N = 1$" — meaning selective filtering of high-reward samples is actually harmful, because it reduces the diversity of training data.

Model size and initialization effects (Section 4.3, "Model size and initialization affect exploration"):

  • Larger models (13B vs 7B) engage in more diverse exploration and benefit more from RL fine-tuning (Appendix B, Figures 11, 12).
  • Models with better generalization (pretrained with few-shot prompt vs. SFT) engage in more diverse exploration. SFT models have already been constrained to the SFT data distribution, limiting their ability to generate novel solution paths during RL exploration.
  • This creates a tension: SFT provides a better starting point (higher initial accuracy), but restricts exploration; pretrained initialization allows more diverse exploration but starts from a lower baseline. The paper shows that RL training largely closes the gap between pretrained and SFT initializations (from a >30% gap to <10% on GSM8K, Tables 1 and 2), primarily by improving the pretrained model's consistency rather than teaching new capabilities.

The SFT overfitting phenomenon (Section 4.3, "Model size and initialization affect exploration" and Table 4):

The paper provides a striking example: a model trained for 2 epochs of SFT (SFT 2) has maj@1 = 0.36 and pass@96 = 0.76. A model trained for 4 epochs (SFT 4) has higher maj@1 = 0.42 but lower pass@96 = 0.72. This is the maj@1 vs. pass@96 tradeoff identified by Cobbe et al. (2021). The paper diagnoses the cause: SFT 2 produces 3.7 unique correct solutions per question when sampled 96 times; SFT 4 produces only 2.9. The extra SFT training collapses solution diversity — the model memorizes specific solution patterns and loses the ability to generate alternatives.

RL fine-tuning breaks this tradeoff: PPO fine-tuning of SFT 2 improves maj@1 from 0.36 to 0.43 while maintaining its pass@96 diversity advantage. EI similarly improves maj@1 without the pass@96 collapse seen with continued SFT. The mechanism: online RL generates its own training data each iteration, which remains more diverse than the static SFT dataset because it's sampled from the model's current (exploratory) policy rather than from a fixed set of reference solutions.

RCRL label balance (Appendix A, Table 5):

The paper ablates the ratio of "[GOOD]" to "[BAD]" labels in RCRL training data, varying from all-positive (only "[GOOD]" labels, which reduces to standard EI on that data) to balanced (1:1 ratio). Best performance occurs with a high proportion of positive labels; balanced labels yield the worst performance. This is interpreted as evidence that the 7B model "doesn't effectively learn from the provided negative demonstrations" — the model cannot usefully incorporate information about what constitutes a "[BAD]" step, perhaps because the signal is too subtle for this model scale, or because the "[BAD]" label covers too diverse a set of failure modes for coherent pattern extraction.

Curriculum learning experiments (Appendix D, Figure 19):

The paper experiments with Prioritized Level Replay (PLR) and Backtracking — algorithms from the curriculum learning literature that aim to construct easier subproblems for the model to learn from before tackling harder ones. Backtracking initializes partial solutions from ground-truth intermediate steps (setting a "backtracking fraction" $\tau_0 = 0.9$, meaning the model starts from 90% of the way through a reference solution) and gradually increases the difficulty by removing more steps. Neither method exceeds default PPO performance (Figure 19). The paper hypothesizes this is because "limited exploration the model engages in from the start, due to both pretraining and supervised fine-tuning" — the model already has good solution priors, so curriculum strategies that assume the model needs to explore from scratch provide no benefit.

Data augmentation experiments (Appendix E, Figure 20, Table 6):

The paper attempts to generate synthetic training data via backtranslation: train a model to map answers to questions ($M_{A \to Q}$) and answers to alternative answers ($M_{A \to A}$), generate synthetic question-answer pairs, then filter by checking whether the student model can recover the intended answer. The filtering step produces a score for each synthetic pair (fraction of student rollouts that reach the correct answer). The paper finds that introducing any amount of synthetic data degrades performance (Table 6), even when filtered to moderate-difficulty pairs (scores near 0.5). Manual inspection reveals a high rate of false positives — questions where the student model consistently makes the same mistake as the generator model, resulting in high "recovery" scores for questions with incorrect intended answers. The paper provides a concrete example (Table E in the appendix) of a synthetic question whose intended answer is 100 but where the student computes 120 in 47% of rollouts — yet this pair would be scored high because 47% of student rollouts "recover" the (incorrect) intended answer. This failure mode arises because generator and verifier are trained on similar distributions and make correlated errors.

4. Key Insights and Innovations

Innovation 1: The Dominance of the Pretraining Prior — RL Algorithm Choice Matters Less Than the Exploration Budget

The paper's most conceptually distinctive contribution is not any single algorithm but rather the empirical demonstration that the choice of RL algorithm is a secondary concern when fine-tuning pretrained LLMs on reasoning tasks — what matters far more is simply generating diverse correct solutions and training on them. Prior to this work, the RLHF literature had established PPO as the default algorithm (Ouyang et al., 2022; Bai et al., 2022), carrying an implicit assumption that PPO's clipped surrogate objective, advantage estimation, and KL constraints were important ingredients for successful RL fine-tuning of LLMs. The reasoning-RL literature had scattered results using different algorithms (Expert Iteration in Yuan et al., 2023 and Zelikman et al., 2022; PPO in Luo et al., 2023) but no controlled comparison existed to test whether the algorithmic machinery actually mattered.

The paper overturns the default-PPO assumption through a systematic head-to-head comparison where all algorithms run on identical base models, identical tasks, and identical reward sources (Tables 1, 2, 3). Expert Iteration — which has no clipped objective, no advantage estimation, no value function, and no KL penalty — matches or exceeds PPO in nearly every configuration. The finding that EI achieves this with comparable sample complexity (Figures 5, 6) is particularly striking because it contradicts the standard RL narrative where PPO's on-policy updates are far more sample-efficient than behavior cloning approaches that discard suboptimal data. In classical RL training from scratch, PPO would dominate EI by orders of magnitude in sample efficiency. Here, EI needs at most ~10^6 samples to converge from a pretrained checkpoint — roughly the same order of magnitude as PPO's ~60,000, and when EI's per-prompt oversampling is reduced (K=4 instead of K=96), the sample complexities become identical.

What makes this a conceptual advance rather than merely a benchmarking result is the explanatory framework the paper provides: the pretraining prior is so strong that it constrains the policy to a narrow manifold of reasonable completions, within which simple rejection sampling (filter for correct, train on correct) is essentially optimal. The deterministic environment dynamics mean there is no need for PPO's stochasticity-handling machinery. The KL constraint that proves essential for PPO's stability (coefficient 0.05, Section 4.3) is a symptom of this — the model must be explicitly prevented from drifting off the pretraining manifold, whereas EI avoids drift by construction because it resets to the pretrained base model and trains only on correct solutions that were generated from that same manifold. This framing reframes the question from "which RL algorithm is best?" to "how can we generate the most diverse set of correct solutions for behavior cloning?" — a fundamentally different research direction.

The significance extends beyond reasoning tasks. The paper explicitly connects this finding to RLHF, noting that PPO's advantage over EI in prior RLHF studies (Gulcehre et al., 2023; Dubois et al., 2023) likely stems not from better optimization but from PPO's protective machinery against reward model over-optimization — the clipped objective and KL penalty prevent the policy from exploiting the learned reward model's blind spots. In reasoning tasks with ground-truth rewards, this protection is unnecessary because the reward signal is clean. This distinction — PPO as defense against reward hacking rather than as superior optimizer — provides a unified framework for understanding when simpler approaches suffice and when PPO's complexity is justified.

Innovation 2: The maj@1 vs. pass@96 Tradeoff as a Diagnostic for Overfitting — and RL as the Solution

The paper identifies and experimentally isolates a phenomenon that Cobbe et al. (2021) had observed but not explained: continued supervised fine-tuning on a static GSM8K dataset improves greedy decoding accuracy (maj@1) while simultaneously degrading the model's ability to generate correct solutions when sampled many times (pass@96). The paper's diagnostic contribution is to identify the mechanism as solution diversity collapse and to demonstrate that online RL algorithms break this tradeoff.

The evidence is crisp (Section 4.3, Table 4): a model trained for 2 SFT epochs achieves maj@1 = 0.36 and pass@96 = 0.76, producing on average 3.7 unique correct solutions per question when sampled 96 times. Continuing SFT for 2 more epochs (4 total) raises maj@1 to 0.42 but drops pass@96 to 0.72, with unique correct solutions falling to 2.9 per question. The model is getting better at producing the right answer on the first try, but worse at generating diverse correct reasoning paths — it is overfitting to the specific solution patterns in the static training set. This matters enormously for real-world deployment because test-time compute strategies (majority voting, verifier reranking) depend on generating multiple diverse candidate solutions. A model with higher maj@1 but collapsed pass@96 is actually less capable under a best-of-N evaluation budget.

The paper's innovation is not just documenting the tradeoff but showing that RL fine-tuning improves both metrics simultaneously. EI fine-tuning achieves both higher maj@1 (0.485 vs. 0.42 for SFT on 7B) and comparable or better pass@96 (Table 1). The mechanism: online RL generates its own training data each iteration by sampling from the current exploratory policy (at temperature T=1.0 for EI), which produces more diverse solutions than the static SFT dataset. The model is trained on a growing, self-generated corpus rather than a fixed set of reference solutions, preventing the diversity collapse. This is not merely an empirical improvement over SFT — it is a conceptual reframing of the fine-tuning objective: rather than training on human-provided demonstrations, train on the model's own successful explorations. The model bootstraps from its own diversity rather than collapsing toward a fixed target distribution.

The paper further shows that the pass@96 benefits of RL are not unbounded — the best RL-trained models achieve pass@96 scores within ~3% of what can be achieved with light SFT (2 epochs) given the same 96-sample budget (Section 4.1). This means RL primarily preserves diversity that SFT destroys rather than creating new diversity beyond what the pretrained model already possesses. This is a negative result with important implications: it means the ceiling on reasoning capability is largely set by pretraining, and RL fine-tuning can at best help the model realize its existing potential rather than transcend it.

Innovation 3: The Exploration Bottleneck — Models Don't Discover Novel Solutions During RL Fine-Tuning

The paper's most impactful negative finding is that LLMs do not engage in meaningful exploration beyond their pretraining neighborhood during RL fine-tuning — they primarily refine and make more reliable the solution strategies they already possess, rather than discovering genuinely new reasoning approaches. This is not presented as a limitation of the paper's specific implementation but as a fundamental bottleneck for the entire paradigm of RL-based LLM improvement.

The evidence is cumulative and multifaceted. First, the pass@96 scores across all algorithms and initializations cluster tightly (Tables 1, 2, 3) — with enough sampling budget, most models can solve roughly the same set of problems. A prompted pretrained 13B model with no SFT data achieves pass@96 = 0.83 on GSM8K, close to the SFT model's 0.84. This means the pretrained model already "knows" how to solve most of the problems it will ever solve; RL training makes it more reliable (higher maj@1) but does not meaningfully expand the set of solvable problems. Second, the sample complexity of all algorithms is tiny by RL standards — convergence in tens of thousands of rollouts rather than millions — suggesting the model is not searching a large behavior space but rather quickly settling into a local optimum near its initialization. Third, solution diversity metrics (Appendix B, Figures 11, 12) show that diversity increases early in training (as the model gains confidence to explore) but plateaus or declines in later iterations, consistent with the model settling into a limited repertoire of strategies. Fourth, larger models explore more diversely than smaller models (13B produces significantly more unique solutions than 7B, Figure 12), and models with better generalization explore more diversely than overfit models — both consistent with exploration being constrained by the pretraining prior rather than actively expanded by RL.

What makes this a conceptual contribution rather than mere observation is the explanatory framework it enables for reconciling conflicting prior results. Huang et al. (2022) found LLMs can self-improve; other work found they cannot self-correct reasoning. The paper's framework resolves this: self-improvement works when it amplifies existing capability (making the model more consistent at solutions it already knows), but fails when it requires discovering fundamentally new reasoning patterns. The paper explicitly connects this to classical RL: in environments like Atari or Go, agents trained from scratch must explore vast state spaces to discover reward, and sophisticated exploration algorithms are essential. For pretrained LLMs, the state space is effectively pre-pruned — the model assigns near-zero probability to the vast majority of token sequences, so exploration is restricted to a tiny manifold of "reasonable" completions. Within this manifold, simple algorithms suffice; beyond it, no current algorithm helps.

This finding redirects the research agenda: rather than developing more sophisticated RL algorithms for LLM fine-tuning (which the paper shows provide diminishing returns), effort should be directed toward (1) improving the pretraining prior so the model's initial solution manifold covers more hard problems, (2) developing prompting or search strategies (Tree of Thoughts, evolutionary algorithms) that can guide the model to solution regions it wouldn't reach through standard autoregressive sampling, or (3) designing exploration mechanisms that explicitly push the model beyond its pretraining distribution — for example, by adding novelty bonuses or using separate exploration policies. The paper positions this as the central open problem for the field, not a solved issue.

Innovation 4: Reward Signal Fidelity Trumps Reward Density — Why Dense and Learned Rewards Don't Help

The paper provides a systematic empirical refutation of the intuitively appealing hypothesis that providing more frequent, information-rich reward signals should improve RL training for LLM reasoning. The finding is stark: sparse ground-truth reward (+1 for correct final answer, 0 otherwise) is sufficient, and augmenting it with ORM-based, dense, or heuristic rewards provides no benefit to final converged performance (Section 4.1, Figures 3, 4, 14-17).

This is conceptually significant because it inverts the standard RL intuition. In classical RL, sparse rewards are a major challenge — agents struggle to learn without intermediate feedback because the credit assignment problem becomes exponentially harder with horizon length. The standard solution is reward shaping: provide dense rewards that guide the agent toward the sparse objective. The paper shows that for pretrained LLMs, this intuition breaks. The model already knows how to produce reasonable solution structures; it just needs to know which complete solutions are correct. Providing intermediate rewards — whether heuristic (matching reference solution steps) or learned (ORM predictions at each step) — either provides no benefit (ORM dense) or actively harms performance (heuristic dense) by encouraging the model to copy reference solutions rather than developing its own reasoning patterns.

The ORM reward case is particularly instructive. The ORM is a separately trained classifier that predicts solution correctness probability from intermediate prefixes. Using its predictions as a dense reward during PPO or EI training provides a continuous, step-level signal. The paper finds this "improves sample efficiency but not performance" (Section 4.1) — models converge faster but reach the same final accuracy as sparse ground-truth reward. In some configurations, ORM-guided training slightly underperforms sparse reward (Figures 14, 15). The interpretation is that the ORM introduces noise (false positives and false negatives in its predictions) that offsets the benefit of a continuous signal. The ground-truth binary reward is perfectly accurate; the ORM is approximately correct. At the model scales studied (7B-13B), the approximation error dominates any benefit from reward density.

This finding has practical implications for RLHF and other LLM fine-tuning settings. It suggests that effort invested in training better reward models for use as training signals should be redirected toward (1) using reward models only for evaluation-time reranking (where their noise can be averaged over many samples) rather than as training rewards, (2) improving the ground-truth signal by collecting more high-quality demonstrations (which benefits all algorithms through better behavior cloning data), or (3) developing methods that are robust to reward noise, which PPO's conservative update already partially provides through its clipped objective and KL constraint. The paper's framing — that reward fidelity dominates reward density for pretrained models — provides a principled basis for these design decisions rather than leaving them as empirical tuning choices.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses two math word problem benchmarks: GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021). GSM8K consists of approximately 7,500 training and 1,319 test questions; SVAMP contains 1,000 test questions (Section 4). The exact training split sizes are not specified. Both benchmarks require multi-step arithmetic reasoning in natural language with a final numeric answer. GSM8K additionally provides reference solutions with explicit intermediate steps, which the paper uses for dense heuristic reward experiments and SFT training.

  • Base model(s). All experiments use instruction-tuned Llama-2 models at two scales: 7B and 13B parameters (Section 4). The paper states it uses "instruction-tuned Llama-2 7B and Llama-2 13B models" — these are the instruction-tuned variants of Touvron et al. (2023), not the base pretrained models, though the experiments do fine-tune from both pretrained and SFT checkpoints depending on the initialization condition. The models are chosen to be representative of current open-source LLM capabilities at modest scale, allowing controlled algorithmic comparison without the computational demands of 70B+ models.

  • Metrics. The paper evaluates four metrics, each computed on the test set (Section 4):

    • maj@1: Greedy decoding accuracy — a single deterministic sample per question (temperature 0). Measures the model's default, compute-minimal performance.
    • maj@96: Majority vote accuracy — sample K = 96 solutions per question (temperature 0.7) and take the most common final answer. Measures ensemble performance without a learned verifier.
    • rerank@96: ORM-reranked accuracy — sample K = 96 solutions per question, score each with the trained Outcome-Based Reward Model, and select the answer with the highest ORM score. Measures the usefulness of a learned verifier.
    • pass@96: Oracle best-of-96 accuracy — sample K = 96 solutions per question and return 1 if any of them matches the ground-truth answer. Measures the model's fundamental capability ceiling under a 96-sample budget — the fraction of problems for which the model can produce at least one correct solution.

    The combination of maj@1 and pass@96 is particularly diagnostic: if maj@1 increases while pass@96 decreases, the model is overfitting to greedy decoding at the expense of solution diversity; if both increase, the model is genuinely improving.

  • Baselines. The paper compares against several baselines:

    • SFT (supervised fine-tuning): The Llama-2 model fine-tuned for 4 epochs on the GSM8K training set with a global batch size of 128, initial learning rate 2e-5 decayed to 2e-7 with cosine warmup (Section 4.1). This is the pre-RL starting point in the SFT initialization experiments.
    • SFT 2: A lighter SFT baseline trained for only 2 epochs (Section 4.3, Table 4), used to study the maj@1 vs. pass@96 tradeoff.
    • Prompted pretrained: The base Llama-2 model with a 2-shot prompt drawn from the GSM8K validation set, used in the no-SFT initialization experiments (Section 4.2).
    • EI (Expert Iteration) baseline: The initial round of EI (EI round 0) uses the SFT or prompted pretrained model as the exploration policy, so these also serve as baselines for subsequent EI rounds.
    • External reference baselines (bottom of Table 1): The paper lists results from Llemma (Azerbayev et al., 2023), GPT-3 (Cobbe et al., 2021), and GPT-4 (OpenAI, 2023) for context, though these are not directly comparable due to different model families, scales, and training procedures.

  • Generation budget / compute accounting. The primary unit of compute tracked throughout is model rollouts — the total number of complete solutions generated during training (Section 4, Figures 1, 5, 6 and Appendix C). For PPO, this is the number of sampled solutions across all training iterations (approximately 1,024 rollouts per update × number of updates, with best performance at ~60,000 rollouts from SFT initialization). For EI, this is K samples per question × number of train questions × number of iterations (with K = 96 per question by default, reaching at most ~10^6 rollouts for 5 iterations × ~7,000 train questions). The paper explicitly compares algorithms at equivalent rollout budgets in sample complexity plots (Figures 1, 5, 6). Wall-clock time is noted but not used as the primary axis — the paper reports that PPO and EI both take "about a day" to train despite different rollout counts, due to PPO's memory constraints reducing throughput (Section 4.1).

  • Cross-validation / statistical protocol. The paper does not report formal cross-validation. The best checkpoint for each algorithm is selected "via performance on a validation set" (Section 3), with the specific validation split unspecified. EI uses a held-out validation set to determine when to stop iterating. For the SFT overfitting analysis (Table 4), the paper compares SFT 2 and SFT 4 directly on the test set, which is standard for benchmark evaluation but provides no confidence intervals or significance tests. The sample complexity curves (Figures 1, 3-6, 14-17) plot aggregate test metrics against cumulative rollouts, with each point representing a single training run at that checkpoint — no error bars or multiple seeds are reported.

Main Quantitative Results

The paper's results are organized around two initialization regimes (with and without SFT data) and three algorithmic families. I'll structure this by the paper's own logical groupings.

Results with SFT Initialization (Section 4.1, Table 1, Figures 1-4)

Headline finding: Expert Iteration achieves the best performance across all metrics for both model sizes, while requiring comparable or modestly higher sample complexity than PPO.

Table 1 reports all metrics for Llama-2 7B and 13B models initialized from SFT checkpoints and evaluated on GSM8K:

For the 7B model:

  • SFT baseline: maj@1 = 0.42, maj@96 = 0.58, rerank@96 = 0.64, pass@96 = 0.72
  • EI (best, after n=2 iterations): maj@1 = 0.485, maj@96 = 0.66, rerank@96 = 0.73, pass@96 = 0.79
  • PPO (sparse, best variant): maj@1 = 0.46, maj@96 = 0.64, rerank@96 = 0.69, pass@96 = 0.75
  • PPO (ORM-guided): maj@1 = 0.47, maj@96 = 0.64, rerank@96 = 0.68, pass@96 = 0.76
  • RCRL: maj@1 = 0.44, maj@96 = 0.62, rerank@96 = 0.67, pass@96 = 0.74

EI gives a ~6.5 percentage point improvement in maj@1 over the SFT baseline (0.485 vs. 0.42). PPO gives a ~3.5–5 point improvement depending on reward type. The gap between EI and PPO is largest in rerank@96 (0.73 vs. 0.69), though the paper notes this may be inflated by ORM overfit to EI-generated training data (Section 4.1).

For the 13B model:

  • SFT baseline: maj@1 = 0.47, maj@96 = 0.63, rerank@96 = 0.72, pass@96 = 0.84
  • EI (best, after n=2 iterations): maj@1 = 0.53, maj@96 = 0.72, rerank@96 = 0.77, pass@96 = 0.83
  • PPO (sparse): maj@1 = 0.49, rerank@96 = 0.72, pass@96 = 0.82 (maj@96 not reported)
  • PPO (ORM-guided): maj@1 = 0.52, rerank@96 = 0.72, pass@96 = 0.83
  • RCRL: maj@1 = 0.51, rerank@96 = 0.73, pass@96 = 0.81

The pattern holds at 13B: EI leads in maj@1 (0.53), with ORM-guided PPO close behind (0.52). The SFT 13B baseline starts higher (maj@1 = 0.47 vs. 0.42 for 7B), and the absolute gains from RL are smaller (~6 points for EI, ~5 points for PPO), consistent with the paper's observation that larger models benefit less from RL fine-tuning relative to their SFT starting points.

Convergence behavior of EI (Figure 2): For SFT-initialized models, performance plateaus after n = 2 EI iterations on both 7B and 13B. The first iteration provides the largest gain; the second adds marginal improvement. Pretrained-initialized models require n = 4 iterations to converge, with monotonic improvement at each round (Figure 2 shows GSM8K test accuracy against iteration count; SVAMP convergence is shown in Figure 10, Appendix B).

Sample complexity comparison (Figure 1): Figure 1 plots maj@1 test accuracy against the number of model rollouts (log scale) for SFT-initialized 7B models. PPO reaches its best maj@1 (~0.46–0.47) after approximately 60,000 rollouts. EI requires approximately 10× more rollouts to reach its best performance (~0.485) — on the order of 600,000 — because each iteration samples K = 96 solutions per question across ~7,000 training questions (96 × 7,000 = 672,000 rollouts per iteration). However, the paper notes that reducing K from 96 to 4 dramatically lowers EI's sample complexity to "the same... as PPO on GSM8K" (Section 4.2), though this experiment is reported in the no-SFT setting, not SFT-initialized.

ORM guidance provides limited benefit for both algorithms (Figures 3, 4): Figure 3 plots maj@1 for EI and ORM-aided EI over the course of training; Figure 4 does the same for PPO. In both cases, the ORM improves sample efficiency — models converge faster in terms of rollouts — but does not improve final converged performance. For EI, the ORM-guided variant initially rises faster but plateaus at roughly the same accuracy as the sparse variant. For PPO, the ORM-guided variant shows a similar pattern. The paper summarizes: "the ORM slightly improves PPO performance and negligibly impacts EI performance" in final accuracy, while providing "an improvement in terms of sample complexity" (Section 4.1).

Dense rewards provide no benefit (or harm): The paper states that "Giving dense rewards at best provides no extra benefit to performance when given either heuristically or via the ORM. Giving a heuristic dense reward even slightly harms model performance relative to the sparse setting" (Section 4.1). Specific numbers for this claim are not reported in the main text tables or figures; the paper references the observation that heuristic dense rewards likely "encourage more overfit to exact solutions in the train set, limiting solution diversity," without quantifying the performance gap.

Pass@96 stability and the SFT tradeoff (Table 4): The paper highlights that RL fine-tuning preserves pass@96 while improving maj@1, unlike continued SFT:

  • SFT 2 (2 epochs): maj@1 = 0.36, pass@96 = 0.76
  • SFT 4 (4 epochs): maj@1 = 0.42, pass@96 = 0.72
  • SFT 2 + PPO fine-tune: maj@1 = 0.43, pass@96 = not explicitly reported but stated to maintain SFT 2's diversity advantage

The paper also notes that most models in Table 1 have pass@96 scores within ~3% of each other (0.72–0.79 for 7B, 0.81–0.84 for 13B), supporting the claim that RL training "mostly impacts maj@1 accuracy without significantly improving on a pass@n accuracy which can be achieved with a light amount of supervised fine-tuning" (Section 4.1).

Rerank@96 and ORM overfit: The EI models show a ~5 percentage point lead over PPO in rerank@96 (0.73 vs. 0.69 for 7B). The paper explicitly acknowledges that "a non-trivial percentage of this gap is due to overfit of the ORM to the EI model which was used to generate its training data" (Section 4.1). This means the EI rerank@96 advantage is partly an artifact of ORM training bias, not purely a reflection of EI solution quality.

Results without SFT Initialization (Section 4.2, Tables 2, 3, Figures 5, 6)

Headline finding: Without SFT data, EI and PPO achieve comparable performance, with EI achieving the best results on SVAMP while PPO slightly outperforms on GSM8K, and both algorithms converge with remarkably low sample complexity (tens of thousands of rollouts).

GSM8K results (Table 2): Starting from a prompted pretrained 7B Llama-2 model with maj@1 approximately 0.05:

  • EI (n=5 iterations): maj@1 = 0.23, maj@96 = 0.57, rerank@96 = 0.56, pass@96 = 0.73
  • PPO (sparse): maj@1 = 0.31, maj@96 = 0.58, rerank@96 = 0.59, pass@96 = 0.73

For the 13B model starting from maj@1 approximately 0.03:

  • EI (n=4 iterations): maj@1 = 0.38, maj@96 = 0.67, rerank@96 = 0.69, pass@96 = 0.82
  • PPO (sparse): maj@1 = 0.40, maj@96 = 0.66, rerank@96 = 0.64, pass@96 = 0.81

PPO achieves slightly higher maj@1 on GSM8K (0.31 vs. 0.23 for 7B; 0.40 vs. 0.38 for 13B), but the two algorithms are within a few percentage points on all metrics. The pass@96 scores are nearly identical (7B: 0.73 vs. 0.73; 13B: 0.82 vs. 0.81). Notably, the 13B prompted model achieves pass@96 = 0.83 (from the SFT baseline in Table 1, cited in Section 4.2) vs. 0.81–0.82 after RL, suggesting RL does not meaningfully expand the set of solvable problems beyond what the prompted model can already solve with 96 samples.

SVAMP results (Table 3): Starting from prompted pretrained models with maj@1 of 0.06 (7B) and 0.05 (13B):

  • 7B EI (n=5 iterations): maj@1 = 0.58
  • 7B PPO (sparse): maj@1 = 0.48
  • 13B EI (n=4 iterations): maj@1 = 0.69
  • 13B PPO (sparse): maj@1 = 0.59

On SVAMP, EI substantially outperforms PPO — a ~10 percentage point gap for both model sizes. The paper's SVAMP experiments report only maj@1, not the full suite of metrics used for GSM8K.

Sample complexity analysis (Figures 5, 6): Figure 5 plots sample complexity on GSM8K from pretrained initialization. PPO converges after approximately 30,000 rollouts on GSM8K, reaching its best maj@1. EI appears to have higher sample complexity with the default K = 96 samples per question, requiring more total rollouts to converge but also converging to comparable accuracy. The paper notes: "We test this, we reduce the number of samples per prompt each round of EI from K=96 to K=4. The resulting EI models require more iterations to converge but require far less total samples, also converging in accuracy only a few percentage points lower than K=96 samples per prompt. With K=4 rollouts per prompt EI has the same sample complexity as PPO on GSM8K" (Section 4.2).

Figure 6 shows the same comparison on SVAMP, where "EI appears nearly as sample efficient as PPO" — the two curves are close together in terms of rollout count to reach given accuracy levels, despite EI ultimately converging to a substantially higher accuracy than PPO.

This is framed as a surprising result: "This is a particularly surprising finding when compared to the performance of EI and PPO on more classical RL problems training a neural network from scratch. Often PPO enjoys far better sample complexity in these settings" (Section 4.2). The paper attributes this to the pretraining prior constraining exploration.

Pretrained vs. SFT Initialization Gap After RL

A cross-cutting result that the paper emphasizes (Section 1, "we also observe the gap between pretrained model performance and SFT model performance significantly shrinks... after RL fine-tuning"):

  • GSM8K 7B: SFT-initialized EI achieves maj@1 = 0.485 (Table 1); pretrained-initialized EI achieves 0.23 (Table 2). The gap is still substantial (~25 points), contradicting a literal reading of "shrinks to < 10%." The paper specifies this claim for 13B models: "with larger models having a smaller gap" (Section 1). For 13B, SFT-initialized EI reaches 0.53, pretrained-initialized EI reaches 0.38 — a 15-point gap, still above 10%.
  • GSM8K PPO 13B: SFT-initialized reaches ~0.49–0.52, pretrained-initialized reaches 0.40 — a ~10-point gap.
  • SVAMP: The gap closes more dramatically. No SFT 13B baseline is 0.05; EI reaches 0.69. SFT initialization results are not reported for SVAMP, but the paper presents the 0.69 as comparable to strong fine-tuned performance.

The "< 10% gap" claim appears to reference the PPO 13B case specifically, where SFT-initialized ~0.50 and pretrained-initialized ~0.40 differ by about 10 points. The paper's abstract and introduction elide the model size and algorithm specificity of this finding.

Ablation Studies and Robustness Checks

  • PPO value branch architecture (Figure 7): Using a large value branch (L = 4 transformer layers) and detaching value gradients from the shared trunk is critical — the default configuration (small value head, no gradient detachment) shows substantially lower and more unstable maj@1 scores over training. The exact performance numbers are visible in Figure 7 but not tabulated.

  • Best K of N sampling for PPO (Figure 8): Sweeping K ∈ {1, 2, 4} and N ∈ {1, 4, 8} reveals that K = 4, N = 4 achieves the best performance and fastest runtime. Small K/N ratios (e.g., K = 1, N = 8) yield decreased performance compared to the default K = 1, N = 1, because discarding most generated rollouts reduces training data diversity more than selective filtering helps. Figure 8 shows that K = 4, N = 4 and K = 1, N = 1 perform comparably; all other configurations underperform.

  • Partial fine-tuning for PPO (Section 4.3): Fine-tuning only the top M transformer layers (rather than all layers with LoRA) saves memory but costs "a few percentage points of performance." The paper does not tabulate exact numbers or specify M.

  • KL penalty coefficient for PPO (Section 4.3): A KL penalty of 0.05 is described as "critical for preventing model collapse after more than a hundred gradient updates." The paper states that without this constraint, training becomes unstable, but no ablation figure with varying KL coefficients is provided — only the contrast with Bai et al. (2022), who did not need KL constraints for dialogue RLHF.

  • PPO sampling temperature by initialization (Section 4.3): T = 0.7 works well from SFT initialization; T = 0.2 is necessary from pretrained initialization because higher temperatures cause collapse with prompted pretrained models on math tasks. The paper notes: "Potentially better results for PPO could likely be achieved by annealing the exploration temperature over the course of training" but does not test this.

  • EI number of iterations until convergence (Figures 2, 9, 10): From SFT initialization, performance plateaus after n = 2 iterations for both 7B and 13B on GSM8K (Figure 2). From pretrained initialization, convergence takes n = 4 iterations on GSM8K (Figure 9) and n = 4–5 iterations on SVAMP (Figure 10). Additional iterations beyond convergence do not improve performance and may slightly degrade it (Appendix B notes solution diversity decreases in later iterations even as maj@1 plateaus).

  • EI samples per question (K) reduction (Section 4.2): Reducing K from 96 to 4 samples per question during each EI round increases the number of iterations needed to converge but dramatically reduces total sample complexity, bringing EI's sample complexity in line with PPO's on GSM8K. The final accuracy with K = 4 is "only a few percentage points lower" than with K = 96. Exact numbers are not tabulated; this is described as a qualitative finding in the main text.

  • RCRL label balance (Table 5, Appendix A): Varying the proportion of [GOOD] to [BAD] labels in RCRL training data reveals that a balanced ratio (1:1) yields the worst performance. Increasing the proportion of positive data improves results, with heavily positively-skewed datasets achieving the best performance. The paper interprets this as evidence that the 7B model does not effectively learn from negative demonstrations: "At best, we only see very marginal gains using RCRL" over the data-generating EI-minimal baseline.

  • Solution diversity over EI iterations (Figures 11, 12, Appendix B): Both exact diversity and trace diversity increase during the first two EI iterations, matching when the largest maj@1 gains occur, then plateau or decrease in later iterations. The 13B model produces significantly more diverse solutions than 7B at every iteration. Figure 11 (GSM8K, no SFT) and Figure 12 (SVAMP) show that trace diversity (computational structure uniqueness) peaks after ~2 rounds while exact diversity (surface form uniqueness) continues increasing for 13B, suggesting the model learns to express similar reasoning in different surface forms rather than discovering new reasoning strategies.

  • Curriculum learning (PLR and Backtracking, Figure 19, Appendix D): Neither Prioritized Level Replay nor Backtracking exceeds default PPO performance when fine-tuning from an SFT-initialized Llama-2 7B on GSM8K. The paper hypothesizes this is because "limited exploration the model engages in from the start" — the curriculum strategies assume the model needs guidance through a difficulty hierarchy, but the pretrained model already has strong solution priors that make such guidance unnecessary or even counterproductive.

  • Synthetic data augmentation via backtranslation (Table 6, Figure 20, Appendix E): Any amount of synthetically generated (Q, A) pairs — filtered to moderate difficulty based on student recovery scores — degrades performance compared to training on ground-truth SFT data alone. The paper provides a concrete failure example (Table E in appendix) where a synthetic question with intended answer 100 is "solved" by the student model 47% of the time by consistently making the same mistake (computing 120), creating a high recovery score for a pair with an incorrect intended answer. This reveals correlated errors between generator and verifier as the core failure mode.

  • 13B vs. 7B exploration diversity (Appendix B, Figures 11, 12): The 13B model consistently produces more diverse solutions than the 7B model across all EI iterations, with the gap widening as training progresses. The 13B model achieves approximately "an exactly unique solution with every sampling after 4 rounds of expert iteration" by exact string match, but its trace diversity peaks after two rounds, indicating that surface form diversity outpaces genuine computational diversity.

Critical Assessment

Do the experiments support the central claim that Expert Iteration performs best in most cases?

The evidence is stronger for some configurations than others. For SFT-initialized models on GSM8K (Table 1), EI achieves the highest scores across all four metrics for both 7B (maj@1 = 0.485 vs. PPO's 0.47) and 13B (maj@1 = 0.53 vs. PPO's 0.52). However, the margin is modest — a 1.5 percentage point advantage over PPO on 7B maj@1 and a 1 point advantage on 13B — and no confidence intervals are reported, making statistical significance unclear at the ~100-question GSM8K test set size. The rerank@96 advantage for EI (0.73 vs. 0.69 on 7B) is partly attributed to ORM overfit, weakening this as evidence of EI's superiority.

For pretrained-initialized models, the "best in most cases" claim requires qualification: on GSM8K (Table 2), PPO achieves higher maj@1 for both 7B (0.31 vs. 0.23) and 13B (0.40 vs. 0.38), while EI leads on some secondary metrics (13B rerank@96: 0.69 vs. 0.64; 7B maj@96: 0.57 vs. 0.58 — essentially tied). On SVAMP (Table 3), EI wins decisively (7B: 0.58 vs. 0.48; 13B: 0.69 vs. 0.59), but only maj@1 is reported for SVAMP. The "most cases" conclusion depends on which metrics and tasks one weights most heavily — on GSM8K maj@1 from pretrained initialization, which is arguably the hardest and most practically relevant setting, PPO wins for both model sizes.

The RCRL comparison is clearer: RCRL underperforms both EI and PPO in essentially all configurations (Table 1), with the possible exception of 13B GSM8K where RCRL's maj@1 = 0.51 is between PPO sparse (0.49) and PPO ORM (0.52). The "all algorithms perform comparably" claim in the findings (Section 5) is more accurate than "EI performs best" — the takeaway should be that the differences between EI and PPO are small and not uniformly in EI's favor.

Does the evidence support the claim that EI has sample complexity comparable to PPO?

The paper provides evidence for this claim but with important caveats. Figure 5 (GSM8K, no SFT) shows PPO reaching its best accuracy around 30,000 rollouts while EI with K=96 requires substantially more — the "same sample complexity" claim requires reducing EI to K=4, which the paper reports in text but does not plot. The statement "EI has the same sample complexity as PPO" (Section 4.2) conflates the K=96 results shown in the figures with the K=4 results described in text. For the SFT-initialized case (Figure 1), EI with K=96 clearly requires ~10× more rollouts than PPO to converge (hundreds of thousands vs. ~60,000), and the paper acknowledges EI trains "with an order of magnitude more" rollouts while noting wall-clock time is similar due to PPO's memory constraints. The sample complexity claim therefore holds only when EI's per-prompt oversampling is reduced to match PPO's — a configuration that was tested but whose results are not fully tabulated or plotted. The paper would be strengthened by a side-by-side figure showing EI with K=4, K=16, and K=96 against PPO on the same axes.

Does the evidence support the claim that RL improves both maj@1 and pass@96 simultaneously?

The evidence for this claim is mixed and incomplete. Table 4 provides the clearest support: SFT 2 has maj@1 = 0.36 and pass@96 = 0.76; SFT 4 has higher maj@1 (0.42) but lower pass@96 (0.72); SFT 2 + PPO achieves maj@1 = 0.43 while "maintaining" SFT 2's pass@96 advantage. This directly demonstrates the claimed effect. However, the pass@96 for SFT 2 + PPO is not explicitly reported in Table 4 — the paper states it maintains the diversity, but the exact pass@96 number is absent. For the full EI and PPO results in Table 1, the pass@96 scores cluster tightly (0.72–0.79 for 7B), and the paper itself notes the "proximity of pass@96 accuracies among most models" (Section 4.1). The best EI model's pass@96 (0.79) is only modestly above the SFT baseline (0.72), and the paper attributes part of this to the SFT baseline being overtrained: "the SFT checkpoint has undergone much more training on a less diverse dataset. Simply supervised fine-tuning for half as many steps results in a checkpoint with... pass@96 = 0.76" (Section 4.1). This means the apparent pass@96 improvement from RL (0.72 to 0.79) is partly an artifact of comparing against an overtrained SFT baseline — comparing against SFT 2 (pass@96 = 0.76) shrinks the gain to 3 percentage points.

Does the evidence support the claim that models fail to explore significantly beyond SFT solutions?

This is the paper's central negative finding, and the evidence is strong though circumstantial. The key pieces:

  1. Pass@96 scores are similar across all models, including prompted pretrained models with no SFT data (prompted 13B achieves pass@96 = 0.83 vs. SFT 13B's 0.84, Section 4.2). This directly shows RL is not expanding the set of solvable problems.
  2. Sample complexity is extremely low (tens of thousands of rollouts), inconsistent with genuine exploration of a large behavior space.
  3. Solution diversity plateaus or decreases after initial EI iterations (Figures 11, 12).
  4. The SFT + RL pass@96 gains are mostly explained by avoiding SFT overfit, not by discovering novel solutions.

However, the paper does not present a direct exploration metric — something like "what fraction of correct RL-discovered solutions are exact matches to solutions the SFT model could already generate?" or "what is the novelty rate of solutions under RL vs. SFT?" These would provide direct rather than circumstantial evidence. The claim is well-supported by the evidence presented, but the evidence remains indirect.

Genuine weaknesses in the experimental design:

Single model family, two benchmarks: All experiments use Llama-2 7B and 13B on GSM8K and SVAMP. The paper's conclusions about exploration limits, PPO vs. EI performance, and sample complexity may not transfer to other model families (e.g., Mistral, Gemma, CodeLlama), other scales (70B+), or other reasoning modalities (code generation, logical deduction, scientific QA). The paper explicitly acknowledges the model scale limitation in its exploration discussion (Section 4.3, "larger models engaged in more diverse exploration"), but does not test beyond 13B.

Missing error bars and multiple seeds: All results are presented as point estimates from single training runs. RL training is notoriously stochastic — different random seeds can produce substantially different outcomes, especially with small validation sets and early stopping. The absence of confidence intervals or multi-seed results means we cannot assess whether the reported differences (e.g., EI maj@1 = 0.485 vs. PPO = 0.47) are statistically meaningful or within run-to-run variance.

The ORM overfit confound for rerank@96: The paper's admission that the ORM's training data comes from the EI model creates a circular evaluation: EI models are evaluated by a verifier trained on EI-generated data, while PPO models are evaluated by the same (EI-trained) verifier. The resulting ~5% rerank@96 advantage for EI over PPO is acknowledged as partly an ORM overfit artifact, but this means the rerank@96 metric — which is supposed to measure solution quality under a verifier — is biased in an unknown proportion. The paper does not attempt to quantify this bias, e.g., by training separate ORMs on EI and PPO outputs and cross-evaluating.

Missing ablation of key EI hyperparameter: the reset to pretrained model each iteration. The paper states that resetting to the pretrained base model at each EI iteration (rather than continuing fine-tuning from the previous iteration's model) is "crucial for achieving best performance," citing Zelikman et al. (2022). This is a non-obvious design choice with important implications — it means EI is effectively doing iterative data expansion with a fixed model initialization rather than sequential policy improvement. However, no ablation comparing "reset to pretrained" vs. "continue from previous iteration" is presented. The paper would be strengthened by showing this ablation, since it is central to understanding why EI works.

The K=4 EI sample complexity result is underreported. The paper makes the important claim that EI with K=4 achieves comparable sample complexity to PPO, but this result is described in prose (Section 4.2) without a supporting figure or table. Exact final accuracies, convergence iteration counts, and total sample counts for K=4 are not provided. Given that this is one of the paper's headline findings — that the simplest algorithm is as sample-efficient as the most complex — the lack of systematic documentation is a significant gap.

SVAMP evaluation is incomplete. The SVAMP experiments (Table 3, Figure 6) report only maj@1, without maj@96, rerank@96, or pass@96. This prevents the paper from making its key distinction between "reliability improvement" and "capability expansion" on SVAMP, weakening the generality of that distinction's empirical support.

The "dense rewards provide no benefit" claim lacks quantified support. While the paper states that heuristic dense rewards harm performance and ORM dense rewards don't help converged performance, this claim is not accompanied by a table or figure in the main text showing the actual performance difference. Figures 14–17 in Appendix C show ORM-guided vs. sparse sample complexity curves, where ORM-guided sometimes slightly underperforms at convergence, but the heuristic dense reward experiments are not plotted. The reader must take this claim on faith without seeing the data.

What experiments would have strengthened the paper:

  1. A multi-seed study (3+ seeds per algorithm per initialization) with error bars, to establish whether the small performance differences between EI and PPO are reliable.
  2. Direct exploration analysis: for a sample of questions, compare SFT model solutions, EI-discovered solutions, and PPO-discovered solutions, measuring overlap and novelty rates.
  3. ORM cross-training: train ORMs on data from each algorithm separately, and cross-evaluate (EI model reranked by PPO-trained ORM, and vice versa) to quantify and correct for ORM overfit.
  4. Systematic sweep of EI samples-per-prompt K on both SFT and pretrained initializations, with final accuracies and sample complexities tabulated.
  5. Larger model scales (at minimum 34B or 70B Llama-2) to test the paper's own hypothesis that larger models engage in more diverse exploration, which might qualitatively change the EI vs. PPO comparison.
  6. Extension to a non-math reasoning benchmark (e.g., strategy QA, code generation) to test whether the exploration bottleneck is specific to math reasoning or general.

Conditions on the paper's claims:

  • "EI performs best in most cases" holds for SFT-initialized GSM8K and SVAMP across all reported metrics (Tables 1, 3), but not for pretrained-initialized GSM8K maj@1 where PPO leads (Table 2). The claim should be conditioned on initialization regime and metric.
  • "RL improves both maj@1 and pass@96 simultaneously" holds relative to continued SFT training (Table 4), but not necessarily relative to light SFT (SFT 2, pass@96 = 0.76 vs. EI pass@96 = 0.79 — a small gain). The claim is more precisely stated as "RL avoids the pass@96 degradation that continued SFT causes."
  • "Models fail to explore significantly" holds for the 7B and 13B Llama-2 models tested, on GSM8K and SVAMP, with the paper's exploration hyperparameters. The paper itself suggests this may change with larger models (Section 4.3, Appendix B) or different exploration temperatures. The claim should not be interpreted as a universal law of LLM fine-tuning.
  • "Sparse ground-truth reward is sufficient; dense and ORM rewards provide no benefit" holds for final converged performance at the tested model scales. ORM rewards do improve sample efficiency (Figures 3, 4, 14–17), so they have practical value for reducing training time at the cost of maintaining a separate ORM. The claim about "no benefit" should be scoped to final accuracy, not to training dynamics.

6. Limitations and Trade-offs

6.1 The Exploration Bottleneck: RL Fine-Tuning Does Not Expand the Set of Solvable Problems

The assumption or constraint. The paper's central negative finding is that LLMs do not engage in meaningful exploration beyond their pretraining neighborhood during RL fine-tuning. The paper states this explicitly in Section 4.1:

"This further suggests RL training mostly impacts maj@1 accuracy without significantly improving on a pass@n accuracy which can be achieved with a light amount of supervised fine-tuning."

And in the Discussion (Section 5):

"This observation, taken together with the fast convergence of both online algorithms and the low-impact of ORM guidance and dense rewards, suggests models are not engaging in a significant amount of exploration beyond pretraining/SFT data."

The paper frames this not as an implementation artifact but as a fundamental limitation: the pretrained model's prior over token sequences is so strong that it constrains exploration to a narrow manifold of "reasonable completions," and no current RL algorithm pushes the model meaningfully beyond this manifold.

The consequence. This limitation imposes a hard ceiling on what RL fine-tuning can achieve. If the pretrained model cannot generate a correct solution to a problem within a 96-sample budget, RL fine-tuning will not enable it to do so — it can only make the model more reliable at producing correct solutions it already "knows" how to generate, not teach it genuinely new reasoning patterns. The paper's evidence for this is the near-identical pass@96 scores across all models and initializations: a prompted pretrained 13B model with zero SFT data achieves pass@96 = 0.83 on GSM8K, close to the SFT model's 0.84 (Section 4.2). This means the boundary between solvable and unsolvable problems is largely set during pretraining, and RL fine-tuning cannot cross it. For practitioners, this translates to a sobering reality: if your base model cannot occasionally stumble onto the correct answer for a problem class through random sampling, no amount of PPO or Expert Iteration will help. The investment in RL infrastructure may be wasted on problems that are genuinely outside the model's capability frontier.

What evidence exists in the paper. The evidence is multi-pronged and consistent across the paper:

  • Pass@96 scores across all algorithms and initializations cluster within ~3% of each other (Table 1: 7B pass@96 ranges from 0.72 to 0.79; 13B from 0.81 to 0.84). The paper notes that "the proximity of pass@96 accuracies among most models is in sharp contrast to the rerank@96 performance" (Section 4.1).
  • A prompted pretrained 13B model achieves pass@96 = 0.83, nearly matching the SFT-initialized EI model's pass@96 = 0.83 (Table 1, Section 4.2). The prompted model's maj@1 is only ~0.03, meaning it has the latent capability to solve these problems but lacks reliability.
  • The solution diversity metrics (Appendix B, Figures 11, 12) show that diversity increases early in RL training but plateaus or declines in later iterations, consistent with models settling into a limited repertoire of strategies rather than discovering new ones.
  • Sample complexity is "extremely small" by classical RL standards — convergence in tens of thousands of rollouts rather than millions (Section 4.2) — inconsistent with genuine exploration of a large behavior space.

Mitigation status. The paper does not resolve this limitation. It explicitly identifies exploration as the primary bottleneck and suggests future directions: "We view the discovery of new techniques encouraging complex, rich exploration of reasoning problems as fundamental to progress in LLM reasoning capability" (Section 5). The paper mentions Tree of Thoughts (Yao et al., 2023) and evolutionary algorithms (Lehman et al., 2022) as promising directions but does not test them. The limitation is therefore identified but not mitigated — the paper diagnoses the disease but does not provide a cure.

6.2 Difficulty Estimation and Adaptive Allocation Are Not Present — All Problems Get the Same RL Treatment

The assumption or constraint. The paper applies RL uniformly across all problems in the training set — every question receives the same sampling budget during exploration (K = 96 per question for EI; N = 4 rollouts per question for PPO), the same reward structure, and the same number of training iterations. There is no mechanism for distinguishing easy from hard problems, no adaptive allocation of exploration budget, and no difficulty-conditioned policy. The paper acknowledges this implicitly by studying difficulty only through the lens of model size (7B vs. 13B) and initialization (pretrained vs. SFT), noting that larger models "engage in more diverse exploration" (Section 4.3) and that models with better generalization explore more diversely. But it never asks the natural follow-up question: within a given model, do easy problems benefit from the same RL treatment as hard problems?

The consequence. The uniform treatment of all problems likely wastes exploration budget. The paper shows that the pretrained model can already solve many problems with high probability — these are the problems driving the pass@96 scores — and these easy problems do not need 96 exploration rollouts per EI iteration. Conversely, the hardest problems (those the model rarely or never solves even with 96 samples) receive the same exploration budget but provide no useful training signal because no correct solutions are discovered. This is a direct parallel to the "compute-optimal test-time scaling" insight from other work in the literature: allocating a fixed budget uniformly across problems of varying difficulty is deeply suboptimal. For practitioners, this means the RL training procedure described in this paper is likely compute-inefficient in an absolute sense — it spends the same resources on problems where the model already has 90% pass@1 as on problems where it has 2% pass@96, getting diminishing returns on the former and zero returns on the latter.

What evidence exists in the paper. The paper provides indirect evidence for this limitation through its pass@96 analysis. The fact that a prompted pretrained 13B model achieves pass@96 = 0.83 (Section 4.2) means that for ~83% of GSM8K problems, the model can already generate a correct solution with 96 samples before any RL training. These problems are the "easy" majority, and RL training on them primarily improves maj@1 — making the model more consistent on problems it already knows. For the remaining ~17% of problems (the hard tail), the paper shows that pass@96 plateaued and did not improve meaningfully with additional RL training (Table 1: pass@96 improves from 0.72 to 0.79 for 7B and 0.84 to 0.83 for 13B — a 3–7 point gain that is partly explained by avoiding SFT overfit rather than by solving new problems). The paper also notes in Figure 2 that performance plateaus after 2 EI iterations for SFT-initialized models — additional iterations consume compute without improving accuracy, suggesting many problems have been "solved" (in the sense of reliable greedy decoding) and further training only helps on a diminishing subset.

The paper does not present a difficulty-stratified analysis of where RL gains come from — for example, breaking down maj@1 improvement by initial pass@1 quintile. Without this, the reader cannot distinguish between "RL helps on easy problems by making them more reliable" and "RL helps on medium problems by discovering new approaches."

Mitigation status. The paper does not address this limitation at all. It does not propose or test any difficulty-aware allocation scheme. The EI procedure's fixed K = 96 samples per question per iteration is presented as a hyperparameter choice rather than a resource allocation decision. The paper's finding that reducing K to 4 achieves comparable sample complexity (Section 4.2) hints that the current allocation is wasteful, but the paper does not explore whether some questions need K = 96 while others are fine with K = 4. A natural extension — using the ORM's predicted difficulty to allocate more exploration budget to problems of intermediate difficulty while saving budget on trivially easy or impossibly hard problems — is entirely absent.

6.3 Single Model Family and Narrow Task Domain: Generality of Findings Is Unverified

The assumption or constraint. All experiments use instruction-tuned Llama-2 7B and 13B models evaluated exclusively on two math word problem benchmarks: GSM8K and SVAMP. The paper makes claims about "RL for LLM reasoning" in general, but the empirical basis is restricted to a single model family performing arithmetic word problems. The paper acknowledges this limitation indirectly by discussing how the "unnatural distribution of text" in math reasoning tasks — "broken natural language and computations enclosed in <<x+y=z>> tags" (Section 4.3) — may affect training dynamics, particularly the need for KL constraints. This implies that different task domains (code generation, logical reasoning, scientific QA, multi-hop question answering) or different model families (non-Llama architectures, different pretraining distributions) could produce qualitatively different results.

The consequence. Several of the paper's key findings may not generalize beyond the tested combination of model and task:

  • The exploration bottleneck might be less severe for code generation tasks, where the output space is more structured and the pretraining corpus (GitHub) may provide a richer set of reasoning patterns. Models might explore more diversely in code than in grade-school math.
  • The sufficiency of sparse ground-truth reward depends on the task having a clean, verifiable correct answer. For open-ended reasoning tasks (summarization, argument generation, creative problem-solving), sparse binary rewards are unavailable, and the reliance on learned reward models (which the paper shows provide limited benefit) would be unavoidable.
  • The PPO vs. EI comparison might flip for tasks requiring robustness to noisy rewards. The paper explicitly notes that PPO's "protective machinery against reward model over-optimization" (Section 5, Discussion) is unnecessary for ground-truth rewards but becomes valuable in RLHF settings. For reasoning tasks without ground-truth answers — where a learned verifier must serve as both training reward and evaluation metric — PPO's KL constraint and clipped objective might provide critical protection that EI's simple filtering does not.
  • The KL penalty necessity (coefficient 0.05 for PPO) is attributed to the unusual text distribution of math reasoning. Tasks closer to natural language (dialogue-based reasoning, instruction following) might not need this constraint, changing the implementation complexity tradeoff between PPO and EI.

What evidence exists in the paper. The paper provides no cross-model or cross-domain evidence. The only variation in model scale is 7B vs. 13B, both within the same Llama-2 family. The paper notes observed differences between these scales — larger models explore more diversely (Appendix B, Figures 11, 12) and benefit less proportionally from RL (comparing SFT to EI gains: 7B improves from 0.42 to 0.485, a 6.5-point gain; 13B improves from 0.47 to 0.53, a 6-point gain) — but does not test whether these trends continue to 34B, 70B, or beyond. The paper's exploration conclusion ("models fail to explore significantly") is based entirely on 7B and 13B Llama-2 models; a 70B or 405B model might exhibit qualitatively different exploration behavior that could change the relative performance of EI vs. PPO or the effectiveness of dense rewards.

The external baselines in Table 1 (Llemma, GPT-3, GPT-4) are included for context but are not comparable — different model families, scales, and training procedures mean no conclusions about algorithmic generality can be drawn from them.

Mitigation status. The paper does not attempt to mitigate this limitation. It does not test on code generation benchmarks (HumanEval, MBPP), logical reasoning tasks (LogiQA, FOLIO), or multi-hop QA (HotpotQA). It does not experiment with non-Llama models (Mistral, Gemma, Qwen). The discussion section does not even flag this as a limitation — the paper presents its findings as general conclusions about "RL for LLM reasoning" without qualification about the specific model family or task domain tested. This is the most significant scope limitation of the paper, and a practitioner considering applying these methods to a different domain would need to treat the paper's conclusions as hypotheses to verify rather than established facts.

6.4 The ORM Overfit Confound Undermines the rerank@96 Metric and the EI Advantage Claim

The assumption or constraint. The Outcome-Based Reward Model used for both training-time reward signals and evaluation-time reranking (rerank@96) is trained on data generated by the best Expert Iteration model. The paper explicitly acknowledges this circularity in Section 4.1:

"At first glance this seems contradictory with relatively similar pass@96 performance. However, we believe a non-trivial percentage of this gap is due to overfit of the ORM to the EI model which was used to generate its training data."

This means the ORM — which is supposed to be an impartial verifier of solution quality — has been optimized to recognize correct solutions in the specific style and distribution produced by the EI-trained model. When applied to solutions from PPO or RCRL models, the ORM may systematically underrate them not because they are less correct, but because they deviate from the EI model's output patterns.

The consequence. This creates two problems for the paper's claims:

  1. The rerank@96 metric is biased. The paper reports rerank@96 scores as a primary evaluation metric alongside maj@1 and pass@96 (Tables 1, 2). For example, the 7B SFT-initialized EI model achieves rerank@96 = 0.73 vs. PPO's 0.69 (Table 1) — a 4-point gap that the paper presents as evidence of EI's superiority. But if part of this gap is ORM overfit, then rerank@96 is no longer a fair comparison metric. The paper does not quantify how much of the gap is due to overfit, stating only that it is "non-trivial." This means the reader cannot distinguish between "EI produces solutions that are genuinely better under a fair verifier" and "EI's solutions look more like the data the verifier was trained on."

  2. The ORM-guided training results are confounded. When an ORM is used as a reward source during PPO or EI training, the model is optimizing against the same ORM that was trained on EI-generated data. This creates an asymmetry: EI models are being trained to produce solutions that look like what the ORM expects, while PPO models, starting from different initializations and using different optimization objectives, may produce solutions that the ORM scores less accurately. The paper's finding that "the ORM slightly improves PPO performance and negligibly impacts EI performance" (Section 4.1) could be explained by this asymmetry — the ORM provides a cleaner signal for EI models because it was trained on their predecessors' outputs.

What evidence exists in the paper. The paper provides only the acknowledgment quoted above, without any quantitative analysis of the overfit effect. No experiment is presented where:

  • An ORM is trained on PPO-generated data and used to evaluate EI models (cross-validation of the verifier)
  • An ORM is trained on a mix of EI and PPO outputs
  • The correlation between ORM scores and ground-truth correctness is compared across EI, PPO, and RCRL model outputs
  • The impact of ORM overfit on rerank@96 differences is estimated through any statistical or experimental procedure

The paper's SVAMP results (Table 3) report only maj@1, avoiding the rerank@96 issue on that benchmark, but this also means the reader cannot compare the overfit effect across datasets.

Mitigation status. The paper does not mitigate this limitation. It acknowledges it and moves on, treating the rerank@96 metric as still informative despite the known bias. For the EI vs. PPO comparison, the paper's maj@1 and pass@96 results (which do not depend on the ORM) provide cleaner evidence of relative algorithm performance, and the paper's key claims about EI's superiority are supported by these metrics as well as rerank@96. However, the absolute rerank@96 numbers and the relative ranking of algorithms on this metric should be treated as unreliable. A practitioner interested in deploying verifier-based reranking would need to train their ORM on data from whatever model they intend to deploy, performing the kind of cross-validation that this paper omits.

6.5 The Difficulty Estimation Cost for Compute-Optimal Allocation Is Absent — But the Uniform Budget Is Wasteful

The assumption or constraint. The paper's experimental design gives every training question the same exploration budget: K = 96 rollouts per question per EI iteration, N = 4 rollouts per question per PPO batch. This uniform allocation ignores problem difficulty, and the paper does not measure or account for any difficulty estimation overhead because it does not attempt difficulty-aware allocation at all. This is distinct from Limitation 6.2, which addressed the wastefulness of uniform allocation. This limitation is about the missing measurement: the paper does not report what fraction of the exploration budget is spent on problems that provide no useful training signal (either because the model already solves them reliably or because it never solves them), making it impossible for practitioners to estimate the waste or to design a more efficient procedure.

The consequence. A practitioner attempting to replicate or deploy this method cannot know how much compute is being wasted. The paper reports total sample complexity (~60,000 rollouts for PPO, ~10^6 for EI with K=96) without decomposing this into productive vs. unproductive exploration. This has several downstream effects:

  • Cost estimation is impossible. If a practitioner has a different problem distribution — say, one skewed toward harder problems than GSM8K — the paper's sample complexity numbers may not apply because a larger fraction of problems would fall into the "impossibly hard" category where exploration produces no correct solutions.
  • No guidance on budget allocation. The paper's finding that EI with K=4 achieves comparable sample complexity to PPO (Section 4.2) suggests the K=96 default is wasteful, but the paper does not explore whether K=4 is sufficient for all questions or only for easy ones. A practitioner might need K=96 for the hardest 10% of problems and K=1 for the easiest 50%, but the paper provides no data to support such a schedule.
  • The "no SFT data" results are artificially pessimistic about EI. The pretrained-initialized EI results (Table 2: maj@1 = 0.23 for 7B) might be substantially improved by allocating more exploration budget to questions where the model occasionally succeeds and less to questions where it always fails. The uniform K=96 allocation may be simultaneously too little for some questions and too much for others.

What evidence exists in the paper. The paper provides suggestive but incomplete evidence of this waste:

  • The pass@96 analysis shows that ~83% of problems are solvable by the pretrained 13B model with 96 samples (Section 4.2). For these problems, EI with K=96 is producing many redundant correct solutions — the exploration is "discovering" solutions the model already knew.
  • Conversely, for the ~17% of problems where pass@96 is 0, EI with K=96 produces zero correct solutions per iteration — the exploration budget is entirely wasted.
  • Figure 2 shows EI performance plateauing after n=2 iterations for SFT-initialized models, meaning iterations 3 and beyond provide diminishing returns — but the paper doesn't decompose this plateau by difficulty to show whether it's because easy problems are already solved or because hard problems are unsolvable.
  • Appendix B, Figures 11 and 12 show solution diversity plateauing or declining after early iterations, consistent with the model exhausting its repertoire on the solvable problems while making no progress on unsolvable ones.

Mitigation status. The paper does not address this limitation. It does not report difficulty-stratified training dynamics, does not propose or test adaptive exploration budgets, and does not measure the fraction of exploration rollouts that produce novel correct solutions vs. redundant correct solutions vs. incorrect solutions. The paper's own data strongly suggests that a difficulty-aware allocation would improve efficiency, but the paper stops short of designing or testing one. This is a missed opportunity: the paper's comprehensive experimental framework — multiple algorithms, reward types, and initializations — would have been ideal for studying how the value of additional exploration varies with problem difficulty.

6.6 The Revision Model and Search Mechanisms Are Studied in Isolation — No Combined Approach Is Tested

The assumption or constraint. The paper studies three RL algorithms (PPO, EI, RCRL) as independent policy improvement mechanisms, evaluating each in isolation. It does not combine RL fine-tuning with complementary test-time strategies that could amplify its benefits. Specifically:

  • No combination with search or verifier-guided decoding. The ORM is used for evaluation (rerank@96) and as a training reward source, but the paper never tests whether RL fine-tuning improves the model's performance when combined with test-time search (beam search, best-of-N with ORM reranking, majority voting). The rerank@96 metric approximates this but is confounded by ORM overfit (Limitation 6.4), and the paper doesn't analyze how RL affects the scalability of test-time compute — i.e., whether the RL-trained model benefits more from additional samples than the SFT baseline.
  • No combination with prompt engineering. The pretrained-initialized experiments use a fixed 2-shot prompt; the SFT-initialized experiments use the SFT model's formatting. The paper does not explore whether RL fine-tuning interacts with prompting strategies (chain-of-thought, self-consistency, etc.).
  • No iterative refinement or self-correction at test time. The paper's RCRL experiments (Section 3, "Return Conditioned RL") attempt to train a model that can condition on desired return, but this is a training-time mechanism, not a test-time strategy. The paper does not test whether an RL-trained model is better at self-correcting its own errors when given multiple attempts — the pass@96 metric captures the potential for this but not the model's actual self-correction ability.

The consequence. The paper's conclusions about RL's benefits may be understated in a deployment context where test-time compute strategies are available. If RL fine-tuning makes the model's correct solutions more distinguishable from incorrect ones (even if it doesn't increase the absolute number of solvable problems), then the combination of RL + ORM reranking could yield practical gains beyond what maj@1 or pass@96 alone suggest. Conversely, if RL fine-tuning makes the model less diverse in ways that harm majority voting or best-of-N selection, the paper's maj@1 improvement could be offset by degraded ensemble performance — a tradeoff the paper cannot quantify because it never evaluates ensemble strategies as a function of RL training.

More broadly, the paper positions itself as studying "RL for LLM reasoning" but only examines RL as a training-time fine-tuning approach, not as part of a broader inference-time reasoning system. For practitioners, the relevant question is not "does RL improve maj@1?" but "does RL + test-time compute outperform SFT + test-time compute under a fixed total budget?" The paper provides pass@96 data that partially answers this — showing that RL doesn't expand the set — but doesn't test the combination of RL with verifier-guided selection or search, which could show benefits that individual metrics (maj@1, pass@96 in isolation) miss.

What evidence exists in the paper. The paper's metrics provide partial evidence that is difficult to interpret without combination experiments:

  • The maj@96 results (Table 1: 7B EI maj@96 = 0.66 vs. SFT maj@96 = 0.58) suggest RL improves majority voting performance, but this could be due to the improved maj@1 (more samples land on the correct answer) or to genuinely better ensemble behavior.
  • The rerank@96 results (Table 1: 7B EI rerank@96 = 0.73 vs. SFT rerank@96 = 0.64) suggest RL improves verifier-based selection, but the ORM overfit confound makes this uninterpretable.
  • The paper does not report how the relationship between sample count and accuracy changes with RL training — for instance, whether the EI model's accuracy scales better with additional samples than the SFT model's. The only multi-sample metrics are at K=96; intermediate sample counts (maj@4, rerank@8, pass@16, etc.) are never reported.

Mitigation status. The paper does not address this limitation. It does not test any combination of RL fine-tuning with test-time search, iterative refinement, or verifier-guided decoding. The paper's focus is strictly on training-time algorithm comparison, and it leaves the integration with inference-time strategies as implicitly "future work" without explicitly flagging it as such. Given that the paper's central finding is that RL improves reliability rather than capability (maj@1 up, pass@96 flat), combining RL with test-time strategies that exploit this improved reliability (e.g., using the RL model as part of a self-consistency ensemble) is a natural next step that the paper does not take.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around RL for LLM fine-tuning from algorithm-centric optimization (which RL algorithm is "best"?) to exploration-centric diagnosis (why do all algorithms perform similarly, and what does that tell us about the fundamental limits of RL fine-tuning?). The field's implicit assumption — inherited from the RLHF literature — was that PPO's clipped surrogate objective, advantage estimation, and KL constraints provided meaningful optimization advantages that justified the implementation complexity. This paper systematically dismantles that assumption for reasoning tasks: Expert Iteration, a simple filter-and-distill procedure with no value function, no advantage estimation, and no clipped updates, matches or exceeds PPO in most configurations while requiring comparable sample complexity once per-prompt oversampling is controlled (Section 4.2, K=4 ablation).

The methodological shift is from "which RL algorithm should I use?" to "how much and what kind of exploration data do I need?" The paper demonstrates that the pretraining prior is so dominant in determining what solutions the model can generate that the specific mechanism used to update parameters on those solutions matters far less than whether the solutions are diverse and correct. This reframes RL fine-tuning not as an optimization problem (finding the policy that maximizes expected return) but as a data generation problem (generating the most diverse set of correct demonstrations for behavior cloning). The RL algorithm's primary role is not to cleverly update parameters but to generate exploration data — and on this dimension, EI's rejection sampling and PPO's on-policy exploration produce similarly useful training corpora.

This is more than an incremental benchmarking result. It resolves a tension that had been building in the literature between papers that found complex RL algorithms necessary (Ouyang et al., 2022; Bai et al., 2022 using PPO for RLHF) and papers that found simple filtering-and-retraining sufficient (Gulcehre et al., 2023; Dong et al., 2023; Dubois et al., 2023 using ReST, RAFT, and AlpacaFarm). The paper's resolution is clean: PPO's protective machinery (clipping, KL penalty) matters when the reward signal is noisy and over-optimizable (as in RLHF with learned reward models), but not when the reward signal is a clean ground-truth correctness check. On reasoning tasks with verifiable answers, the reward model is the environment itself, and there is nothing to over-optimize against. This unification explains why Expert Iteration dominates in reasoning benchmarks but PPO remains standard in preference-tuning — and it provides practitioners with a clear decision rule for when to pay PPO's complexity cost.

The paper's most landscape-changing contribution may be its negative result on exploration. The finding that RL fine-tuning does not expand the set of solvable problems — that pass@96 scores cluster within ~3% across all algorithms and initializations (Tables 1, 2), and that a prompted pretrained 13B model with zero SFT data already achieves pass@96 = 0.83 versus an SFT + EI model's 0.83 (Section 4.2) — places a hard ceiling on what current RL fine-tuning methods can achieve for reasoning. This finding redirects research attention from RL algorithm development toward three more promising directions: (1) improving the pretraining prior so the model's initial solution manifold covers more hard problems, (2) developing prompting, search, or tool-use strategies that can guide the model to solution regions beyond standard autoregressive sampling, and (3) designing exploration mechanisms that explicitly push the model beyond its pretraining distribution. Each of these is a fundamentally different research program than "try a new RL algorithm," and the paper's evidence makes a compelling case that the RL-algorithm program has diminishing returns at current model scales.

The paper also establishes the maj@1 vs. pass@96 tradeoff as a diagnostic tool for distinguishing reliability improvements from capability expansions. Prior work (Cobbe et al., 2021) had observed the tradeoff in SFT but not explained it. The paper identifies the mechanism — static dataset diversity collapse during SFT — and demonstrates that online RL breaks the tradeoff by generating its own training data (Table 4: SFT 2 + PPO improves maj@1 from 0.36 to 0.43 while preserving pass@96 that continued SFT destroys). This diagnostic framework provides a template for future work evaluating any fine-tuning method: if maj@1 improves but pass@96 does not, the method is improving reliability on existing capability; if pass@96 also improves, the method is expanding the capability frontier. The paper shows that current RL methods are almost entirely in the "reliability improvement" category — a finding that should recalibrate expectations about what RL fine-tuning can deliver.

Follow-Up Research This Work Enables

Difficulty-stratified analysis of RL gains: where does the maj@1 improvement actually come from? The paper's most actionable open question is whether RL fine-tuning helps most on easy problems (making already-solvable problems more reliable), medium problems (discovering occasional solutions that the model misses under greedy decoding), or hard problems (expanding the frontier). The paper provides aggregate pass@96 numbers showing minimal frontier expansion, but it never decomposes maj@1 gains by initial problem difficulty. A strong follow-up would: (a) bin the GSM8K test set into quintiles by the pretrained model's pass@1 or the SFT model's maj@1, (b) report RL-improvement in maj@1 separately for each bin, (c) compare this against what would be expected from "reliability improvement" alone (improving greedy selection from the existing solution distribution) versus "capability expansion" (increasing the probability mass on correct solutions that previously had near-zero probability). This requires no new algorithms — just the difficulty-stratified analysis that the paper's experimental framework already supports but didn't perform. The result would directly inform practitioners about which problem types benefit from RL fine-tuning investment and which require better pretraining or prompting.

Cross-training ORMs to quantify and correct for verifier overfit. The paper acknowledges that its rerank@96 metric overstates EI's advantage because the ORM was trained on EI-generated data (Section 4.1). A necessary follow-up would train three separate ORMs — one on SFT model outputs, one on EI model outputs, one on PPO model outputs — and then cross-evaluate: score each model's solutions using each ORM, producing a 3×3 matrix of (model, ORM-training-data) rerank@96 accuracies. This would: (a) quantify the overfit effect (the diagonal entries minus the off-diagonal average), (b) provide an unbiased estimate of each algorithm's true verifier-reranking performance (the off-diagonal entries, using ORMs not trained on that algorithm's data), and (c) determine whether the paper's conclusion that "EI performs best" survives this correction. The paper's existing experimental infrastructure — ORM training pipeline, multiple model checkpoints, evaluation protocol — makes this straightforward to execute. A negative result (EI no longer outperforms PPO after ORM bias correction) would significantly change the paper's headline finding, making this a high-priority validation experiment.

Scaling model size to test the exploration bottleneck hypothesis. The paper's central finding — that models fail to explore significantly beyond their pretraining neighborhood — is based entirely on 7B and 13B Llama-2 models. The paper itself provides evidence that this may not hold at larger scales: "larger models engaged in more diverse exploration" (Section 4.3), and the 13B model generates "significantly more diverse solutions than 7B at every iteration" (Appendix B, Figures 11, 12). A critical follow-up would replicate the EI vs. PPO comparison at 34B, 70B, and (if computationally feasible) 405B parameter scales, measuring: (a) whether pass@96 gains from RL increase with model size (indicating that larger models can expand their capability frontier through exploration), (b) whether the EI vs. PPO performance gap changes with scale (if PPO's advantage estimation becomes more valuable when there is a larger behavior space to navigate), and (c) whether the sample complexity parity (EI ~ PPO) holds at larger scales or whether PPO's sample efficiency advantage emerges when exploration is more diverse. This experiment is computationally expensive but conceptually straightforward — it's the same protocol as the paper's existing experiments, just at larger scale. If the exploration bottleneck persists at 70B, the finding becomes a robust principle rather than a scale-limited observation; if it breaks, we learn something fundamental about how pretraining scale interacts with fine-tuning exploration.

Combining RL fine-tuning with test-time compute scaling — does RL improve sample efficiency of inference-time strategies? The paper evaluates RL models using static metrics (maj@1, pass@96) but never tests whether RL fine-tuning changes how model accuracy scales with inference-time compute. A natural extension would take SFT, EI, and PPO models and measure accuracy as a function of sample budget (maj@N for N = 1, 2, 4, 8, ..., 96), producing scaling curves for each model. The hypothesis: RL fine-tuning increases the probability mass on correct solutions, which should steepen the majority-vote scaling curve (accuracy rises faster with N for RL models than for SFT models, even if pass@96 is similar). A stronger version: use an independently-trained ORM to do best-of-N weighted selection, and measure whether RL models benefit more from verifier-guided selection than SFT models. If RL models show steeper scaling (more accuracy gain per additional sample), this would demonstrate a practical benefit of RL fine-tuning that the paper's current metrics miss — RL would be valuable not because it expands the capability frontier, but because it makes the model more efficient at converting inference compute into accuracy. This connects directly to the compute-optimal test-time scaling literature and would position RL fine-tuning as a method for improving the returns to inference-time compute rather than improving raw capability.

Replication on code generation: does the exploration bottleneck generalize? All the paper's experiments use math word problems (GSM8K, SVAMP). The paper hypothesizes that the "unnatural distribution of text in reasoning tasks" — broken natural language with computation tags — contributes to the need for KL constraints and possibly to the exploration bottleneck. Code generation is a natural counterpoint: the output space (programming languages) is more structured than grade-school math reasoning, the pretraining corpus (GitHub) provides a richer and more diverse set of reasoning patterns, and the reward signal (unit tests) is similarly clean. A replication on HumanEval or MBPP using CodeLlama or DeepSeek-Coder models would test whether: (a) EI still matches or exceeds PPO (or whether the structured output space makes PPO's advantage estimation more useful), (b) pass@96 improves with RL fine-tuning (indicating genuine capability expansion is possible in code), and (c) the sample complexity parity holds. A finding that RL fine-tuning does expand the pass@k frontier for code generation would suggest the exploration bottleneck is task-specific (math reasoning is somehow special) rather than a general property of LLM fine-tuning, redirecting attention toward understanding what properties of the pretraining distribution enable or inhibit fine-tuning exploration.

Novelty bonuses and explicit exploration mechanisms for LLM fine-tuning. The paper's conclusion that "discovery of new techniques encouraging complex, rich exploration of reasoning problems [is] fundamental to progress" (Section 5) is a diagnosis without a prescription. A natural follow-up would test whether classical RL exploration mechanisms — count-based exploration bonuses, curiosity-driven intrinsic rewards, entropy regularization — can push LLMs beyond their pretraining neighborhood during fine-tuning. The specific experiment: add an intrinsic reward to PPO training that is proportional to the novelty of generated solutions, where novelty is measured by embedding similarity to previously generated solutions (semantic novelty) or by n-gram overlap with the SFT dataset (dataset novelty). Track whether: (a) pass@96 improves beyond what standard PPO achieves (indicating the exploration bonus discovers genuinely new solution strategies), (b) solution diversity increases more than standard PPO (Figures 11, 12 show this plateaus; a novelty bonus should prevent or delay this plateau), and (c) the maj@1 improvement is maintained or traded off against diversity gains. This experiment is directly motivated by the paper's finding that "regardless of the type of algorithm used... all student models engage in similar exploration" — the question is whether we can break this uniformity by explicitly rewarding exploration. A negative result (novelty bonuses don't help) would strengthen the paper's implicit claim that the pretraining prior is fundamentally un-escapable through current methods; a positive result would open a new research direction in LLM fine-tuning.

Practical Applications and Downstream Use Cases

Cost-efficient supervised fine-tuning with self-generated data. The paper's Expert Iteration procedure — generate many solutions, filter for correct ones, retrain — provides a concrete recipe for improving model accuracy without additional human-labeled data. For a team that already has a base model and a task with verifiable answers (math problem solving, code generation with unit tests, factual QA with known answers), the paper's results quantify the expected gains: ~6.5 percentage points maj@1 improvement on GSM8K for a 7B model (Table 1: SFT 0.42 → EI 0.485) using only the model's own generated solutions, with training taking "about a day" (Section 4.1). The key practical insight is the model reset design choice: fine-tune from the pretrained base model on filtered correct solutions rather than continuing from the SFT checkpoint. This costs no additional human annotation and requires only inference compute (generating solutions) plus standard fine-tuning infrastructure. For organizations with access to a base model and a task-specific SFT dataset, this is a low-cost, low-risk procedure that reliably improves greedy decoding accuracy without the instability risks of PPO training. The paper's finding that K=4 samples per question achieves comparable sample efficiency to K=96 (Section 4.2) means the procedure can be run economically: for a 7,000-question dataset, 4 samples per question × 4 iterations = 112,000 total generations, which is feasible on a single 8-GPU node.

Avoiding SFT diversity collapse in deployment. The paper's diagnostic of the maj@1 vs. pass@96 tradeoff (Table 4) has immediate practical implications for teams deploying models with test-time compute strategies (majority voting, verifier reranking). The finding is that continuing SFT beyond 2 epochs improves greedy decoding accuracy (maj@1: 0.36 → 0.42) but reduces ensemble accuracy (pass@96: 0.76 → 0.72) and the number of unique correct solutions per question (3.7 → 2.9). For a deployment that uses majority voting (generating ~8–16 samples and taking the consensus answer), the 4-epoch SFT model would actually perform worse than the 2-epoch model, despite its higher maj@1 — because the ensemble depends on diverse correct solutions, which SFT overfit destroys. The practical prescription is: (a) monitor both maj@1 and pass@N during SFT training, not just validation loss, (b) stop SFT when pass@N begins to decline even if maj@1 is still improving, and (c) apply a round of Expert Iteration or PPO fine-tuning after this point to recover the lost diversity while keeping the maj@1 gains. The paper's evidence that RL fine-tuning of the 2-epoch SFT model improves maj@1 to 0.43 without sacrificing diversity (Section 4.3, Table 4) provides a concrete procedure for escaping the tradeoff.

Lightweight RL fine-tuning for pretrained models without SFT data. The paper's most surprising practical result is that RL fine-tuning works even without any supervised fine-tuning data — starting from a prompted pretrained model with maj@1 ~0.05 on GSM8K, PPO reaches 0.40 (13B) with only ~30,000 rollouts (Table 2, Section 4.2). This matters for domains where human-written reasoning traces are expensive or unavailable. The recipe: (a) start with a base pretrained LLM, (b) construct a small prompt (2–4 examples) from whatever labeled examples are available, (c) run PPO or EI with sparse ground-truth reward (the model generates answers, checks against answer keys), (d) after the first round of data generation, remove the few-shot prompt and continue training on the model's own generated solutions. The paper demonstrates this procedure produces a model with maj@1 = 0.40 (13B) on GSM8K, and on SVAMP reaches 0.69 (13B) — a >50 percentage point improvement over the prompted baseline (0.05 → 0.69, Table 3). For practitioners in domains with answer keys but no reasoning traces (many educational, scientific, and industrial settings where only final answers are recorded), this provides a path to bootstrapping a reasoning-capable model from a pretrained checkpoint without human annotation of intermediate steps. The key practical consideration is the prompt construction: the paper used 2 examples from GSM8K's validation set, and these few examples must be sufficient to elicit recognizable solution formats from the pretrained model (since exploration quality depends on the prompt).