ArXiv: 2412.16145

🎯 Pitch

OREO shows that a 1.5B LLM can solve 52.5% of MATH problems by learning from unpaired, sparsely-rewarded trajectories—without ever needing online exploration or human preferences—and the resulting value function improves accuracy by another 17.9% at test time through simple beam search.

1. Executive Summary

This paper introduces OREO (Offline REasoning Optimization), an offline reinforcement learning method for improving LLM multi-step reasoning that jointly learns a policy model and an explicit value function by optimizing the soft Bellman Equation. Evaluated on mathematical reasoning benchmarks (GSM8K, MATH) with Qwen-2.5-Math-1.5B and DeepSeekMath-7B-Instruct, and on embodied agent control (ALFWorld) with MiniCPM-2B, OREO eliminates two key limitations of Direct Preference Optimization—the requirement for paired preference data (learning instead from unpaired trajectories with only sparse terminal rewards) and the lack of fine-grained credit assignment (achieved via step-level value function training that assigns per-step advantage estimates). The method delivers consistent improvements: a 1.5B model reaches 52.5% accuracy on MATH (a 10.5% relative gain over SFT), while the 7B model achieves 85.9% on GSM8K and 49.2% on MATH, and in ALFWorld unseen environments OREO yields a 17.7% relative improvement over baselines. The learned value function further enables test-time beam search that provides up to a 17.9% relative improvement over greedy decoding on MATH, establishing that explicitly parameterized value functions outperform implicit policy-derived value estimates for guiding search—but only when trained through the soft Bellman consistency objective rather than through preference-based approximations.

2. Context and Motivation

The Core Problem: Offline RL for LLM Reasoning Is Stuck Between Expensive Online Methods and Ineffective Preference-Based Ones

The fundamental question this paper tackles is: How do we improve an LLM's multi-step reasoning ability using only a static offline dataset, without requiring costly online interaction or painstakingly constructed pairwise preference data?

This matters because multi-step reasoning—the ability to chain together logical steps to solve complex problems—is one of the most valuable capabilities of modern LLMs. Whether it's solving a competition-level math problem (Hendrycks et al., 2021), navigating a household environment as an embodied agent (Shridhar et al., 2020), or browsing the web to complete a task (Deng et al., 2024), the ability to reason step-by-step determines whether an LLM is genuinely useful or merely plausible-sounding. Improving this capability is therefore a central challenge for the field, with direct implications for deploying LLMs in education, scientific research, software engineering, and autonomous systems.

However, the dominant approach to improving LLM reasoning through reinforcement learning—using Proximal Policy Optimization (PPO, Schulman et al., 2017) in an online RLHF pipeline (Ouyang et al., 2022)—is prohibitively expensive for most users (Hu et al., 2023). Online RL requires generating language on-the-fly during training, interacting with an environment (or reward model) to collect fresh experience, and maintaining multiple models in memory simultaneously (the policy, a reference model, a value model or reward model, and potentially a critic). This creates a high barrier to entry that limits who can improve LLM reasoning through RL to a small number of well-resourced organizations. The paper explicitly names this cost as a key motivator:

"tuning LLMs with Proximal Policy Optimization (PPO) is often prohibitively expensive for most users, which limits practical applications" (Section 1)

This cost asymmetry creates an urgent practical need for methods that can achieve similar improvements using only pre-collected, offline data—datasets that many practitioners already possess or can generate once and reuse.

The Gap: Offline Methods Exist but Fall Short for Multi-Step Reasoning

If online RL is too expensive, the natural alternative is offline RL—methods that learn from a fixed dataset without any live interaction. The most prominent such method in the LLM alignment literature is Direct Preference Optimization (DPO, Rafailov et al., 2024b), which has shown considerable success in aligning LLM responses with human preferences for general dialogue and instruction-following tasks. However, the paper identifies that DPO is specifically ill-suited for multi-step reasoning tasks due to two structural limitations that are not merely incidental but are fundamental consequences of DPO's theoretical formulation.

Limitation 1: DPO Requires Paired Preference Data, Which Is Unnatural for Reasoning Tasks

DPO is derived from the Bradley-Terry preference model (Bradley and Terry, 1952), which assumes that the probability of one response being preferred over another is determined by the relative exponential rewards of the two responses. This requires training data in the form of pairs (τw, τl)—a winning response and a losing response—where the preference is expressed as τw ≻ τl (Equation 5 in the paper). For chat alignment, this is natural: human annotators can compare two dialogue responses and indicate which one they prefer.

For multi-step reasoning, however, this pairing is artificial and wasteful. The paper explains why in Section 3.3:

"While the BT model is reasonable for a general dialogue system where the reward can only be implicitly inferred from human preference, it's unnecessary for multi-step reasoning tasks where a ground-truth reward exists."

In mathematical reasoning, embodied agent control, and similar tasks, the reward signal is sparse but objective: a math answer is either correct or incorrect (verified by a grading script), and an agent task either succeeds or fails (the goal state is reached or not). Each trajectory carries an unambiguous binary label—success or failure—without any need for relative comparison. To apply DPO to these tasks, prior work has had to retroactively construct preference pairs from independently collected trajectories (Song et al., 2024; Yuan et al., 2024; Pang et al., 2024), a process that discards information (why should every success be paired with a failure in a specific way?) and inflates data complexity (N independent trajectories become O(N²) possible pairs to manage). The paper characterizes this as "an inefficient usage of offline data" (Section 3.3).

The empirical consequences of this are visible in the paper's results (Table 1): DPO provides only marginal improvement over the SFT baseline on math reasoning tasks (e.g., 74.4 vs. 73.5 on GSM8K for the 1.5B model; 82.4 vs. 82.9 for the 7B model), and in some cases DPO underperforms the SFT checkpoint it was initialized from. This is consistent with prior findings that "the direct usages of DPO are not all successful" for reasoning (Yuan et al., 2024; Chen et al., 2024b; Section 2.2).

Limitation 2: DPO Treats All Tokens Uniformly, Preventing Fine-Grained Credit Assignment

The second limitation is more subtle and is grounded in DPO's mathematical derivation, which the paper carefully unpacks in Section 3.3. DPO's loss function (Equation 6) operates by telescoping the soft Bellman Equation across the entire trajectory:

∑t=0T−1r(st,at)=V∗(s0)+∑t=0T−1βlog⁡π∗(at∣st)πref(at∣st)\sum_{t=0}^{T-1} r(s_t, a_t) = V^*(s_0) + \sum_{t=0}^{T-1} \beta \log \frac{\pi^*(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)}

This equation—Equation 4 in the paper—collapses the per-step relationship between the optimal policy and its value function into a single aggregate statement about the whole trajectory. The consequence is that DPO's loss function (Equation 6) compares the total log-ratio of policy to reference probabilities across the entire winning and losing trajectories, without ever attending to individual steps:

L=−E(τw,τl)∼D[log⁡σ(∑t=0T−1βlog⁡π∗(atw∣stw)πref(atw∣stw)−∑t=0T−1βlog⁡π∗(atl∣stl)πref(atl∣stl))]\mathcal{L} = -\mathbb{E}_{(\tau^w, \tau^l) \sim \mathcal{D}} \left[ \log \sigma \left( \sum_{t=0}^{T-1} \beta \log \frac{\pi^*(a^w_t | s^w_t)}{\pi_{\text{ref}}(a^w_t | s^w_t)} - \sum_{t=0}^{T-1} \beta \log \frac{\pi^*(a^l_t | s^l_t)}{\pi_{\text{ref}}(a^l_t | s^l_t)} \right) \right]

This is a trajectory-level credit assignment: the model learns that the whole winning trajectory is better than the whole losing trajectory, but receives no signal about which specific steps made the difference. For multi-step reasoning tasks where "correctness often depends on a few key tokens" (Section 3.3)—a single arithmetic error in a 20-step solution, a single misidentified object in an agent plan, a single logical misstep in a proof—this is a critical weakness. The model has no mechanism to learn that certain types of steps are particularly error-prone or that certain reasoning patterns reliably lead to failure, because all tokens in the winning trajectory are encouraged equally and all tokens in the losing trajectory are discouraged equally.

The paper formalizes this point by contrasting DPO's telescoped objective with the full soft Bellman Equation (Equation 3), which applies at every time step:

V∗(st)−V∗(st+1)=r(st,at)−βlog⁡π∗(at∣st)πref(at∣st)V^*(s_t) - V^*(s_{t+1}) = r(s_t, a_t) - \beta \log \frac{\pi^*(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)}

When this per-step consistency is enforced (as OREO does), the model can learn that the value drops at specific steps (indicating an error was introduced) and rise at others (indicating progress), enabling fine-grained credit assignment. DPO's trajectory-level telescoping destroys this granularity.

The Simpler Alternative: Rejection Sampling Works but Leaves Value on the Table

Given DPO's limitations, a common practical approach for improving reasoning from offline data is rejection sampling (also known as STaR (Zelikman et al., 2022), RAFT (Dong et al., 2023), or REST (Gulcehre et al., 2023)): generate multiple candidate solutions for each training problem, filter out the incorrect ones, and fine-tune the model on only the successful trajectories. This is simple, requires no paired data, and is often effective—as the paper's results confirm (Table 1: rejection sampling achieves 74.9 on GSM8K and 50.3 on MATH for the 1.5B model, outperforming DPO).

However, rejection sampling has a fundamental limitation that the paper identifies (Section 1):

"it fails to fully exploit the offline dataset's potential—particularly the opportunity to learn from failure experience and enhance model robustness"

When you discard all incorrect trajectories, you throw away a rich signal about what not to do. This is especially costly because, in multi-step reasoning, incorrect trajectories often diverge from correct ones at only one or two critical steps—the rest of the trajectory may contain perfectly valid reasoning. By seeing only successes, the model never learns to recognize and avoid the specific failure patterns that cause otherwise good reasoning to go wrong. This manifests empirically in the paper's iterative training results (Figure 2): rejection sampling shows signs of saturation by the third iteration, with diminishing returns from additional data, while OREO—which learns from both successes and failures—continues to improve.

The paper cites Kumar et al. (2022) to ground this observation in the offline RL literature, where it is well-established that behavioral cloning (learning only from successful demonstrations) can be suboptimal compared to methods that also learn from negative examples, particularly when the dataset contains a mix of high-quality and low-quality trajectories.

Where Process Reward Models Fit and What They Miss

A parallel line of work attempts to address the credit assignment problem by training Process Reward Models (PRMs)—models that score each intermediate reasoning step as correct or incorrect, rather than only judging the final answer (Lightman et al., 2023; Wang et al., 2024a,b; Luo et al., 2024). The standard training approach for PRMs is Monte Carlo rollout: from each intermediate step in a solution, generate multiple possible completions, check which ones reach the correct final answer, and use the success rate as a soft label for that step's correctness (Wang et al., 2024a).

This is a natural idea and has shown promise for guiding test-time search. However, the paper points out (Section 2.2) that existing PRM training methods involve "extensive heuristic designs"—decisions about how many rollouts to perform, how to aggregate their results, whether to use binary or soft labels, how to handle cases where rollouts diverge significantly from the original trajectory, etc. These heuristics make PRM training an art as much as a science, and they are not derived from a unified theoretical framework.

The paper's key insight—developed formally in Sections 3.2 and 4—is that the value function learned through the soft Bellman Equation is, in fact, a principled generalization of the PRM. When trained with the MSE objective in Equation 8, the value function Vφ(st) estimates the expected future reward from state st, which is exactly what a PRM is trying to approximate through Monte Carlo rollouts. The difference is that OREO provides a coherent theoretical foundation (maximum-entropy RL, soft Bellman consistency) that eliminates the need for ad-hoc design choices. As the paper states:

"Our principled formulation removes the need for the extensive heuristic designs commonly required in prior works" (Section 4.4)

Unifying the Fragmented Landscape

The paper's positioning can be understood as an attempt to unify and improve upon three fragmented strands of prior work, each of which captures part of the solution but none of which fully addresses the problem:

  1. Preference-based methods (DPO, KTO): These are practical (offline, no environment interaction) but structurally misaligned with reasoning tasks—they require artificial paired data and cannot do credit assignment. The paper shows that DPO's trajectory-level telescoping of the Bellman Equation (Section 3.3) is mathematically why it fails here.

  2. Rejection sampling (STaR, RAFT, ReST): These are simple and use the natural success/failure signal available in reasoning tasks, but discard all failure data and plateau with iterative training. The paper shows that OREO's ability to learn from failed trajectories (via the value function's estimates on unsuccessful states) is what sustains iterative improvement (Figure 2).

  3. Process reward models (PRMs): These enable fine-grained credit assignment and test-time search, but are built on heuristic training procedures without a unifying theory. The paper shows that PRMs are essentially a special case of the value function in maximum-entropy RL (Section 4.4), and provides a principled training objective.

OREO is positioned as the method that synthesizes the strengths of all three: it is offline (practical), it uses unpaired data with sparse rewards (natural for reasoning), it learns from both successes and failures (no information discarded), and it provides an explicit value function for credit assignment and test-time search (principled, not heuristic). The paper makes this synthesis explicit in its abstract and introduction, framing OREO as addressing "the limitations of DPO in reasoning with a principled solution" (Section 2.2) while noting that its value function training objective is "essentially similar" to PRM Monte Carlo rollout but "removes the need for extensive heuristic designs" (Section 4.4).

The Theoretical Bridge: Maximum-Entropy RL and the Soft Bellman Equation

The intellectual backbone that makes this synthesis possible is the maximum-entropy reinforcement learning framework (Ziebart, 2010; Nachum et al., 2017; Haarnoja et al., 2017), which the paper connects to the RLHF objective in Section 3.2. The argument proceeds as follows:

The standard RLHF objective (Equation 1) maximizes expected reward with a KL penalty that keeps the learned policy close to a reference policy:

π∗=arg⁡max⁡πE(s0,...,sT)∼ρπ∑t=0T[r(st,at)−βlog⁡π(at∣st)πref(at∣st)]\pi^* = \arg\max_\pi \mathbb{E}_{(s_0,...,s_T) \sim \rho_\pi} \sum_{t=0}^T \left[ r(s_t, a_t) - \beta \log \frac{\pi(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)} \right]

The paper shows (Appendix B) that this can be rewritten as a maximum-entropy RL objective with a modified reward:

JRLHF(π)=Eτ∼π[∑t=0T−1(rtask(st,at)+βlog⁡πref(at∣st)+βH(π(⋅∣st)))]J_{\text{RLHF}}(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t=0}^{T-1} \left( r_{\text{task}}(s_t, a_t) + \beta \log \pi_{\text{ref}}(a_t | s_t) + \beta \mathcal{H}(\pi(\cdot | s_t)) \right) \right]

This is standard maximum-entropy RL with reward rtask(st,at)+βlog⁡πref(at∣st)r_{\text{task}}(s_t, a_t) + \beta \log \pi_{\text{ref}}(a_t | s_t) and entropy bonus β\beta. From well-established results in maximum-entropy RL (Haarnoja et al., 2017; Nachum et al., 2017), the optimal policy and its value function satisfy the soft Bellman Equation (Equation 3):

V∗(st)−V∗(st+1)=r(st,at)−βlog⁡π∗(at∣st)πref(at∣st)V^*(s_t) - V^*(s_{t+1}) = r(s_t, a_t) - \beta \log \frac{\pi^*(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)}

The key theorem (Theorem 2, adapted from Nachum et al., 2017) is that satisfying this equation for all states and actions is both necessary and sufficient for the policy and value function to be optimal:

"If a policy π(a | s) and state value function V(s) satisfy the consistency property for all states s and actions a (where s' = f(s, a)), then π = π* and V = V*" (Section 3.2)

This provides a constructive training objective: simultaneously learn a policy πθ and a value function Vφ by minimizing the inconsistency in the soft Bellman Equation across all states in the offline dataset. This is precisely what OREO does (Equations 8 and 9), making it a direct application of Path Consistency Learning (Nachum et al., 2017) to LLM reasoning.

The connection to DPO (Section 3.3) is then illuminating: DPO can be derived from this same framework by making two additional assumptions—(1) telescoping the per-step consistency into a trajectory-level equation, and (2) introducing the Bradley-Terry preference model to compare trajectories. By showing that these are the exact two steps that cause DPO's limitations for reasoning tasks (the telescoping destroys credit assignment; the BT model forces paired data), the paper provides a principled explanation for why DPO underperforms on reasoning—and why OREO's approach of enforcing the full per-step consistency without the BT model should work better.

3. Technical Approach

3.1 Reader Orientation

OREO is an offline reinforcement learning algorithm that jointly trains a language model policy and a separate value function (a "critic" network) from a static dataset of reasoning trajectories, without requiring any live interaction with an environment or any pairwise preference labels. The problem it solves is: given a dataset of multi-step reasoning attempts—each labeled only with a terminal success/failure signal—how can we improve the policy model beyond what supervised fine-tuning on successes alone can achieve, while also producing a value function that can guide test-time search? The solution has the shape of a soft Bellman consistency objective: train the policy to make each step's probability ratio (policy vs. reference) match the drop in estimated future value across that step, and train the value function to predict the actual future returns observed in the data.

3.2 Big-Picture Architecture (Diagram in Words)

The OREO system has four interconnected components:

  1. Policy model ($\pi_\theta$) — the language model being trained to generate better reasoning. It starts as a supervised fine-tuned (SFT) checkpoint and is updated to maximize rewards while staying close to a reference policy.

  2. Value model ($V_\phi$) — a separate neural network (typically the same architecture as the policy, with a regression head) that estimates the expected future KL-regularized reward from any partial reasoning state $s_t$. It is trained via MSE regression against actual outcomes observed in the dataset.

  3. Reference policy ($\pi_{\text{ref}}$) — a frozen copy of the initial SFT model that serves as an anchor to prevent the policy from drifting too far during training.

  4. Offline dataset ($\mathcal{D}$) — a collection of reasoning trajectories, each consisting of a sequence of token-level states and actions $(s_0, a_0, \ldots, s_T)$ with a sparse terminal reward $R_T \in \{0, 1\}$ indicating success or failure. Crucially, no pairwise preference structure is required; trajectories are independent.

Information flows as follows: a trajectory is sampled from the offline dataset → the value model computes estimates $V_\phi(s_t)$ for each state → the policy model computes log-ratios $\log \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{ref}}(a_t|s_t)}$ for each action → the soft Bellman consistency error is computed for each time step (comparing the value drop $V_\phi(s_t) - V_\phi(s_{t+1})$ against the policy log-ratio, adjusted for reward) → both the policy parameters $\theta$ and value parameters $\phi$ are updated via gradient descent to minimize this inconsistency, while the reference policy remains frozen.

At test time, the trained policy $\pi_\theta$ can be used directly for autoregressive generation, or the trained value function $V_\phi$ can be used to guide step-level beam search by scoring partial trajectories and retaining the most promising beams.

3.3 Roadmap for the Deep Dive

  • First, I will explain the theoretical foundation—the soft Bellman Equation (Equation 3) and why it provides a constructive training objective for jointly learning policy and value functions. This is the mathematical core that distinguishes OREO from DPO and rejection sampling.

  • Second, I will derive OREO's value function training objective (Equation 8), explaining what each term represents, how the soft labels are constructed from observed data, and why MSE is the appropriate loss for this regression task.

  • Third, I will derive OREO's policy training objective (Equation 9), explaining the stop-gradient mechanism, the role of the regularization term, and how the policy update connects to the value function's estimates.

  • Fourth, I will discuss the two loss variants (step-level OREO and response-level OREO), explaining how they differ from the standard token-level objective and what trade-offs they represent.

  • Fifth, I will explain the iterative extension of OREO, describing how new data is collected in each iteration and how the method leverages the updated policy for further improvement.

  • Sixth, I will describe the test-time search procedure that uses the learned value function to guide step-level beam search, connecting it to the PRM literature and explaining why the explicit value function outperforms implicit policy-derived values.

  • Finally, I will summarize the practical hyperparameter configurations and design choices that make the method work.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper that proposes a new offline RL training algorithm grounded in maximum-entropy reinforcement learning. The core idea is that by jointly training a policy and an explicit value function to satisfy the soft Bellman Equation at every time step, we can overcome the credit assignment and data requirements limitations of DPO while providing a principled framework for both policy improvement and test-time search.

The Soft Bellman Equation as a Training Objective

The theoretical foundation of OREO is the soft Bellman Equation, which characterizes the relationship between the optimal KL-regularized policy $\pi^*$ and its state value function $V^*$. The paper presents two key theorems adapted from the maximum-entropy RL literature (Nachum et al., 2017; Haarnoja et al., 2017).

Theorem 1 (Equation 3 in the paper) states that the optimal policy and its value function satisfy the following per-step consistency:

V∗(st)−V∗(st+1)=r(st,at)−βlog⁡π∗(at∣st)πref(at∣st)V^*(s_t) - V^*(s_{t+1}) = r(s_t, a_t) - \beta \log \frac{\pi^*(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)}

where $s_t$ is the current state (the prompt plus all tokens generated so far), $a_t$ is the action (the next token generated), $s_{t+1} = f(s_t, a_t)$ is the deterministic next state (the state with the token concatenated), $r(s_t, a_t)$ is the task reward (non-zero only at the terminal step for reasoning tasks), $\beta > 0$ controls the strength of the KL penalty that keeps the learned policy close to the reference policy $\pi_{\text{ref}}$, and $V^*(s_t)$ is the expected future KL-regularized cumulative reward from state $s_t$ under the optimal policy.

What this equation computes: The left side $V^*(s_t) - V^*(s_{t+1})$ is the drop in expected future value when moving from state $s_t$ to state $s_{t+1}$. If the action $a_t$ is good (moves the reasoning closer to a correct conclusion), the value should stay roughly the same or drop only slightly; if the action is bad (introduces an error), the value should drop significantly. The right side represents the immediate reward plus a penalty for deviation from the reference policy, scaled by $\beta$. The equation says: for an optimal agent, the value drop across any step exactly equals the KL-penalized reward of that step. If the agent takes an action that the reference policy would have been very unlikely to take—i.e., $\pi^*(a_t|s_t) \gg \pi_{\text{ref}}(a_t|s_t)$—then $\log \frac{\pi^*(a_t|s_t)}{\pi_{\text{ref}}(a_t|s_t)}$ is large and positive, making the right side smaller, which means the value drop must also be smaller. This formalizes the intuition that deviating from the reference is only justified if the action genuinely improves the expected outcome (keeps the value from dropping).

Why this form: The soft Bellman Equation extends the standard Bellman Equation ($V^*(s_t) - V^*(s_{t+1}) = r(s_t, a_t)$ when $\gamma = 1$) by incorporating the KL penalty term. This is essential because in LLM fine-tuning, unconstrained reward maximization leads to reward hacking and catastrophic forgetting—the policy would learn to produce gibberish that happens to score well under a learned reward model or exploit spurious patterns in the verification signal. The KL term ensures that the policy only diverges from the reference when there is clear evidence (via the value function) that the divergence leads to higher actual returns. The $\beta$ hyperparameter controls this trade-off: small $\beta$ means the policy is more willing to deviate from the reference to pursue reward; large $\beta$ means the policy stays very close to the reference regardless of reward. The derivation in Appendix B shows that this equation follows directly from rewriting the RLHF objective (Equation 1) as a maximum-entropy RL problem with modified reward $r_{\text{task}}(s_t, a_t) + \beta \log \pi_{\text{ref}}(a_t | s_t)$.

Theorem 2 (stated in Section 3.2) provides the key insight that makes the soft Bellman Equation a valid training objective:

"If a policy $\pi(a | s)$ and state value function $V(s)$ satisfy the consistency property (3) for all states $s$ and actions $a$ (where $s' = f(s, a)$), then $\pi = \pi^*$ and $V = V^*$."

The proof is a straightforward extension of the Path Consistency Learning (PCL) result from Nachum et al. (2017), adjusted for the KL-regularized setting. This theorem is the constructive heart of OREO: it establishes that enforcing the soft Bellman Equation over all state-action pairs in the dataset is both necessary and sufficient for converging to the optimal policy and value function. This means we can train $\pi_\theta$ and $V_\phi$ by simply minimizing the inconsistency in Equation 3, without needing to solve the full RL problem (which would require on-policy rollouts, importance sampling, or other expensive techniques).

For the sparse-reward setting that OREO targets (reward is non-zero only at the terminal step $T$), the paper telescopes Equation 3 across time to obtain a more practical form (Equation 7):

V∗(st)=Rt−β∑i≥tlog⁡π∗(ai∣si)πref(ai∣si)V^*(s_t) = R_t - \beta \sum_{i \geq t} \log \frac{\pi^*(a_i | s_i)}{\pi_{\text{ref}}(a_i | s_i)}

where $R_t = \sum_{i \geq t} r(s_i, a_i)$ is the cumulative reward from step $t$ onward. Since $r(s_t, a_t) = 0$ for all non-terminal steps and $r(s_{T-1}, a_{T-1}) \in \{0, 1\}$ depending on whether the final answer is correct, $R_t$ is simply 1 for all steps of a successful trajectory (except possibly the first step if the trajectory fails) and 0 for all steps of a failed trajectory. This telescoped form says: the value at any state $s_t$ equals the eventual outcome minus the cumulative KL penalty accumulated from that point forward. If the policy deviated heavily from the reference on the path from $s_t$ to the end, those deviations are subtracted from the final reward when computing the value of $s_t$.

Key comparison to DPO (Section 3.3): DPO uses only the special case of Equation 7 with $t = 0$, telescoping across the entire trajectory:

∑t=0T−1r(st,at)=V∗(s0)+∑t=0T−1βlog⁡π∗(at∣st)πref(at∣st)\sum_{t=0}^{T-1} r(s_t, a_t) = V^*(s_0) + \sum_{t=0}^{T-1} \beta \log \frac{\pi^*(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)}

DPO then introduces the Bradley-Terry preference model to compare two complete trajectories. This trajectory-level telescoping is what destroys DPO's ability to do per-step credit assignment: the model sees only that the total log-ratio across an entire winning trajectory is larger than the total log-ratio across a losing trajectory, without any per-step signal about which specific actions caused the difference. OREO's core innovation is to enforce Equation 7 at every time step, not just $t = 0$, thereby decomposing the trajectory-level signal into per-step value estimates.

Value Function Training Objective (Equation 8)

The value model $V_\phi$ is a neural network—in practice, another instance of the same transformer architecture as the policy, but with a regression head that outputs a scalar instead of a token distribution. It is trained to minimize the mean squared error between its predictions and the "target values" implied by Equation 7:

LV(ϕ)=1T∑t=0T−1(Vϕ(st)−[Rt−β∑i≥tlog⁡πθ(ai∣si)πref(ai∣si)])2\mathcal{L}_V(\phi) = \frac{1}{T} \sum_{t=0}^{T-1} \left( V_\phi(s_t) - \left[ R_t - \beta \sum_{i \geq t} \log \frac{\pi_\theta(a_i | s_i)}{\pi_{\text{ref}}(a_i | s_i)} \right] \right)^2

where $T$ is the length of the trajectory, $V_\phi(s_t)$ is the value model's prediction for state $s_t$, $R_t \in \{0, 1\}$ is the terminal reward (1 for all steps of a successful trajectory, 0 for all steps of a failed one), $\beta = 0.03$ is the KL penalty coefficient, $\pi_\theta(a_i | s_i)$ is the current policy's probability of the action that was actually taken in the dataset, and $\pi_{\text{ref}}(a_i | s_i)$ is the frozen reference policy's probability of that same action.

What it computes: For each time step $t$ in each trajectory, the target value is constructed as follows: start with the terminal outcome $R_t$ (1 for success, 0 for failure), then subtract $\beta$ times the cumulative log-ratio of policy-to-reference probabilities for all actions from step $t$ to the end. This cumulative subtraction adjusts the outcome downward when the policy deviated from the reference. The value model's prediction $V_\phi(s_t)$ is compared to this target via squared error, and the average over all time steps is the loss.

Why this form: MSE is the appropriate loss because the target values are real-valued and the value function represents an expected future return, which is a conditional expectation. The squared error is a proper scoring rule for the mean of a distribution, meaning that minimizing it encourages $V_\phi$ to learn the conditional expectation of the target. This is identical in principle to how value functions are trained in deep RL (e.g., DQN and its successors). The alternative of using a cross-entropy or classification loss would be inappropriate because the targets are continuous (due to the KL penalty adjustment) and represent magnitudes, not categories.

The arithmetic-scale detail: The log-ratio $\log \frac{\pi_\theta(a_i|s_i)}{\pi_{\text{ref}}(a_i|s_i)}$ can be positive (when the policy assigns higher probability than the reference) or negative (when the policy assigns lower probability). A positive log-ratio means the policy "used more probability mass" on this action than the reference would have, which incurs a KL penalty; this penalty is subtracted from the reward when computing the target value. A negative log-ratio means the policy was more conservative than the reference, which actually provides a KL "bonus" (a negative penalty) that is effectively added to the reward. This mechanism incentivizes the policy to concentrate probability mass on actions that lead to success and away from actions that lead to failure, because doing so makes the value targets $R_t - \beta\sum_{i\geq t} \log \frac{\pi_\theta}{\pi_{\text{ref}}}$ higher for successful trajectories (the policy is rewarded for correctly identifying good actions) and lower for failed trajectories (the policy is penalized for incorrectly identifying bad actions as likely).

Connection to process reward models (PRMs): The target in Equation 8 is conceptually identical to the Monte Carlo rollout labels used to train PRMs (Wang et al., 2024a; Luo et al., 2024). In PRM training, the label for a step is the empirical success rate of completions sampled from that step onward. OREO's target replaces those sampled rollouts with the actual observed outcome $R_t$ from the dataset trajectory, adjusted by the policy's agreement with the reference. This avoids the computational expense of Monte Carlo sampling at training time (though at the cost of relying on the dataset's coverage of state space). The paper acknowledges this connection explicitly (Section 4.4):

"previous PRM methods train their models using Monte Carlo rollouts, which are essentially similar to the objective used for training the value function in our approach"

Practical training detail: For the 7B model experiments, the value model is trained using LoRA (Low-Rank Adaptation) to save computation. The LoRA rank and alpha are both set to 64, and the learning rate for the critic is set to $10^{-4}$, which is higher than the policy learning rate to ensure the value function adapts quickly enough to provide useful signals for policy updates.

The importance of joint training: A crucial design choice is that the value function and policy are trained simultaneously rather than sequentially. If the value function were trained first and then frozen, it would be trained on data from the reference policy but used to guide a policy that is diverging from the reference, creating a distribution shift problem. If the policy were trained first and the value function trained afterward on the policy's outputs, the value function would have no influence on the policy's training. Joint training means the value function adapts to the evolving policy's distribution while simultaneously providing the signal that shapes that evolution—a co-adaptation that is standard in actor-critic RL methods.

Policy Training Objective (Equation 9)

The policy $\pi_\theta$ is trained to minimize the inconsistency in the soft Bellman Equation, using the value model's estimates to ground the per-step credit assignment:

Lπ(θ)=1T∑t=0T−1(Vϕ(st)−Rt+βlog⁡πθ(at∣st)πref(at∣st)+sg[β∑i>tlog⁡πθ(ai∣si)πref(ai∣si)])2+αLreg\mathcal{L}_\pi(\theta) = \frac{1}{T} \sum_{t=0}^{T-1} \left( V_\phi(s_t) - R_t + \beta \log \frac{\pi_\theta(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)} + \text{sg}\left[ \beta \sum_{i > t} \log \frac{\pi_\theta(a_i | s_i)}{\pi_{\text{ref}}(a_i | s_i)} \right] \right)^2 + \alpha \mathcal{L}_{\text{reg}}

where $\text{sg}[\cdot]$ denotes the stop-gradient operator (the expression inside is treated as a constant during backpropagation, not contributing to the gradient computation), $\alpha = 0.01$ is the weight of the regularization term, and $\mathcal{L}_{\text{reg}}$ is a separate KL regularization penalty:

Lreg=1T∑t=0T−1KL[πθ(⋅∣st)∥πref(⋅∣st)]\mathcal{L}_{\text{reg}} = \frac{1}{T} \sum_{t=0}^{T-1} \text{KL}[\pi_\theta(\cdot | s_t) \parallel \pi_{\text{ref}}(\cdot | s_t)]

where $\text{KL}[\pi_\theta(\cdot | s_t) \parallel \pi_{\text{ref}}(\cdot | s_t)] = \sum_a \pi_\theta(a | s_t) \log \frac{\pi_\theta(a | s_t)}{\pi_{\text{ref}}(a | s_t)}$ is the full Kullback-Leibler divergence between the policy's output distribution and the reference policy's output distribution at state $s_t$.

What it computes: For each time step $t$, the policy loss compares two quantities:

  1. The value model's estimate of expected future return from the current state: $V_\phi(s_t)$
  2. The "realized" value of the path actually taken: the terminal reward $R_t$ minus $\beta$ times the policy's log-ratio for the current action $a_t$, minus $\beta$ times the cumulative log-ratio for all future actions $a_{t+1}, \ldots, a_{T-1}$ (but the future actions' contribution is stopped from contributing gradients).

The squared difference between these two quantities is the per-step inconsistency. The average over all time steps, plus the additional KL regularization term scaled by $\alpha$, is the total policy loss.

Why the stop-gradient on future actions: Without the stop-gradient, the policy's log-ratios for future actions would appear in the target for the current step's update, creating a complex interdependence: changing $\pi_\theta(a_{t+1}|s_{t+1})$ would affect the gradient at step $t$, changing $\pi_\theta(a_t|s_t)$ would affect the gradient at step $t-1$, and so on. This entanglement can cause unstable training and make the effective learning rate vary across steps. The stop-gradient operator breaks this entanglement, making each step's gradient depend only on $\log \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{ref}}(a_t|s_t)}$ directly, with the future contributions treated as a fixed offset. The paper states this rationale explicitly:

"$\text{sg}[\cdot]$ denotes the stop gradient operator, which makes each step have the same scale in the gradient" (Section 4.1)

Rearranging to see the credit assignment clearly: The expression inside the square can be rearranged as:

(Vϕ(st)−Vϕ(st+1)⏟value drop−(r(st,at)−βlog⁡πθ(at∣st)πref(at∣st))⏟KL-penalized step reward)2\left( \underbrace{V_\phi(s_t) - V_\phi(s_{t+1})}_{\text{value drop}} - \underbrace{\left( r(s_t, a_t) - \beta \log \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{ref}}(a_t|s_t)} \right)}_{\text{KL-penalized step reward}} \right)^2

where we substitute $V_\phi(s_{t+1}) = R_t - \beta \sum_{i > t} \log \frac{\pi_\theta(a_i|s_i)}{\pi_{\text{ref}}(a_i|s_i)}$ (the target for step $t+1$, which is exactly what the value function should predict, ignoring the stop-gradient). In this form, the objective directly enforces Equation 3: the drop in value across the step must equal the KL-penalized immediate reward.

The role of $\mathcal{L}_{\text{reg}}$: The additional KL regularization term serves a different purpose from the KL penalty already embedded in the Bellman consistency objective. The Bellman equation enforces consistency on the actions that were actually taken in the dataset. It does not constrain the policy's behavior on other actions that could have been taken but weren't. The extra $\mathcal{L}_{\text{reg}}$ term penalizes the policy's entire output distribution diverging from the reference on all possible actions, providing a blanket regularization that prevents the policy from assigning extreme probabilities to actions not observed in the dataset. This is particularly important in offline RL, where the dataset provides limited coverage of the action space and unregularized policies can extrapolate wildly to unobserved actions (a well-known failure mode in offline RL; Kumar et al., 2022). The paper states this term "helps stabilize training" (Section 4.1), which is consistent with standard practice in offline RL.

Why squared error for the policy too: Using squared error for the policy loss—rather than a policy gradient or likelihood-based objective—is unusual from a traditional RL perspective but follows directly from the PCL framework (Nachum et al., 2017). The justification is that the Bellman Equation is an equality constraint that the optimal policy must satisfy. Minimizing the squared residual of this equality is a natural way to find parameters that approximately satisfy it, similar to how temporal difference learning minimizes the squared TD error. This is fundamentally different from policy gradient methods, which optimize the expected return directly and are typically on-policy.

The $\beta$ and $\alpha$ hyperparameters: The paper sets $\beta = 0.03$ and $\alpha = 0.01$ uniformly across all main experiments. The $\beta$ value of 0.03 is relatively small—it means the policy is allowed to deviate substantially from the reference when there is evidence that doing so improves outcomes. The $\alpha$ value of 0.01 means the additional blanket KL regularization is weighted at one-third the strength of the task-specific Bellman consistency objective, providing a modest stabilization force without dominating training.

Telescoping for Sparse Rewards and the Stop-Gradient Mechanics

A subtle but important technical detail is how the policy objective handles the sparse-reward setting. The paper focuses on the case where reward is non-zero only at the terminal step: $r(s_t, a_t) = 0$ for all $t < T-1$, and $r(s_{T-1}, a_{T-1}) \in \{0, 1\}$. For intermediate steps, Equation 3 becomes:

V∗(st)−V∗(st+1)=−βlog⁡π∗(at∣st)πref(at∣st)V^*(s_t) - V^*(s_{t+1}) = -\beta \log \frac{\pi^*(a_t | s_t)}{\pi_{\text{ref}}(a_t | s_t)}

which says: if the action $a_t$ is more probable under the optimal policy than the reference, then the value must drop across that step by exactly $\beta \log \frac{\pi^*}{\pi_{\text{ref}}}$. This is because the KL penalty is the only "cost" being paid at non-terminal steps. At the terminal step, the equation is:

V∗(sT−1)−V∗(sT)=r(sT−1,aT−1)−βlog⁡π∗(aT−1∣sT−1)πref(aT−1∣sT−1)V^*(s_{T-1}) - V^*(s_T) = r(s_{T-1}, a_{T-1}) - \beta \log \frac{\pi^*(a_{T-1} | s_{T-1})}{\pi_{\text{ref}}(a_{T-1} | s_{T-1})}

If the episode is successful ($r = 1$), this means the reward offsets the KL penalty; if unsuccessful ($r = 0$), the value drop equals only the negative KL penalty (which may be positive or negative depending on the log-ratio). This per-step structure is what enables credit assignment: the value model learns to predict lower values for states that precede errors, and the policy learns to reduce the probability of actions that cause large value drops.

When implementing the loss in Equation 9, the telescoping is handled by the term $R_t - \beta \sum_{i \geq t} \log \frac{\pi_\theta(a_i | s_i)}{\pi_{\text{ref}}(a_i | s_i)}$. For a successful trajectory ($R_t = 1$ for all $t$), this is 1 minus the cumulative future log-ratio. For a failed trajectory ($R_t = 0$ for all $t$), this is 0 minus the cumulative future log-ratio. The policy update at step $t$ compares $V_\phi(s_t)$—what the value model thinks should happen from here—against this realized outcome. If the realized outcome is worse than $V_\phi(s_t)$ predicted, the term is negative, and the squared error penalizes the policy—but only for the log-ratio of $a_t$, since future log-ratios are stopped.

Loss Variants: Step-Level and Response-Level OREO

The paper introduces two architectural variants of the OREO objective to isolate the effects of granularity and to connect more clearly to DPO. Both are presented in Section 4.2.

Step-level OREO: Here, an "action" is an entire reasoning step—a multi-token segment like "In May, Natalia sold 48 / 2 = 24 clips." rather than individual tokens. The probability of taking this step-level action is the product of token-level probabilities:

π(a∣s)=∏i=1kp(ti∣s,t1,…,ti−1)\pi(a | s) = \prod_{i=1}^{k} p(t_i | s, t_1, \ldots, t_{i-1})

where $a = (t_1, t_2, \ldots, t_k)$ is the sequence of tokens comprising the reasoning step, and $p$ is the language model. The step-level objective then uses the same Equations 8 and 9 but with these step-level actions and state transitions defined at step boundaries rather than token boundaries. Segmentation is done using line breaks (for GSM8K, where steps are already separated by line breaks) and both line breaks and periods (for MATH, where the original solutions use a mix of formatting conventions; Appendix A.2).

Why step-level might help: Reasoning errors typically occur at the granularity of reasoning steps, not individual tokens. A step like "48 / 2 = 24" contains five tokens but only one semantic unit—the entire computation is either correct or incorrect. Grouping tokens into steps means the value function estimates the expected return at natural decision boundaries, and the policy receives credit assignment at the same granularity at which errors manifest. This could reduce noise compared to token-level assignment, where the model tries to disentangle the contribution of each individual token within a coherent computation.

Why step-level might hurt: The value function's estimates at step boundaries may be less accurate than at token boundaries because there are fewer training samples per trajectory (one per step instead of one per token), and the signal is more compressed. Additionally, the segmentation itself introduces a heuristic element—the choice of separators (line breaks, periods) is arbitrary and may not perfectly align with semantic reasoning steps, especially for models that produce unconventional formatting.

Response-level OREO (Equation 10): This variant collapses the per-step consistency to only the initial state $s_0$, making the policy objective:

Lπresp(θ)=(Vϕ(s0)−R0+β∑i≥0log⁡πθ(ai∣si)πref(ai∣si))2+αLreg\mathcal{L}^{\text{resp}}_\pi(\theta) = \left( V_\phi(s_0) - R_0 + \beta \sum_{i \geq 0} \log \frac{\pi_\theta(a_i | s_i)}{\pi_{\text{ref}}(a_i | s_i)} \right)^2 + \alpha \mathcal{L}_{\text{reg}}

What this variant does: It compares the value function's estimate at the very beginning of the trajectory ($V_\phi(s_0)$) against the entire trajectory's realized outcome ($R_0 - \beta \sum_{i \geq 0} \log \frac{\pi_\theta(a_i|s_i)}{\pi_{\text{ref}}(a_i|s_i)}$), with no per-step decomposition. This is structurally similar to DPO (which also compares complete trajectories) but with two key differences: (1) it uses a squared-error loss rather than Bradley-Terry pairwise comparison, and (2) it still trains an explicit value function $V_\phi$.

Why this variant matters for understanding DPO's limitations: The empirical results (Table 5, Figure 5) show that response-level OREO performs worse than token-level OREO (49.4 vs. 52.5 on MATH; 75.8 vs. 77.3 on GSM8K for the 1.5B model). This directly demonstrates that the per-step granularity—not just the squared error loss or the explicit value function—is a crucial contributor to OREO's gains. DPO is essentially response-level optimization with a preference loss; OREO's response-level variant isolates the effect of switching from preference loss to Bellman consistency while keeping the trajectory-level granularity. The performance gap between response-level and token-level OREO quantifies the value of per-step credit assignment.

The loss scale issue: The paper notes that different OREO variants produce loss values at different scales, which requires adjusting the regularization weight $\alpha$. For step-level OREO, $\alpha = 0.1$; for response-level OREO, $\alpha = 0.3$; for standard token-level OREO, $\alpha = 0.01$. These differences reflect that the response-level objective sums over the entire trajectory before squaring, producing larger loss values that need stronger regularization to keep the policy from diverging rapidly, while the token-level objective averages many small per-step terms and thus needs lighter regularization.

Iterative OREO

Section 4.3 describes how OREO can be extended to an iterative framework, following the pattern established by prior work on iterative preference optimization (Pang et al., 2024; Song et al., 2024; Xiong et al., 2024). The procedure at each iteration $k$ is:

  1. Data collection: Use the current policy model $\pi_{\theta}^{(k)}$ (trained in the previous iteration) to generate new reasoning trajectories for each training problem. The paper uses 10 samples per query for 1.5B models (Appendix A.1). Outcomes are determined by checking whether the final answer matches the ground truth (math) or whether the task was successfully completed (ALFWorld).

  2. Dataset construction: The new trajectories are labeled with terminal rewards (1 for success, 0 for failure) and added to the training dataset. The paper does not specify whether old data from previous iterations is retained, but the standard practice in iterative RL for LLMs is to either replace the dataset entirely with new on-policy data or to maintain a replay buffer that mixes old and new data.

  3. Retraining: OREO is run again from the previous iteration's checkpoint, using the expanded dataset. The reference policy $\pi_{\text{ref}}$ is typically kept as the original SFT model (not updated to the previous iteration's policy), ensuring the method doesn't drift further from the initial model than $\beta$ allows.

Why iterative training helps (Figure 2): The first iteration of OREO learns from the SFT model's outputs, which have a particular distribution of errors and successes. After training, the policy $\pi_{\theta}^{(1)}$ makes different errors than the SFT model did—it has fixed some failure modes but potentially introduced new ones (a phenomenon known as "distribution shift" in RL). The second iteration collects data from $\pi_{\theta}^{(1)}$, exposing OREO to these new failure patterns and allowing the value function to learn to recognize them and the policy to learn to avoid them. This process can continue as long as each iteration's data provides novel learning signal.

The empirical results in Figure 2 validate this: OREO improves steadily across three iterations on both GSM8K and MATH, while rejection sampling shows diminishing returns and begins to saturate by the third iteration. The key difference is that rejection sampling only trains on successes, so each new iteration can only add more successful trajectories (which become increasingly similar to existing successes as the policy improves, providing less novel information). OREO trains on both successes and failures, so each new iteration's failures—even if fewer in number—provide valuable information about the specific ways the improved policy still goes wrong.

Test-Time Search with the Learned Value Function

Section 4.4 describes how the value function $V_\phi$ can be used to guide step-level beam search at inference time, providing additional performance gains without any further training.

The beam search procedure for math reasoning tasks works as follows:

  1. Initialization: The search maintains a set of $B$ candidate partial trajectories (beams). Initially, there is one beam containing only the prompt.

  2. Expansion: For each of the $B$ current beams, the policy model $\pi_\theta$ generates $B$ possible next reasoning steps (a step is defined as all tokens up to the next line break or period, using the same segmentation as step-level OREO). This produces $B \times B = B^2$ candidate extended partial trajectories.

  3. Scoring: The value function $V_\phi$ evaluates each of the $B^2$ candidates by computing $V_\phi(s_{\text{end}})$, where $s_{\text{end}}$ is the state after appending the new reasoning step. This is a single scalar per candidate representing the expected future return from that point.

  4. Pruning: The $B^2$ candidates are sorted by their value scores, and only the top $B$ are retained for the next expansion round.

  5. Termination: The process repeats until each beam generates a complete solution (the model produces an end-of-sequence token or a final answer marker). The final answer from the beam with the highest value score is selected as the output.

Why this works (Section 5.4, Figure 3): The value function has been trained to identify states that are likely to lead to successful outcomes. During beam search, it acts as a heuristic that guides the search away from unpromising partial solutions and toward more promising ones, effectively pruning the search space. As the beam width $B$ increases, the search explores more alternatives and is more likely to find a path to the correct answer. The results confirm this: with $B = 7$, the value-guided beam search provides an 11.4% relative improvement over greedy decoding on GSM8K and a 17.9% relative improvement on a 500-question subset of MATH.

Connection to process reward models (PRMs) and test-time compute scaling: This is essentially the same procedure used in prior work on inference-time scaling with PRMs (Snell et al., 2024; Lightman et al., 2023; Hao et al., 2024a), but with the value function $V_\phi$ serving as the step-level scorer instead of a separately trained PRM. The paper emphasizes that OREO provides this value function "for free" as a byproduct of training, without the need for the "substantial data engineering and heuristic design efforts" (Section 5.4) typically required for PRM training. This is a significant practical advantage: a single training run produces both an improved policy and a step-level verifier that can guide search.

Embodied agent tasks (ALFWorld): For interactive environments where the transition dynamics depend on an external simulator (the agent performs an action, the environment state changes in ways not known in advance), beam search over future states is not possible because the environment's response to hypothetical actions is unknown. Instead, the paper uses best-of-K action selection at each step: the policy generates $K$ candidate actions, the value function scores each ($V_\phi$ of the current state concatenated with the candidate action), and the action with the highest value score is executed. This is equivalent to a one-step lookahead using the value function as the evaluation heuristic. The results (Figure 4) show that best-of-5 action selection with the value function improves success rates substantially in ALFWorld unseen environments, demonstrating the value function's usefulness even in settings where full tree search is infeasible.

The Implicit vs. Explicit Value Function Distinction

A theoretically important and empirically consequential design choice in OREO is the use of a separately parameterized value model $V_\phi$ rather than deriving value estimates from the policy itself. Section 5.3 provides both theoretical motivation and empirical evidence for this choice.

The implicit value function from DPO theory: Rafailov et al. (2024a) showed that in the DPO framework, the policy model can be viewed as implicitly representing a value function:

Vθ(st)=V(s0)+∑t′=0tβlog⁡πθ(at′∣st′)πref(at′∣st′)V_\theta(s_t) = V(s_0) + \sum_{t'=0}^t \beta \log \frac{\pi_\theta(a_{t'} | s_{t'})}{\pi_{\text{ref}}(a_{t'} | s_{t'})}

This follows directly from the telescoped soft Bellman Equation (Equation 4): if we telescope from step 0 to step $t$ instead of from 0 to $T$, the cumulative log-ratio up to $t$ plus the initial value $V(s_0)$ equals the value at $t$. This means the policy's probabilities implicitly encode a value function—the advantage of an action can be read off as $\beta \log \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{ref}}(a_t|s_t)}$. In an ideal world where the soft Bellman Equation holds perfectly, the implicit value function and the explicit value function would be identical.

The softmax bottleneck problem: However, as the paper argues and empirically demonstrates (Figure 1), the implicit value function is substantially weaker than the explicit one, particularly on challenging reasoning tasks. The paper attributes this to the softmax bottleneck (Yang et al., 2017): the policy's output distribution $\pi_\theta(\cdot | s_t)$ over all possible next tokens is generated from a single hidden state via a linear transformation followed by softmax. The log-ratio $\log \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{ref}}(a_t|s_t)}$ is then essentially a difference of logits, constrained by the representational capacity of a single linear layer operating on a shared hidden representation.

In contrast, the explicit value function $V_\phi(s_{t+1})$ processes the entire next-state text sequence through the full transformer network, producing a scalar output. This allows the value function to develop rich, non-linear representations of the state that are specifically optimized for predicting future returns, without the bottleneck of having to also serve as a token distribution. The paper's case studies (Figure 1) show instances where the explicit value function correctly identifies a reasoning step as erroneous (the value drops significantly, producing a large negative advantage), while the implicit value function's advantage estimate is "much more conservative" and fails to clearly distinguish correct and incorrect steps.

Why this matters for test-time search: The quality of beam search pruning depends entirely on the quality of the scoring heuristic. A weak value function that cannot reliably distinguish promising from unpromising partial solutions will provide little benefit over random exploration, and may even hurt performance by pruning away correct solutions. The explicit value function's superior discriminative ability—demonstrated both quantitatively in the beam search results (Figure 3) and qualitatively in the case studies (Figure 1)—is what makes OREO's test-time search effective. This finding suggests that even when a policy has been trained with DPO or similar preference-based methods, adding a separately trained value function could unlock significant test-time compute scaling benefits.

Practical Hyperparameter Summary

The paper uses consistent hyperparameters across all OREO experiments, with minor variations for model size and task. The key settings (Appendix A.3) are:

  • Policy learning rate: $5 \times 10^{-6}$ for all models and tasks
  • Value (critic) learning rate: $10^{-4}$ for 7B models (using LoRA with rank=64, alpha=64); not separately specified for 1.5B models but presumably similar or identical to the policy learning rate
  • KL penalty coefficient $\beta$: $0.03$ — this is the temperature-like parameter controlling the trade-off between reward maximization and staying close to the reference policy
  • Regularization weight $\alpha$: $0.01$ for token-level OREO; $0.1$ for step-level; $0.3$ for response-level
  • SFT learning rate: $2 \times 10^{-5}$ for 1.5B and 2B models (MiniCPM)
  • Batch size: 128 for all experiments
  • Training epochs: 1 epoch for 1.5B models, 3 epochs for 2B ALFWorld model (using LoRA on the critic), 2 epochs for 7B models (to match total sample count)
  • Dataset sampling: 10 responses per query for 1.5B models on GSM8K and MATH; 16 responses per query for 7B models with at most 4 positive and 4 negative instances selected; 5 rollouts per task for ALFWorld
  • DPO comparison: $\beta = 0.1$, learning rate $5 \times 10^{-7}$, at most 6 preference pairs per query (1.5B) or 10 pairs (7B)
  • KTO comparison: learning rate $5 \times 10^{-8}$ (1.5B) or $10^{-7}$ (7B)

The design choice to keep $\beta$ small (0.03) reflects the reasoning setting: the reference policy already has reasonable performance (around 47.5% on MATH for the 1.5B SFT model), so large deviations from it are not needed—the goal is to refine the policy's probability assignments to favor correct reasoning patterns, not to discover entirely new behaviors. This contrasts with some RLHF applications where $\beta$ is set larger (0.1 or higher) to prevent the policy from producing unnatural or unsafe outputs.

The design choice to train for only 1-3 epochs reflects the offline RL setting: with a fixed dataset, over-training leads to overfitting the value function to the specific trajectories in the data and the policy exploiting inaccuracies in the value estimates (the offline RL analog of reward hacking). Early stopping is a standard regularization technique in offline RL to prevent this degradation.

The design choice to balance positive and negative instances for the 7B experiments (ensuring the number of positive instances does not exceed the number of negative instances, while keeping at least one positive per query if available) addresses the class imbalance problem. In math reasoning, the base model may produce many more failures than successes for hard problems. If the dataset is dominated by failure trajectories, the value function will learn to predict low values for almost all states, which provides no discriminative signal for credit assignment. Balancing ensures roughly equal representation of successes and failures, giving the value function a meaningful contrastive signal.

4. Key Insights and Innovations

Innovation 1: The Soft Bellman Equation as a Unifying Diagnostic That Explains Why DPO Fails on Reasoning

The paper's most distinctive conceptual contribution is not proposing yet another training objective, but rather providing a principled diagnostic framework that explains why the dominant offline alignment method—DPO—systematically underperforms on multi-step reasoning, and why that failure is structural rather than incidental.

What the field assumed before this work. The LLM reasoning community had accumulated a pattern of puzzling empirical results: DPO sometimes worked, sometimes didn't, and often was beaten by simple rejection sampling (Yuan et al., 2024; Chen et al., 2024b; Pang et al., 2024). The working assumption was that DPO's struggles were implementation-specific—wrong hyperparameters, insufficient data, poor pair construction. Papers proposed fixes like better pair selection strategies (Chen et al., 2024a; Song et al., 2024) or iterative DPO (Xiong et al., 2024), implicitly assuming the core framework was sound and just needed better engineering.

The diagnostic move. Section 3.3 performs a surgical derivation: it shows that DPO can be obtained from the soft Bellman Equation by applying exactly two transformations—(1) telescoping the per-step consistency into a trajectory-level sum, and (2) introducing the Bradley-Terry preference model. Each transformation corresponds directly to one of DPO's failures on reasoning:

  • Telescoping destroys credit assignment. By collapsing Equation 3 into Equation 4, DPO erases the per-step structure that identifies which states contributed to success or failure. The model learns that the entire winning trajectory is better than the entire losing trajectory, with no signal about where the critical divergence occurred. This is not a hyperparameter issue or a data quality problem—it is baked into the mathematical form of the objective.

  • The Bradley-Terry model forces paired data. For dialogue alignment, where reward can only be inferred from comparative human judgments, the BT model is a necessary modeling assumption. But for reasoning tasks with ground-truth success/failure labels, it is an artificial constraint that converts N independent trajectories—each carrying an unambiguous binary signal—into O(N²) synthetic preference pairs, discarding the natural structure of the data.

This derivation is what transforms the paper's contribution from "another loss function" into a conceptual diagnostic. It doesn't just say DPO performs poorly on reasoning; it identifies the precise mathematical steps that cause the poor performance and explains why those steps, while reasonable for chat alignment, are structurally misaligned with reasoning tasks.

Significance beyond performance. This diagnostic framework equips future researchers with a decision procedure: when tackling a new domain, ask whether the reward signal is naturally pairwise (suitable for BT-based methods) or naturally per-instance (suitable for Bellman-based methods), and whether credit assignment requires per-step granularity (requiring non-telescoped objectives) or trajectory-level granularity is sufficient. The paper's response-level OREO ablation (Table 5) provides direct empirical evidence for this framework: when you restore per-step credit assignment (token-level OREO), MATH accuracy jumps from 49.4 to 52.5; when you strip it away (response-level), performance degrades to near-DPO levels despite using Bellman consistency rather than preference loss. This isolates the granularity of credit assignment as a causal factor, not just a correlation.

Distinction from prior attempts to fix DPO. Prior work on improving DPO for reasoning (Chen et al., 2024a; Song et al., 2024; Yuan et al., 2024) operated within the DPO paradigm—keeping the BT model and the trajectory-level comparison, but engineering better pairs or adding auxiliary losses. OREO's analysis reveals that these are attempts to patch the symptoms rather than address the root cause. The root cause is that the mathematical structure of DPO is mismatched to the structure of reasoning tasks, and no amount of better pair construction can fix the loss of per-step credit assignment. This is a fundamental insight, not an incremental refinement.

Innovation 2: Explicit Value Functions as a Principled Alternative to Heuristic Process Reward Models

The paper's second major conceptual contribution is reframing the problem of training process reward models (PRMs) for LLM reasoning as a special case of value function learning in maximum-entropy RL, thereby replacing an ad-hoc engineering practice with a theoretically grounded training procedure.

What the field did before. The standard approach to building PRMs was Monte Carlo rollout (Wang et al., 2024a,b; Luo et al., 2024): from each intermediate step in a solution, sample multiple possible completions, check which reach the correct answer, and use the empirical success rate as a soft correctness label for that step. This approach, while often effective, required practitioners to make a series of heuristic decisions with no theoretical guidance: how many rollouts to sample (affecting label quality vs. computational cost), whether to use binary or soft labels, how to handle cases where rollouts diverge significantly from the original trajectory, how to aggregate across multiple rollouts, and whether to normalize scores across different problems. These decisions made PRM training more art than science, and results were sensitive to implementation details that were rarely documented or theoretically motivated.

The reframing. Section 4.4 makes a simple but powerful observation: the value function V_phi(s_t) in maximum-entropy RL—when trained with the soft Bellman Equation—estimates exactly the quantity that a PRM is trying to approximate, namely the expected probability of eventual success from state s_t. The MSE training objective in Equation 8 is mathematically equivalent to the Monte Carlo regression target used in PRM training, but derived from first principles (the Bellman consistency condition) rather than from sampling heuristics. This reframing has several consequences:

  • Eliminates heuristic design choices. The number of rollouts, the label type (soft vs. binary), and the aggregation method are all determined by the structure of the Bellman Equation and the observed data, not by practitioner judgment. This makes the method reproducible and principled in a way that heuristic PRM training is not.

  • Unifies policy improvement and verifier training. In standard PRM-based pipelines, training the policy (via RL or rejection sampling) and training the verifier (via Monte Carlo rollouts) are separate processes with different objectives, different data requirements, and different hyperparameters. OREO shows that both can be derived from the same underlying principle (soft Bellman consistency), trained jointly, and mutually reinforce each other—the value function provides credit assignment for the policy update, and the policy's evolving distribution provides richer training signal for the value function.

  • Provides the value function "for free." The paper emphasizes this point repeatedly: the beam search results (Section 5.4, Figure 3) show a 17.9% relative improvement over greedy decoding on MATH, competitive with dedicated PRM-based search methods, but achieved with no additional training beyond what was already done to improve the policy. This is a significant practical advantage—a single training run yields both a better policy and a search-capable verifier.

Significance beyond performance. This reframing changes how the field should think about PRMs. Rather than viewing them as a separate model class requiring specialized training pipelines, they should be understood as value functions in a maximum-entropy RL formulation. This opens up the entire toolkit of value-based RL—temporal difference learning, eligibility traces, distributional value functions, ensemble methods for uncertainty estimation—for improving PRM training, rather than requiring the community to invent new heuristics from scratch. The paper explicitly notes this connection to Path Consistency Learning (Nachum et al., 2017), but the implications extend further: techniques like Retrace, V-trace, or quantile regression for value functions, all well-studied in the RL literature, become immediately applicable to LLM verifier training.

Distinction from incremental PRM improvements. Prior work on improving PRMs (Luo et al., 2024; Zhang et al., 2024) made incremental changes to the training recipe—different rollout strategies, different label aggregation, entropy regularization. OREO's contribution is different in kind: it provides the theoretical foundation that explains why Monte Carlo rollouts work for PRM training (they approximate the value function's regression target) and what the optimal target should be (the Bellman-consistent expected return, not an arbitrary empirical average). This is a conceptual upgrade of PRMs from heuristic tools to principled components of a coherent RL framework.

Innovation 3: Demonstrating That Learning from Failures Sustains Iterative Improvement While Success-Only Training Saturates

The paper provides a clean empirical demonstration of a phenomenon that the offline RL literature has long predicted but that had not been convincingly shown in the LLM reasoning setting: learning from failed trajectories enables continued improvement across multiple iterations, while training only on successes plateaus rapidly.

What the field assumed. Iterative self-training methods for LLM reasoning—STaR (Zelikman et al., 2022), ReST^EM (Singh et al., 2023), and related approaches—all follow the same template: generate new solutions using the current model, filter for correctness, fine-tune on the successes, repeat. The implicit assumption is that each iteration produces better solutions, which when used as training data produce an even better model, leading to a virtuous cycle of improvement. Prior work generally reported positive results from this cycle, with each iteration adding some performance.

What the paper shows (Figure 2). Over three iterations on both GSM8K and MATH, rejection sampling (the success-only method) shows clear signs of saturation: gains diminish sharply from iteration 1 to iteration 2, and are negligible from iteration 2 to iteration 3. The 1.5B model's MATH accuracy plateaus around 50-51% for rejection sampling after the second iteration. In contrast, OREO—which trains on both successes and failures using the value function—continues to improve across all three iterations, with MATH accuracy rising from approximately 49% to approximately 53% by iteration 3.

The mechanism. The paper's interpretation (Section 5.2) is that success-only training suffers from diminishing information content: as the policy improves, each new iteration's successful trajectories look increasingly similar to existing successes, providing little new learning signal. The model converges to a local optimum defined by the distribution of its own early successes, without the ability to escape through exposure to diverse failures. OREO, by training on failures, extracts information from exactly the trajectories that are most informative about the policy's remaining weaknesses. Each new iteration's failures—even as they become rarer—reveal the specific patterns the model still gets wrong, and the value function uses these to shape the policy away from those patterns.

Significance beyond the specific result. This finding has direct implications for how iterative self-improvement pipelines should be constructed. It suggests that the common practice of discarding failures—while computationally simpler (no need to train a value function or handle negative examples)—imposes a fundamental ceiling on what iterative training can achieve. It also explains a pattern visible in prior work: methods like STaR showed impressive initial gains that tapered off, which was often attributed to data diversity or exploration issues. OREO's results suggest the saturation may instead be inherent to the success-only learning paradigm itself, and that value-function-based methods provide a principled escape.

Caveat on scale of evidence. This finding is demonstrated on 1.5B models over three iterations. Whether the gap between OREO and rejection sampling continues to widen over more iterations, or whether OREO too eventually saturates, is unknown. The paper's computational constraints (acknowledged in Section 7) prevented running more iterations or testing on larger models. The finding is therefore best understood as a strong proof of concept rather than a fully characterized scaling law.

Distinction from prior iterative RL work. Prior work on iterative DPO (Xiong et al., 2024; Pang et al., 2024) showed that preference-based methods can also benefit from multiple iterations, but did not decompose whether the benefit came from on-policy data collection versus from the learning algorithm's ability to use that data. OREO's comparison with rejection sampling—which also uses on-policy data in each iteration but a different learning algorithm—isolates the effect of the learning objective itself. The finding that success-only training saturates while value-function-based training does not demonstrates that the learning algorithm, not just data freshness, determines iterative scalability.

The paper identifies and empirically validates a distinction between explicit value functions (separately parameterized networks trained via Bellman regression) and implicit value functions (derived from policy log-ratios, as in DPO) that has significant practical consequences for test-time compute scaling, independent of training-time performance.

What the field assumed. Rafailov et al. (2024a) elegantly showed that DPO-trained policies implicitly represent a value function: the log-ratio of policy to reference probabilities at each step equals the advantage of that action under the optimal value function. This theoretical result was interpreted by many as evidence that explicit value functions were unnecessary—the policy already contains the value information, and any capability a value function would provide is redundant. The practical implication was that DPO-based training pipelines could skip the complexity of training a separate critic, since the policy itself could serve as both generator and verifier.

What the paper demonstrates (Section 5.3, Figure 1). The implicit and explicit value functions diverge substantially on challenging reasoning problems, and the explicit value function is systematically better at distinguishing correct from incorrect reasoning steps. The case studies in Figure 1 show concrete instances where:

  • The explicit value function drops sharply (from 0.816 to 0.385) upon encountering an error, producing a clear negative advantage of -0.431.
  • The implicit value function derived from the policy's log-ratios produces a much smaller magnitude signal and in the harder MATH example, fails to correctly identify which reasoning step is correct—the implicit advantage actually favors the wrong continuation.

The paper attributes this gap to the softmax bottleneck (Yang et al., 2017): the policy's implicit value estimates are constrained by the fact that they must be computed from the same hidden representation that produces token logits, via a single linear transformation followed by softmax. The explicit value function processes the full state representation through the entire transformer, enabling richer, non-linear representations that are specifically optimized for predicting expected returns rather than token distributions.

Significance for test-time compute scaling. This finding has direct practical implications for the growing literature on inference-time search (Snell et al., 2024; Wu et al., 2024; Brown et al., 2024). The quality of beam search, tree-of-thought, or best-of-N selection depends fundamentally on the discriminator's ability to rank partial solutions. A weak value function—even one derived from a strong policy—will prune away good solutions and retain bad ones, potentially making search worse than greedy decoding. The paper's beam search results (Figure 3) bear this out: the explicit value function provides monotonically improving accuracy as beam width increases (from B=1 to B=7), demonstrating that it is making genuinely informative distinctions between partial trajectories. An implicit value function with the demonstrated weaknesses would likely show flatter or even inverted scaling curves.

This matters because it challenges a simplifying assumption in the test-time compute scaling literature: that improvements in the base policy automatically translate to improvements in the verifier used for search. OREO's results suggest that policy quality and verifier quality are partially decoupled, and that investments in explicit verifier training may yield disproportionate returns for test-time search, even beyond the returns from improving the policy itself.

Distinction from ensemble or distillation approaches. The standard approach to improving verifiers for search is to train a separate PRM (Lightman et al., 2023; Wang et al., 2024a) or to use majority voting across multiple samples. OREO's explicit value function is neither—it is a regression-trained critic that learns from the same data as the policy, with the same architecture, but optimized for a different (value prediction) objective. This is a third category of verifier that is more integrated with the policy training process than a standalone PRM, yet more expressive than an implicit value function. The paper's innovation is identifying this category and demonstrating its empirical advantage, not inventing a new training technique per se.

Caveat. The comparison between explicit and implicit value functions is qualitative (case studies) and based on advantage estimates. The paper does not provide a quantitative metric (like AUROC for step correctness classification) that would enable rigorous comparison. The claim that the explicit value function is "more effective" rests on the beam search results (which show monotonic improvement with beam width) and the qualitative examples, but a direct head-to-head comparison—training beam search with the implicit value function as the scoring heuristic and measuring performance degradation—is not provided. The finding is therefore strongly suggestive but not definitively quantified.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses three datasets spanning two domains. For mathematical reasoning: GSM8K (Cobbe et al., 2021) — 7,473 training problems and 1,319 test problems of grade-school math word problems; and MATH (Hendrycks et al., 2021) — 7,500 training problems and 5,000 test problems of competition-level mathematics, with all problems including step-by-step ground-truth solutions. For embodied agent control: ALFWorld (Shridhar et al., 2020) — interactive TextWorld environments for household tasks, with 3,119 training trajectories annotated with rationales by Song et al. (2024), an evaluation set of 140 tasks in seen environments, and 134 tasks in unseen environments. All datasets provide ground-truth correctness signals (math answers verified via the DeepSeekMath grading script; ALFWorld tasks verified by environment success conditions) rather than requiring human preference judgments.

  • Base model(s). Three model families at two scales are tested. For mathematical reasoning: Qwen2.5-Math-1.5B (Yang et al., 2024) — a 1.5-billion-parameter model specifically designed for mathematical tasks, chosen to demonstrate that OREO can substantially improve a small model's reasoning without requiring large-scale compute; and DeepSeekMath-7B-Instruct (Shao et al., 2024) — a 7-billion-parameter instruction-tuned model that has already been heavily fine-tuned on 776K mathematical reasoning samples, chosen to test whether OREO can improve performance even when starting from a strong SFT baseline. For embodied agent control: MiniCPM-2B-dpo-bf16 (Hu et al., 2024) — a 2-billion-parameter model, chosen to test whether the method generalizes beyond pure text reasoning to interactive agent tasks. The paper states that it selects these models to be "representative of the capabilities of many contemporary LLMs" (Section 5), though all experiments are limited to models under 7B parameters due to computational constraints (Section 7).

  • Metrics. For mathematical reasoning: accuracy — the fraction of test problems for which the model's final answer, extracted using the DeepSeekMath evaluation script, matches the ground-truth answer. For GSM8K, accuracy is reported on the full 1,319-problem test set. For MATH, accuracy is reported on both the full 5,000-problem test set (for main results, Table 1) and a 500-problem subset (for test-time search experiments, Section 5.4), where the 500-problem subset follows the convention established by Lightman et al. (2023). For ALFWorld: success rate — the fraction of tasks for which the agent successfully completes the task within 40 steps, reported separately for seen and unseen environments. All metrics are deterministic given the final answer or task outcome, with no subjective evaluation component.

  • Baselines. Four baseline methods are compared against OREO:

    • SFT (Supervised Fine-Tuning): The base model fine-tuned on the training split of each dataset. For Qwen2.5-Math-1.5B, SFT is performed for 3 epochs on GSM8K and MATH training data with learning rate 2 × 10⁻⁵. For DeepSeekMath-7B-Instruct, the model is already instruction-tuned by Shao et al. (2024), so "SFT" refers directly to the released checkpoint with no additional training. For MiniCPM-2B, SFT is performed for 2 epochs on the ALFWorld annotated trajectories from Song et al. (2024) with learning rate 2 × 10⁻⁵.
    • Rejection Sampling (also referred to as STaR by Zelikman et al., 2022; RAFT by Dong et al., 2023; REST by Gulcehre et al., 2023; ReST^EM by Singh et al., 2023): The base model is fine-tuned for 1 epoch on only the successful trajectories from the offline dataset (those with correct final answers or successful task completions). Learning rate is 5 × 10⁻⁶ for math reasoning and 2 × 10⁻⁵ for embodied agent tasks. This is the most competitive baseline and represents the standard practical approach to improving LLM reasoning from offline data.
    • DPO (Direct Preference Optimization) (Rafailov et al., 2024b): The SFT model is fine-tuned for 1 epoch using pairwise preference data constructed from the offline dataset. For each query, at most 6 preference pairs (1.5B models) or 10 pairs (7B models) are sampled without replacement from successful and failed trajectories. Learning rate is 5 × 10⁻⁷ with β = 0.1. This represents the dominant preference-based offline RL method.
    • KTO (Kahneman-Tversky Optimization) (Ethayarajh et al., 2024): A variant of DPO that uses the Kahneman-Tversky model of human utility and can operate on unpaired data. Trained for 1 epoch (1.5B models) or 2 epochs (7B models) with learning rate 5 × 10⁻⁸ (1.5B) or 10⁻⁷ (7B). This tests whether simply removing the pairwise requirement (while keeping the preference-modeling framework) is sufficient to match OREO's performance.

  • Generation budget / compute accounting. The paper does not use a unified compute budget for fair comparison — each method is trained on the same offline dataset (sizes described in Appendix A.1), and performance is compared at test time with greedy decoding unless otherwise specified. For the test-time search experiments (Section 5.4), compute is measured by beam width B (the number of beams maintained during step-level beam search), with results reported for B ∈ {1, 2, 3, 4, 5, 6, 7}. For ALFWorld best-of-K action selection, compute is measured by K, the number of candidate actions sampled per step, with K ∈ {1, 2, 3, 4, 5}. There is no cross-method comparison at equalized test-time compute (e.g., OREO beam search vs. rejection sampling with best-of-N), which means the test-time search gains are measured relative to OREO's own greedy decoding, not against baselines with equivalent inference budgets.

  • Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, standard deviations, or statistical significance tests for any result. All numbers are point estimates from single evaluation runs on the standard test splits. For the iterative training experiments (Section 5.2), each data point represents the accuracy after a complete training cycle on all available data from that iteration, with no mention of multiple seeds or variance estimation. The beam search results (Section 5.4) similarly report single-run accuracies without error bars. The absence of variance estimates is a limitation for interpreting marginal differences (e.g., the 0.3 percentage point gap between some baseline comparisons in Table 1 could be noise).

  • Dataset construction details. The paper constructs offline datasets by sampling multiple responses per training query from the base SFT models (Appendix A.1): 10 responses per query for GSM8K and MATH with 1.5B models; 16 responses per query for 7B models, with at most 4 positive and 4 negative instances selected per query to maintain class balance (the number of positive instances is capped to not exceed the number of negative instances, ensuring at least one positive instance if any exist). For ALFWorld, 5 rollouts per training task are performed using the SFT model. For DPO, preference pairs are constructed from these labeled trajectories by pairing successes with failures, sampling at most 6 pairs (1.5B) or 10 pairs (7B) per query without replacement. All trajectories receive a sparse terminal reward: 1 for correct final answer or successful task completion, 0 otherwise. No intermediate rewards or process-level labels are used.

Main Quantitative Results

Mathematical Reasoning: OREO Consistently Outperforms All Baselines Across Model Sizes and Datasets

The headline results are presented in Table 1. For the 1.5B Qwen model:

  • OREO achieves 77.3% on GSM8K, compared to 73.5% for SFT (a 5.2% relative improvement), 74.9% for rejection sampling, 74.4% for DPO, and 73.4% for KTO.
  • OREO achieves 52.5% on MATH, compared to 47.5% for SFT (a 10.5% relative improvement), 50.3% for rejection sampling, 49.2% for DPO, and 48.3% for KTO.

For the 7B DeepSeekMath model:

  • OREO achieves 85.9% on GSM8K, compared to 82.9% for SFT (a 3.6% relative improvement), 83.6% for rejection sampling, 82.4% for DPO, and 82.5% for KTO.
  • OREO achieves 49.2% on MATH, compared to 46.8% for SFT (a 5.1% relative improvement), 47.2% for rejection sampling, 47.2% for DPO, and 46.9% for KTO.

Several patterns are notable beyond the raw rankings. First, the relative improvement is larger on the harder dataset (MATH) than on the easier one (GSM8K) for both model sizes, suggesting that OREO's credit assignment mechanism provides greater benefit when problems are more challenging and errors are more likely to occur at specific identifiable steps. Second, the gains over the strong DeepSeekMath-7B checkpoint — which was already heavily fine-tuned on 776K samples — demonstrate that OREO can extract additional signal from offline data even when the base policy is already well-tuned. Third, DPO underperforms rejection sampling in most comparisons (except on DeepSeekMath-7B MATH where they are tied at 47.2%), consistent with prior findings that "the direct usages of DPO are not all successful" for reasoning (Section 2.2). Fourth, KTO — which removes DPO's pairwise requirement — actually performs worse than DPO in most settings (73.4 vs. 74.4 on 1.5B GSM8K; 48.3 vs. 49.2 on 1.5B MATH; 82.5 vs. 82.4 on 7B GSM8K), demonstrating that simply removing the pairwise constraint is not sufficient — the fundamental issue is the lack of per-step credit assignment, not just data format.

Embodied Agent Control: OREO Generalizes to Unseen Environments Where Rejection Sampling Fails

Table 2 presents results on ALFWorld, which tests whether the method works in settings requiring interaction with an environment rather than pure text generation.

  • In seen environments, OREO achieves 80.7% success rate, compared to 62.9% for SFT (a 28.3% relative improvement), 79.3% for rejection sampling, and 64.3% for DPO.
  • In unseen environments, OREO achieves 79.1% success rate, compared to 67.2% for SFT (a 17.7% relative improvement), 68.7% for rejection sampling, and 69.4% for DPO.

The critical finding is the asymmetry between seen and unseen environments. Rejection sampling performs well in seen environments (79.3%, nearly matching OREO's 80.7%), suggesting that for tasks similar to those in the training distribution, simply training on successful trajectories is sufficient. However, in unseen environments, rejection sampling barely improves over SFT (68.7% vs. 67.2%), while OREO maintains a large margin (79.1%). The paper interprets this as evidence that "OREO effectively leverages the failed trajectories, which results in more generalizable capabilities" (Section 5.1). In embodied control, failures often arise from specific missteps (choosing a wrong object, navigating to a wrong location) while the rest of the plan is sound. Rejection sampling discards these trajectories entirely, losing the opportunity to learn which specific actions to avoid. OREO's value function learns to associate these failure-causing states with lower expected returns and the policy learns to reduce the probability of those actions, producing behavior that transfers to novel environments where similar failure modes may arise.

DPO performs slightly better than rejection sampling in unseen environments (69.4% vs. 68.7%) but substantially worse in seen environments (64.3% vs. 79.3%), producing an unusual pattern where the method that was weaker in-distribution becomes marginally stronger out-of-distribution. The paper does not provide an explanation for this inversion, but it may reflect DPO's trajectory-level comparison being less prone to overfitting to specific environmental configurations than rejection sampling's success-only memorization.

Iterative Training: OREO Sustains Improvement While Rejection Sampling Saturates

Figure 2 plots accuracy on GSM8K and MATH over three iterations of data collection and retraining, using the 1.5B Qwen model. Each iteration: the current model generates 10 new responses per training query, outcomes are determined, and the method (OREO or rejection sampling) is retrained on the expanded dataset.

  • GSM8K: OREO improves from approximately 72% (iteration 0, SFT baseline) to approximately 77% (iteration 1), to approximately 79% (iteration 2), to approximately 80% (iteration 3). Rejection sampling improves from 72% to approximately 75% (iteration 1), to approximately 76% (iteration 2), and shows negligible improvement in iteration 3 (remaining at approximately 76-77%).
  • MATH: OREO improves from approximately 47% (iteration 0) to approximately 52% (iteration 1), to approximately 53% (iteration 2), to approximately 53-54% (iteration 3). The iteration 3 gain on MATH is small (approximately 0.5-1 percentage point), suggesting OREO may also be approaching saturation, though the trajectory is still upward. Rejection sampling improves from 47% to approximately 50% (iteration 1), to approximately 51% (iteration 2), and shows no improvement in iteration 3.

The key pattern is that rejection sampling's gains diminish sharply — the majority of improvement occurs in iteration 1, with iteration 2 providing a small boost and iteration 3 providing essentially none. OREO shows more sustained improvement, with iteration 2 providing meaningful gains (approximately +2% on GSM8K, +1% on MATH) beyond what iteration 1 achieved. The paper attributes this to OREO's ability to learn from new failure patterns in each iteration's generated data, while rejection sampling's success-only training receives diminishing informational returns as successful trajectories become increasingly similar across iterations. However, the deceleration in OREO's gains from iteration 2 to 3 (especially on MATH) suggests that even with failure learning, there may be limits to what iterative offline training can achieve without fundamental changes to the exploration strategy or data diversity.

The paper does not report iterative results for DPO or KTO, making it unclear whether the failure-learning mechanism (as opposed to some other property of OREO's objective) is the causal factor in sustained improvement. An iterative DPO comparison would have strengthened the claim that learning from failures — specifically — enables continued gains.

Loss Variants: Per-Step Credit Assignment Is the Critical Component, Not the Loss Function Form

Table 5 (presented as Figure 5 in the paper) compares three variants of the OREO objective using the 1.5B Qwen model:

  • Token-level OREO (standard): 77.3% on GSM8K, 52.5% on MATH.
  • Step-level OREO: 78.1% on GSM8K, 51.2% on MATH.
  • Response-level OREO: 75.8% on GSM8K, 49.4% on MATH.

Three insights emerge. First, the response-level variant — which collapses per-step credit assignment into a single trajectory-level objective — performs substantially worse than token-level OREO (by 1.5 points on GSM8K, 3.1 points on MATH). Since response-level OREO differs from DPO primarily in using a squared-error Bellman consistency loss rather than a Bradley-Terry preference loss, this result isolates the effect of credit assignment granularity from the effect of the loss function form. The fact that response-level OREO outperforms DPO (49.4 vs. 49.2 on MATH; 75.8 vs. 74.4 on GSM8K) suggests that the Bellman consistency formulation has a modest inherent advantage over preference-based optimization, but the large gap between response-level and token-level OREO demonstrates that per-step granularity is the dominant factor in OREO's gains.

Second, step-level OREO performs comparably to token-level OREO, slightly better on GSM8K (78.1 vs. 77.3) and slightly worse on MATH (51.2 vs. 52.5). This non-uniformity may reflect dataset-specific properties: GSM8K problems have clearly delineated steps (separated by line breaks in the original data), making step-level boundaries informative, while MATH problems have more variable formatting, potentially introducing noise from imperfect segmentation. The paper speculates that step-level optimization may suffer from "the value function's limited accuracy at each step, which introduces noise into policy learning" (Section 5.1), though this explanation would predict underperformance on both datasets, not the observed pattern.

Third, the regularization weight α differs substantially across variants (0.01 for token-level, 0.1 for step-level, 0.3 for response-level) because the loss magnitudes differ — response-level OREO's loss is computed over the entire trajectory before squaring, producing larger values that require stronger regularization to prevent the policy from diverging rapidly. This sensitivity of the optimal α to the loss variant is a practical concern: practitioners adopting OREO would need to tune α for their specific variant, and the paper does not provide a principled method for doing so beyond grid search.

Test-Time Search: The Explicit Value Function Provides Monotonic Gains with Beam Width

Figure 3 presents beam search results on GSM8K and MATH500 (the 500-problem subset) using the 1.5B Qwen model trained with OREO:

  • GSM8K: Greedy decoding (B=1) achieves approximately 77-78% accuracy. Beam search with B=7 achieves approximately 86%, representing an 11.4% relative improvement.
  • MATH500: Greedy decoding achieves approximately 39% accuracy. Beam search with B=7 achieves approximately 46%, representing a 17.9% relative improvement.

The accuracy curves are monotonically increasing with beam width for both datasets, with no signs of saturation at B=7. This is significant because it demonstrates that the value function is making genuinely informative distinctions between partial trajectories — if the value function were noisy or easily exploited, performance would plateau or degrade at higher beam widths (as occurs with verifier over-optimization in PRM-based search; Snell et al., 2024). The larger relative improvement on MATH (17.9% vs. 11.4% on GSM8K) is consistent with the intuition that search provides greater benefit on harder problems where the policy's greedy decoding is less reliable and exploration of alternatives is more valuable.

The paper frames these results as demonstrating that "the explicit value function is more effective than the policy in distinguishing between correct and incorrect reasoning steps" (Section 5.4). However, no direct comparison with policy-guided beam search (using the implicit value function as the scoring heuristic) is provided. The "rejection sampling" line in Figure 3 — which shows substantially lower accuracy at all beam widths — is not clearly described: it is unclear whether this represents beam search using the rejection sampling model (which has no value function and would need a different scoring heuristic) or some other configuration. This missing ablation makes it difficult to attribute the beam search gains specifically to the explicit value function rather than to OREO's policy improvement or to the beam search procedure itself.

ALFWorld Best-of-K: Value-Guided Action Selection Shows Rapid Gains That Stabilize at K=5

Figure 4 presents success rates in ALFWorld when sampling K candidate actions at each step and selecting the one with the highest value function score:

  • With K=1 (no selection, equivalent to greedy decoding), success rate is approximately 79%.
  • With K=2, success rate rises to approximately 84%.
  • With K=3, success rate rises to approximately 86%.
  • With K=4, success rate rises to approximately 87%.
  • With K=5, success rate rises to approximately 88%.

The gains are largest from K=1 to K=2 (+5 percentage points) and diminish rapidly thereafter, with K=5 providing only +1 point over K=4. This rapid saturation contrasts with the math beam search results, which show continued improvement out to B=7. Two factors may contribute: first, ALFWorld actions are high-level decisions (e.g., "go to desk 1") rather than token sequences, making the action space smaller and easier to cover with a few samples; second, the value function in ALFWorld must generalize across different environment configurations (seen vs. unseen), and its estimates may be less precise for novel states, limiting the benefit of evaluating many candidates.

The paper does not compare best-of-K with alternative selection strategies (e.g., majority voting across K independently executed trajectories, or using the policy's own probabilities as scores). This makes it difficult to assess whether the value function provides information beyond what the policy's likelihoods already encode about action quality.

Ablation Studies and Robustness Checks

  • Implicit vs. explicit value function comparison (Section 5.3, Figure 1): The paper provides qualitative case studies comparing advantage estimates from the explicit value function V_φ against those derived from the policy's implicit value function (via the cumulative log-ratio). In a GSM8K example, both correctly favor the correct continuation, but the explicit advantage (-0.431 for an incorrect step) has substantially larger magnitude than the implicit advantage. In a more challenging MATH example, the explicit advantage correctly identifies the correct reasoning step while the implicit advantage fails. The paper attributes this to the "softmax bottleneck" (Yang et al., 2017): the policy's log-ratios are constrained by the linear-then-softmax architecture of the language modeling head, while the explicit value function processes the full state through the entire transformer. This ablation is qualitative, not quantitative — no metric like step-classification AUROC is reported, and no beam search experiment using the implicit value function as the scoring heuristic is performed to quantify the downstream impact.

  • OREO vs. DQO comparison (Appendix C, Table 3): The paper reproduces DQO (Direct Q-function Optimization, Liu et al., 2024a) — a concurrent work that derives a similar algorithm from the SAC framework — and compares it with OREO using the same base model (Qwen2.5-Math-1.5B) and training data. OREO achieves 77.3% on GSM8K and 52.5% on MATH, while DQO achieves 75.1% and 49.4% respectively. The paper identifies three differences between the methods: (1) derivation approach (PCL vs. SAC), (2) inclusion of a KL regularization term, and (3) exploration of loss variants. It does not ablate which difference(s) account for the performance gap. The DQO reproduction uses the paper's description without access to original code ("As their work is not yet open-sourced"), which introduces reproducibility uncertainty — the gap could reflect implementation differences rather than algorithmic superiority.

  • Dataset balance for 7B model training (Appendix A.1): For the DeepSeekMath-7B experiments, the paper enforces a class balance constraint: the number of positive (successful) trajectories does not exceed the number of negative (failed) trajectories, with at least one positive instance per query if any exist, up to a maximum of 4 of each type from 16 sampled responses. This constraint is not applied in the 1.5B experiments, where all 10 sampled responses are used regardless of class ratio. The paper does not report an ablation comparing balanced vs. unbalanced training for the 1.5B model, making it impossible to determine whether the class balance — rather than model size or base capability — contributes to the 7B results.

  • LoRA for critic on 7B models (Appendix A.3): To reduce computational cost, the value model for 7B experiments is trained using LoRA with rank 64 and alpha 64, while the 1.5B experiments use full fine-tuning for both policy and critic. No ablation comparing LoRA vs. full fine-tuning for the value function is reported, leaving open the possibility that full fine-tuning would further improve the 7B results.

  • Segmentation method for step-level OREO (Appendix A.2): For step-level OREO and beam search, reasoning steps are segmented using line breaks (GSM8K) and both line breaks and periods (MATH). The paper acknowledges that this is a heuristic — "the choice of separators is arbitrary and may not perfectly align with semantic reasoning steps, especially for models that produce unconventional formatting" — but does not ablate alternative segmentation strategies (e.g., using a learned step boundary detector, or segmenting by sentence). The sensitivity of step-level OREO and beam search to segmentation quality is therefore unknown.

  • Iterative training limited to 1.5B model and three iterations (Section 5.2, Figure 2): The iterative training experiments use only the 1.5B Qwen model and stop after three iterations. The paper's computational limitations prevent testing whether OREO's advantage over rejection sampling continues to widen with more iterations or whether OREO too eventually saturates. Similarly, iterative training is not tested for ALFWorld or for the 7B model, leaving the generalizability of the sustained-improvement finding unverified.

  • No ablation on the stop-gradient operator (Equation 9): The stop-gradient on future actions is described as making "each step have the same scale in the gradient" (Section 4.1), but no experiment compares training with and without stop-gradient. In standard PCL (Nachum et al., 2017), the stop-gradient is part of the algorithm, but its necessity in the LLM fine-tuning context — where training dynamics differ from traditional RL — is not empirically validated.

  • Negative result with ReST^EM (Appendix K, referenced in the prior sections but not evaluated in detail in this paper): The paper mentions that an attempt to further optimize the revision model using ReST^EM "backfired" and caused performance to degrade, but the main OREO experiments do not include this negative result. The absence of negative results in the main experimental sections — every reported comparison shows OREO outperforming baselines — may reflect selective reporting or the method genuinely dominating in all tested configurations.

Critical Assessment

The paper advances four central claims through its experiments. I assess each against the evidence presented.

Claim 1: OREO outperforms existing offline learning methods (DPO, KTO, rejection sampling) on multi-step reasoning tasks.

This claim is supported by the evidence in Table 1 and Table 2, but with two important caveats about scope. First, the claim holds for the specific models tested (Qwen-2.5-Math-1.5B, DeepSeekMath-7B-Instruct, MiniCPM-2B) and the specific datasets (GSM8K, MATH, ALFWorld). Whether OREO's advantage generalizes to different model families (e.g., LLaMA, Gemma), different reasoning domains (e.g., code generation, scientific reasoning), or larger model scales (e.g., 70B+) is entirely untested. The paper's computational constraints (Section 7) mean the evidence is from a narrow slice of the possible experimental space. The consistent improvement across all tested configurations is encouraging but does not constitute proof of general superiority.

Second, the margin of improvement varies substantially. On GSM8K with the 1.5B model, OREO's 2.4-point gain over rejection sampling (77.3 vs. 74.9) is meaningful but modest — a 3.2% relative improvement. On MATH with the 1.5B model, the 2.2-point gain (52.5 vs. 50.3) is a 4.4% relative improvement. On DeepSeekMath-7B GSM8K, the 2.3-point gain (85.9 vs. 83.6) is only a 2.7% relative improvement. These are real but incremental gains, not transformative leaps. The largest relative gains appear in the ALFWorld unseen setting (79.1 vs. 68.7, a 15.1% relative improvement), suggesting OREO's benefits may be most pronounced in out-of-distribution generalization scenarios rather than in-distribution accuracy. The lack of confidence intervals means we cannot assess whether the smaller margins (e.g., 0.3 points on DeepSeekMath-7B MATH) are statistically distinguishable from noise.

A notable missing baseline is PPO with online data collection — the method OREO is positioned as a cheaper alternative to. Without an online RL baseline, we cannot assess how much performance is left on the table by going offline. If online PPO with the same base model achieves, say, 55% on MATH for the 1.5B model (compared to OREO's 52.5%), then OREO trades some performance for practicality — a reasonable trade-off, but one the paper does not quantify. If online PPO achieves only 52% (equal to OREO), then OREO is strictly better. The reader cannot distinguish these scenarios from the reported results.

Claim 2: OREO eliminates the need for paired preference data and enables fine-grained credit assignment.

This claim is supported by two complementary lines of evidence: (1) OREO outperforms DPO and KTO (which require or approximate pairwise preferences) while using only unpaired data with terminal rewards, and (2) the loss variant ablation (Table 5) shows that per-step credit assignment (token-level OREO) substantially outperforms trajectory-level credit assignment (response-level OREO). However, the evidence for the credit assignment mechanism specifically is indirect. The paper argues that OREO enables fine-grained credit assignment, but never directly demonstrates that the value function is correctly attributing credit to specific steps. The case studies in Figure 1 are suggestive (the value function drops on error steps) but qualitative and cherry-picked. A quantitative evaluation — for instance, measuring the value function's step-level correctness classification accuracy against ground-truth step labels, or showing that the policy's probability of error tokens specifically decreases while non-error tokens are unaffected — would have provided direct evidence. As presented, we see that the method with per-step structure outperforms the method without it, but the mechanism (credit assignment) is inferred rather than verified.

Similarly, the claim about eliminating the need for paired data is supported by OREO outperforming DPO, but the comparison with KTO complicates the narrative. KTO also eliminates the pairwise requirement (it works on unpaired data with binary feedback) yet underperforms DPO in several settings (e.g., 73.4 vs. 74.4 on 1.5B GSM8K; 48.3 vs. 49.2 on 1.5B MATH). This suggests that removing the pairwise constraint alone is insufficient — it's the combination of unpaired data plus per-step credit assignment that matters. The paper's abstract claims OREO "reduces the need to collect pairwise data," which is literally true but undersells the fact that the credit assignment mechanism is the more fundamental innovation.

Claim 3: The learned value function can be used for test-time search, providing up to 17.9% relative improvement over greedy decoding.

This claim is supported by Figure 3, which shows monotonic improvement in beam search accuracy as beam width increases from 1 to 7. However, the claim is narrower than it might appear. First, the 17.9% figure is specifically for MATH500 (the 500-problem subset) with the 1.5B model at B=7. On GSM8K, the improvement is 11.4%. On the full MATH test set (5,000 problems), beam search results are not reported — only the 500-problem subset (following Lightman et al., 2023) is used. If the 500-problem subset is not representative of the full MATH distribution, the 17.9% figure may not generalize.

Second, the test-time search results are not compared against alternative test-time compute scaling strategies at equalized compute budgets. OREO with B=7 beam search generates 7 × 7 = 49 candidate continuations at each step, plus the base generation cost — a substantial compute multiplier over greedy decoding. A comparison against best-of-N sampling (generate N complete solutions and select via majority voting or the value function), or against a simpler method like temperature sampling with the same total FLOPs budget, would reveal whether the beam search procedure specifically adds value beyond simply spending more compute. Snell et al. (2024) demonstrated that the optimal test-time compute strategy depends heavily on problem difficulty and that beam search can actually underperform best-of-N for easy problems due to verifier over-optimization. OREO's beam search results are aggregated across all difficulties, potentially masking such heterogeneity.

Third, the paper does not ablate whether the beam search gains come from the value function's quality or from the OREO policy's quality. A simple experiment — train beam search using the rejection sampling model's outputs scored by a separately trained PRM or even by the OREO value function — would decouple these factors. If beam search with the OREO value function but the rejection sampling policy achieves similar gains, then the value function is the key enabler. If not, then OREO's policy improvement is doing most of the work and the value function's contribution to search is secondary.

Claim 4: Iterative OREO sustains improvement while rejection sampling saturates.

The evidence in Figure 2 supports this claim for three iterations on 1.5B models. However, the effect size is modest: by iteration 3, OREO leads rejection sampling by approximately 2-3 percentage points on GSM8K and 2-3 points on MATH. These are meaningful gaps but not dramatic — a practitioner running three full iterations of data collection and retraining might expect larger returns from the additional complexity of training a value function. Moreover, OREO's own gains decelerate (nearly flat from iteration 2 to 3 on MATH), suggesting that even with failure learning, the iterative approach may approach an asymptote. Whether OREO's advantage widens, narrows, or stabilizes with additional iterations (4, 5, 10) is unknown. The paper's computational constraints prevent determining whether OREO's sustained improvement is a transient advantage (rejection sampling saturates earlier but catches up eventually) or a permanent one (OREO converges to a higher asymptote).

The absence of iterative DPO results is a significant omission. If DPO were run iteratively (as in Xiong et al., 2024; Pang et al., 2024) and showed similar or better sustained improvement, then the claimed mechanism (learning from failures) would be called into question — DPO also learns from failures, albeit through pairwise comparison rather than value function regression. The paper's theoretical analysis predicts DPO should underperform due to lack of credit assignment, but iterative on-policy data collection can partially mitigate this (errors appear in new contexts, providing implicit credit assignment through trajectory-level comparison). Without this ablation, we cannot distinguish whether iterative OREO's advantage comes from the learning objective or from some other factor (e.g., better exploration due to the value function's uncertainty, or more effective use of the same data).

Cross-cutting weaknesses across all claims:

  • No statistical reporting: The absence of confidence intervals, standard deviations, or significance tests makes it impossible to assess the reliability of the reported differences. With 1,319 test examples (GSM8K), a 2-point accuracy gap corresponds to approximately 26 examples — modest enough that run-to-run variance from different random seeds (in data generation, training initialization, or evaluation) could plausibly produce it. The 500-example MATH subset results have even wider confidence intervals.

  • Single evaluation protocol: All results are single-run point estimates. There is no mention of multiple seeds, bootstrap confidence intervals, or test-retest reliability. This is standard practice for many LLM papers but is a genuine limitation for interpreting close comparisons.

  • Limited model scale: The largest model tested is 7B parameters. Whether OREO's advantages hold, grow, or diminish at the scales where multi-step reasoning becomes most valuable (70B+) is unknown. The softmax bottleneck argument (Section 5.3) predicts that explicit value functions become more important at larger scales (where the representational constraint of the shared hidden state is more binding), but this prediction is untested.

  • No comparison with process reward model (PRM) baselines: Despite positioning the value function as a principled alternative to PRMs (Section 4.4), the paper never compares against a PRM-based system. A direct comparison — e.g., OREO with its value function vs. the same base policy with a Monte-Carlo-trained PRM guiding beam search — would validate the claim that OREO's principled approach matches or exceeds heuristic PRM training. As presented, we know OREO's value function works for beam search, but not whether it works better than existing PRM methods.

  • ALFWorld results are on a small test set: The 140 seen and 134 unseen tasks provide limited statistical power. A 10 percentage point difference in unseen environments (79.1 vs. 68.7) represents approximately 14 tasks — meaningful but potentially influenced by a small number of hard or easy tasks in the specific split.

Summary of what the experiments do and do not demonstrate: The experiments convincingly show that OREO, as implemented with specific hyperparameters and models, outperforms the tested baselines on the tested benchmarks. They provide suggestive evidence for the importance of per-step credit assignment (via the response-level ablation) and the value of explicit value functions (via the beam search results and qualitative case studies). They do not definitively establish that the credit assignment mechanism works as theorized (direct evidence is qualitative), that the explicit value function is the causal factor in beam search gains (no head-to-head search heuristic comparison), that iterative training provides a sustained and widening advantage (only three iterations tested), or that OREO's advantages generalize beyond the specific experimental configuration (narrow model and dataset coverage, no statistical reporting). The paper's contributions are best understood as a strong proof of concept rather than a fully validated general solution. The theoretical framework is elegant and the empirical results are consistently positive, but the experimental design leaves several causal claims dependent on inference rather than direct demonstration.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Remains Unresolved—and For This Method, It Would Be Even More Expensive

The assumption or constraint. The paper's entire training framework assumes access to a pre-collected offline dataset of reasoning trajectories, each carrying a terminal success/failure label. The paper is explicit that this data already exists before OREO is applied: the method "enable[s] practitioners to tune models using pre-existing datasets, eliminating the need for live interaction or data generation" (Section 1). The construction of this dataset—sampling 10 responses per query for 1.5B models, 16 responses per query for 7B models, performing 5 rollouts per task for ALFWorld (Appendix A.1)—is treated as a one-time upfront cost that is not accounted for in any training or inference budget comparison.

The consequence. This creates a significant practical asymmetry. To apply OREO to a new task or domain, a practitioner must first generate a large number of candidate trajectories from the base model, execute them (in the case of ALFWorld, this means running them through the environment simulator), and verify their correctness. For math reasoning, this means generating and grading perhaps 75,000+ candidate solutions for MATH (7,500 training problems × 10 samples). For embodied agent tasks, this means running 15,000+ environment rollouts (3,119 training tasks × 5 samples). The cost of this initial data collection—measured in FLOPs, API calls, or environment interactions—can easily exceed the cost of the OREO training run itself. For comparison, other offline methods like SFT on ground-truth solutions require zero additional data generation (the training set already exists), and rejection sampling requires only the same generation cost as OREO's data collection step but discards the failures rather than using them. The paper's claim that OREO is "more efficient" than online RL is true in the sense that no additional environment interaction is needed during training, but the upfront data generation cost is substantial and entirely unaccounted for in the efficiency claims. This is not a minor overhead: generating, say, 75,000 candidate MATH solutions at 500 tokens each from a 7B model requires on the order of 3.75 × 10^10 inference FLOPs per query × 75,000 queries, which is roughly comparable to the training FLOPs of fine-tuning the model itself.

What evidence exists in the paper. The paper does not measure or report the computational cost of offline dataset construction. Table 1 reports training-time metrics (accuracy after 1–3 epochs) without reporting the FLOPs or wall-clock time required for data generation. The Appendices describe the sampling procedure (A.1) but do not provide cost estimates. The test-time search experiments (Section 5.4) partially account for inference cost by reporting accuracy as a function of beam width B, but the training data generation cost is never amortized into the reported efficiency comparisons.

Mitigation status. Not addressed. The paper does not discuss the data collection cost, propose methods to reduce it, or suggest that existing public datasets could substitute. The text treats the offline dataset as given—a reasonable assumption for a method paper, but one that a practitioner evaluating deployment costs would need to quantify independently. The paper's positioning as "eliminating the need for live interaction or data generation" (Section 1) is technically accurate for the training phase, but masks the fact that live data generation was already required before training could begin.

The Method Is Demonstrated Only on Models Under 7B Parameters and Two Task Families

The assumption or constraint. The paper evaluates OREO on three model sizes (1.5B, 2B, 7B) from three model families (Qwen-2.5-Math, DeepSeekMath, MiniCPM) across two task types (mathematical reasoning with GSM8K and MATH; embodied agent control with ALFWorld). The authors acknowledge this scope limitation explicitly in Section 7:

"Due to limited computation resources, some of our experiments, including ablation studies, iterative OREO, and test-times search, use 1.5B models. We plan to run experiments on larger scales in the future. Our method has primarily been evaluated on mathematical reasoning and embodied agent tasks. As future work, we aim to extend OREO to a wider variety of tasks, such as coding and web browsing."

The consequence. This scope limitation matters for three concrete reasons, each with different implications for practitioners:

First, the softmax bottleneck argument (Section 5.3) predicts that explicit value functions should become more important at larger model scales, because larger models have more representational capacity in their hidden states and the constraint of routing all token predictions through a single linear projection head becomes more binding relative to that capacity. If this prediction holds, OREO's advantage over DPO and rejection sampling should grow with model scale—the 1.5B and 7B results would represent a lower bound on the method's benefit. If the prediction fails (e.g., larger models already have good implicit value estimates through better-calibrated probabilities), OREO's advantage might shrink or vanish at scales where reasoning capabilities become most commercially valuable. The paper provides no evidence either way.

Second, the diversity of mathematical reasoning in GSM8K and MATH may not represent the full spectrum of reasoning tasks where OREO would be deployed. Both datasets feature problems with clean, verifiable final answers (numeric or mathematical expressions) and well-structured reasoning chains. Code generation (where errors compound differently and correctness is verified by unit tests rather than exact match), scientific reasoning (where steps may involve uncertain or probabilistic inferences), or open-ended planning (where "correctness" is ambiguous) may present fundamentally different credit assignment challenges. The step-level segmentation heuristic (splitting on line breaks and periods, Appendix A.2) would need substantial re-engineering for domains with different formatting conventions like code or structured output.

Third, the 7B DeepSeekMath model was already heavily fine-tuned on 776K mathematical reasoning samples (Shao et al., 2024), meaning OREO's ability to improve on top of this strong baseline is genuinely impressive—but also means the starting point is unusually favorable. A 7B general-purpose model without math-specific pretraining might show different relative gains. The paper does not test whether OREO's advantage depends on the base model's domain alignment.

What evidence exists in the paper. The limitation is fully self-reported (Section 7). All main results (Tables 1, 2; Figures 2, 3, 4) are from models at or below 7B parameters. The paper provides no ablation comparing OREO's relative gain across model sizes—the 1.5B and 7B results use different base model families (Qwen vs. DeepSeekMath) with different pretraining recipes and different dataset construction protocols (10 vs. 16 samples per query, balanced vs. unbalanced class ratios), making cross-model comparison of relative improvement impossible.

Mitigation status. Acknowledged as future work. No evidence is provided that the findings would generalize to larger models or broader task families.

No Online RL Baseline Exists to Quantify the Performance Cost of Going Offline

The assumption or constraint. OREO is positioned as an offline alternative to expensive online RL methods like PPO. The paper's introduction establishes the cost motivation:

"tuning LLMs with Proximal Policy Optimization (PPO, Schulman et al., 2017) is often prohibitively expensive for most users, which limits practical applications" (Section 1).

OREO is then presented as achieving similar objectives without online interaction. However, the paper never actually runs an online RL baseline to establish what performance is being sacrificed for this practicality. The closest comparison is with DPO and rejection sampling—both offline methods themselves.

The consequence. Without an online RL comparison, a practitioner reading this paper cannot answer the central trade-off question: how much performance am I leaving on the table by choosing OREO's offline convenience over PPO's online data collection? If online PPO on the same base model with the same reward signal achieves, say, 55% on MATH (vs. OREO's 52.5% for the 1.5B model), the 2.5-point gap quantifies the cost of going offline—a cost that might be acceptable for a small research team but unacceptable for a production deployment where every point matters. If online PPO achieves only 50% (worse than OREO), then OREO is strictly dominant and the method's value proposition is much stronger. The paper provides no basis for distinguishing these scenarios.

This gap matters concretely because the RLHF literature has shown that online data collection often provides significant benefits over offline methods—the policy can explore states that the base model never visited, receive feedback on those states, and learn from its own mistakes in a way that no fixed offline dataset can replicate. OREO's inability to explore novel states (it is limited to the states present in the offline dataset) means it can only learn to avoid mistakes that already appear in the data. If the base model makes a systematic error that never produces a correct trajectory in the dataset, OREO has no signal for how to fix it—all trajectories containing that error are failures, and the value function can only learn that states following that error are bad, not what alternative action would be good. Online RL could discover a fix through exploration.

What evidence exists in the paper. None. There is no online RL baseline in any experiment. The paper's comparison with DQO (Appendix C, Table 3) is the closest methodological comparison, but DQO is also an offline method. The iterative OREO results (Figure 2) demonstrate that collecting new data with the updated policy and retraining provides additional gains, which is a form of online data collection, but only with the policy model's greedy outputs—not with the kind of exploration bonuses or reward-model-guided search that characterize online PPO pipelines.

The absence of this comparison is not necessarily a flaw in the paper—the authors are explicit that they are proposing an offline method and evaluating it against other offline methods—but it leaves a critical practical question unanswered for anyone trying to decide between "use OREO" and "pay for online PPO."

Mitigation status. Not addressed. The paper never discusses what performance gap might exist between offline and online methods for reasoning tasks, nor does it cite prior work that might quantify this gap.

Training a Separate Value Model Doubles the GPU Memory Footprint and Introduces Hyperparameter Sensitivity

The assumption or constraint. OREO requires training a separate value model V_phi alongside the policy model pi_theta. For the 1.5B experiments, both models are fully fine-tuned. For the 7B experiments, the value model is trained with LoRA (rank 64, alpha 64) to reduce memory usage (Appendix A.3). The value model has its own learning rate (10^{-4} for the critic vs. 5 × 10^{-6} for the policy), its own optimizer state, and must be kept in GPU memory alongside the policy, the reference policy, and the training data. The paper does not report GPU memory usage, peak memory, or training throughput for any experiment, nor does it compare resource requirements against baseline methods.

The consequence. This represents a substantial practical overhead compared to the baseline methods OREO outperforms. Rejection sampling requires only the policy model and the training data—no reference model, no value model, no Bellman consistency loss computation. DPO requires the policy and a frozen reference policy but no value model. Moving from DPO to OREO approximately doubles the number of trainable parameters in memory (policy + value vs. policy alone) and adds the computational cost of forward passes through the value model for every state in every trajectory. For the 7B experiments, the paper mitigates this partially with LoRA on the critic, but this introduces a new trade-off: LoRA reduces memory at the potential cost of value function accuracy. The paper does not report an ablation comparing LoRA vs. full fine-tuning for the value model, so the accuracy cost of this memory-saving choice is unknown.

Beyond memory, the introduction of a separate value model introduces additional hyperparameters that practitioners must tune: the value model's learning rate (set to 10^{-4}, 20× higher than the policy learning rate), the LoRA configuration (rank and alpha), whether to use the same architecture as the policy or a smaller model, and the interaction between policy and value learning rates (if the value model learns too slowly, it provides poor guidance to the policy; if it learns too quickly, it may overfit the offline data and provide misleading credit assignment). The paper's sensitivity analysis on these choices is minimal—the regularization weight alpha is tuned for different OREO variants (0.01, 0.1, 0.3; Appendix A.3), but learning rates and LoRA configurations are reported without ablations.

What evidence exists in the paper. Appendix A.3 reports the hyperparameters without reporting memory usage, training time, or hardware requirements. The paper does not include a table comparing GPU hours or peak memory across methods. The LoRA ablation (full fine-tuning vs. LoRA for the value model) is absent. The sensitivity to value model learning rate is untested. The only evidence of hyperparameter sensitivity comes from the alpha tuning for different loss variants (Section 4.2), which required grid search over {0.01, 0.1, 0.3}—a relatively narrow range that produced different optimal values for different variants, suggesting sensitivity to this parameter.

Mitigation status. Partially addressed through LoRA for 7B models, which the paper acknowledges is "to save computation" (Appendix A.3), but the performance impact of this choice is not measured. The paper does not propose methods to reduce the value model's memory footprint further (e.g., distillation, shared backbone with a value head, or off-policy training schemes that require fewer value function evaluations).

The Method Has No Mechanism to Handle Correct-to-Incorrect Reversions During Iterative Training

The assumption or constraint. OREO's value function V_phi(s_t) is trained to predict expected future returns from the states present in the offline dataset. In the first iteration of OREO, these states are generated by the SFT model, which has a particular error profile. After training, the policy pi_theta is updated and produces a different distribution of states and errors. In the iterative OREO setting (Section 4.3, Section 5.2), the updated policy generates new trajectories that become the training data for the next iteration. The paper assumes this process will be beneficial because "the updated policy model in each new iteration may be able to explore novel failure patterns, and incorporate these insights into the learning process" (Section 5.2).

The consequence. This assumption may not hold in practice for a specific class of failures: errors that the original SFT model did not make, but that the OREO-updated policy introduces. If OREO's first training iteration reduces the probability of certain failure modes while inadvertently increasing the probability of others (a common phenomenon in offline RL known as distributional shift or policy churn), the second iteration's dataset will contain these new failures. The value function, retrained on this new data, may correctly flag these new failure states, and the policy may learn to avoid them—leading to a virtuous cycle as the paper suggests. But it is equally possible that the value function, seeing unfamiliar states in the second iteration's data, produces poorly calibrated estimates that mislead the policy into worsening certain behaviors. The paper provides no mechanism to detect or prevent this form of degradation.

This is not a hypothetical concern. The paper's related work discussion (Section 2.2) explicitly references cases where RL-based self-improvement has backfired—Huang et al. (2023) found that "large language models cannot self-correct reasoning yet" in some settings, and the reference example paper mentions a ReST^EM experiment where "additional sequential revisions substantially hurt performance" (Appendix K). OREO's iterative variant introduces the same structural risk: the policy shifts, the data distribution shifts, and the value function—trained on the old distribution—may not generalize to the new one. Without an online data collection step that can validate the value function's estimates on the policy's own outputs, there is no guardrail against this degradation spiral.

What evidence exists in the paper. The iterative training results (Figure 2) show monotonic improvement over three iterations for both OREO and rejection sampling, with no evidence of degradation. This suggests that distributional shift was not catastrophic in these specific experiments. However, three iterations on the specific datasets and models tested is a narrow evidence base. The paper does not report whether the value function's calibration degrades across iterations (e.g., does the MSE loss on held-out data increase?), whether the policy's output distribution diverges measurably from iteration to iteration, or whether certain problem categories show regression even as aggregate accuracy improves. The absence of degradation in this specific setup does not guarantee absence of degradation in other setups (different models, different tasks, more iterations, different hyperparameters).

Mitigation status. Not addressed. The paper's iterative training procedure (Section 4.3) describes the data collection and retraining loop without discussing stability mechanisms. Standard techniques from offline RL for addressing distributional shift—such as keeping the reference policy fixed (which OREO already does via the KL penalty), using a replay buffer that mixes data from multiple iterations, constraining policy updates to be small (trust region methods), or early stopping based on validation performance—are neither discussed nor evaluated. The KL penalty beta = 0.03 provides some regularization, but its effect in the iterative setting is untested (the paper does not ablate beta's effect on iterative training stability).

Test-Time Search Gains Are Not Compared Against Alternative Compute-Allocation Strategies at Equalized Budgets

The assumption or constraint. Section 5.4 and Figures 3–4 present test-time search results where OREO's value function guides step-level beam search (math reasoning) or best-of-K action selection (ALFWorld). The metric is accuracy or success rate as a function of beam width B or sample count K. The paper concludes that "OREO leverages the value function to achieve progressively higher accuracies as the computational budget increases" (Section 5.4) and reports a 17.9% relative improvement over greedy decoding on MATH500 at B=7.

The consequence. These results demonstrate that spending more inference compute through OREO's value-guided search improves OREO's own performance. They do not demonstrate that OREO's value-guided search is a better use of inference compute than alternative strategies that could be applied to the same policy model. Consider a practitioner who has trained a model with OREO and has a fixed inference budget of, say, 49× the cost of greedy decoding (corresponding to B=7 beam search, which generates 7 continuations from each of 7 beams at each step, plus the base generation cost). Should they run value-guided beam search with B=7? Or should they run best-of-49 sampling—generate 49 complete solutions from the policy and select the best via majority voting or the value function's final-answer score? Or should they run a hybrid: 7 independent solutions with light beam search (B=2 or B=3) on each?

The paper provides no evidence to answer this question. The rejection sampling line in Figure 3 is ambiguous—it is not clearly described what search strategy this represents (beam search with what scoring heuristic? best-of-N? something else?). Even if it represents a meaningful comparison, it compares different training methods (rejection sampling vs. OREO) under different inference strategies, confounding the training effect with the inference effect. What a practitioner needs is: given an OREO-trained model, how does value-guided beam search compare to best-of-N at the same total FLOPs budget? The paper's test-time search results cannot distinguish whether the gains come from spending more compute (which always helps, up to a point) or from spending compute better (via the value function's guidance specifically).

This is a significant gap because the test-time compute scaling literature (Snell et al., 2024; Wu et al., 2024; Brown et al., 2024) has shown that the optimal allocation strategy depends critically on problem difficulty, verifier quality, and budget level. Beam search can over-optimize an imperfect verifier and underperform best-of-N on easy problems (Snell et al., 2024). OREO's aggregated results across all problem difficulties may mask such heterogeneity. A practitioner deploying OREO would need to know whether B=7 beam search is genuinely better than best-of-49, or whether they are paying a 49× compute multiplier for gains that could be achieved with a much cheaper strategy.

What evidence exists in the paper. Figure 3 plots accuracy vs. beam width B but does not include an alternative scaling curve (e.g., accuracy vs. number of samples N in best-of-N) at matched generation counts. Figure 4 plots success rate vs. K for best-of-K action selection but does not compare against executing all K actions to completion and selecting the best final outcome, or against simpler heuristics like policy probability-based selection. The rejection sampling line in Figure 3 is under-specified—the paper does not state what search strategy it uses (beam search with what scorer? greedy decoding? best-of-N with what N?), making interpretation impossible. The compute budget for beam search (B² continuations evaluated per step) is not quantified relative to the cost of generating a complete trajectory, making it difficult even to estimate the FLOPs multiplier.

Mitigation status. Not addressed. The paper frames the test-time search results as demonstrating that "the explicit value function is more effective than the policy in distinguishing between correct and incorrect reasoning steps" (Section 5.4). Even if this is true (the qualitative case studies suggest it is), the downstream consequence—that value-guided beam search is an efficient use of inference compute—requires additional evidence that the paper does not provide. The question of compute-optimal test-time strategy allocation (as studied by Snell et al., 2024) is orthogonal to the method's training contribution and would require a separate experimental design.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around offline RL for LLM reasoning by providing a diagnostic framework rather than just another loss function. The field has accumulated a confusing set of results about what works for improving LLM reasoning from offline data: DPO sometimes helps, sometimes doesn't; rejection sampling is simple and often effective but seems to hit ceilings; PRMs enable test-time search but require substantial engineering. Prior work responded to this confusion by proposing variations on each approach—better pair construction for DPO (Chen et al., 2024a; Song et al., 2024), iterative rejection sampling (Zelikman et al., 2022; Singh et al., 2023), improved PRM training recipes (Luo et al., 2024; Zhang et al., 2024)—without a unified understanding of why certain methods succeed or fail on reasoning tasks specifically.

OREO's core contribution is to show that the soft Bellman Equation serves as a unifying diagnostic that explains this fragmented empirical landscape. The derivation in Section 3.3 demonstrates that DPO can be obtained from the soft Bellman Equation by applying exactly two operations: telescoping the per-step consistency into a trajectory-level sum, and introducing the Bradley-Terry preference model. Each operation corresponds to one of DPO's documented failures on reasoning: telescoping destroys credit assignment (the model cannot distinguish which step caused the error), and the BT model forces an unnatural pairwise structure onto data that naturally carries per-instance success/failure labels. This is not a matter of implementation quality or hyperparameter tuning—it is structural. The response-level OREO ablation (Table 5) provides empirical teeth to this diagnostic: when you restore per-step granularity, MATH accuracy jumps from 49.4 to 52.5; when you strip it away, performance degrades to near-DPO levels even though the Bellman consistency loss is used instead of a preference loss. This isolates credit assignment granularity as a causal factor in DPO's underperformance, not merely a correlated observation.

The reframing is equally significant for the PRM literature. The paper shows that the value function learned by enforcing soft Bellman consistency (Equation 8) is mathematically equivalent to the Monte Carlo rollout labels used to train PRMs, but derived from first principles rather than sampling heuristics. This transforms PRMs from an engineering artifact—requiring practitioners to make ad-hoc decisions about rollout counts, label types, and aggregation methods—into a principled component of a maximum-entropy RL framework. The paper's beam search results (Figure 3: 17.9% relative improvement over greedy decoding on MATH500 at B=7, with monotonically increasing accuracy as beam width grows) demonstrate that this principled approach produces a value function that works for test-time search without the "substantial data engineering and heuristic design efforts" (Section 5.4) that characterize prior PRM work. This is not merely a new PRM training recipe—it is a conceptual upgrade that connects verifier training to decades of theory in value-based RL, opening the door to techniques like temporal difference learning, eligibility traces, distributional value functions, and ensemble uncertainty estimation that the PRM literature has not yet explored.

The paper also provides the first clear empirical demonstration—in the LLM reasoning context—that learning from failed trajectories sustains iterative improvement while success-only training saturates (Figure 2). The offline RL literature has long predicted this (Kumar et al., 2022), but demonstrating it for LLM reasoning required a method that could extract useful signal from failures without requiring online interaction. OREO's three-iteration comparison against rejection sampling on GSM8K and MATH shows that rejection sampling's gains diminish sharply after the first iteration, while OREO continues to improve. This finding has immediate implications for the design of self-improvement pipelines: the common practice of filtering for correctness and discarding failures (STaR, ReST^EM, RAFT) may be fundamentally limited not by data quantity or diversity, but by the structural inability of success-only training to extract information from the most informative trajectories—the failures that reveal what the model still gets wrong.

The magnitude of this contribution is best characterized as a conceptual reframing with strong proof-of-concept evidence, not a paradigm shift. The experiments are limited to models under 7B parameters on two task families (math reasoning and embodied agent control), and several central claims—that the explicit value function enables better credit assignment, that the value function's advantage over implicit alternatives is causally responsible for beam search gains—are supported by qualitative case studies and indirect evidence rather than direct measurement. The theoretical framework is elegant and the empirical results are consistently positive, but the paper opens a research program rather than closing one. The diagnostic framework (soft Bellman consistency as a lens for understanding why methods succeed or fail on reasoning) is likely to be more enduring than any specific implementation detail of OREO's training objective.

Follow-Up Research This Work Enables

Direct comparison of value-guided beam search against best-of-N sampling at matched inference budgets. The paper demonstrates that OREO's value function enables beam search that improves with beam width (Figure 3), but never compares this to simpler test-time compute strategies like generating N complete solutions and selecting the best via the same value function or majority voting. A crucial follow-up would measure whether the beam search procedure specifically adds value beyond simply spending more compute. Concretely: train OREO on a 7B model, then evaluate at three budget levels (e.g., 4×, 16×, 64× the cost of greedy decoding) using (a) value-guided beam search, (b) best-of-N sampling with value function scoring, (c) best-of-N with majority voting, and (d) best-of-N with the policy's own probabilities as scores. The comparison would reveal whether the value function's per-step discriminative ability (demonstrated qualitatively in Figure 1) translates to better compute allocation at test time, or whether the gains come primarily from the policy improvement and any reasonable selection strategy would work. This experiment would directly address the open question left by Section 5.4: is value-guided beam search compute-optimal, or would a practitioner be better off spending the same FLOPs on parallel independent sampling?

Quantitative measurement of credit assignment quality through step-level error detection. The paper argues that OREO enables fine-grained credit assignment—the value function learns to identify which specific steps introduce errors—but the evidence is qualitative (the case studies in Figure 1). A strong follow-up would construct a benchmark for step-level error detection: take reasoning trajectories from a held-out set, have human annotators or automated methods (e.g., checking each arithmetic operation against a symbolic solver) label each step as correct or incorrect, and measure how well OREO's value function (via the advantage estimate A_phi = V_phi(s_{t+1}) − V_phi(s_t)) discriminates correct from incorrect steps, using AUROC or average precision as the metric. Compare against (a) the implicit value function from the policy's log-ratios (A_theta, Equation 12), (b) a PRM trained via Monte Carlo rollouts on the same base model, and (c) a simple baseline like the policy's own sequence-level probability drop. This would transform the paper's claim about credit assignment from an inference ("OREO enables better credit assignment because it uses a per-step objective") to a measurement ("OREO's value function achieves AUROC of X on step-level error detection, compared to Y for the implicit value function"). It would also reveal whether the softmax bottleneck (Yang et al., 2017) argument—that explicit value functions have richer representational capacity—holds up under rigorous comparison.

Scaling OREO to 70B+ models to test the softmax bottleneck prediction. The paper's argument for why explicit value functions outperform implicit ones rests on the softmax bottleneck: the policy's output distribution is constrained by a single linear projection from a shared hidden state, while the value function processes the full sequence through the entire transformer. This predicts that the gap between explicit and implicit value functions should grow with model scale, because larger models have proportionally more representational capacity in their hidden states, making the bottleneck of a single linear head more binding. A direct test: train OREO and DPO on the same base model at 7B, 13B, and 70B scales (using Qwen-2.5 or LLaMA-3 families), measure the step-level error detection AUROC for both the explicit value function (OREO) and the implicit value function (derived from the DPO policy's log-ratios), and test whether the AUROC gap widens with scale. A secondary measurement: compare beam search performance using each value function as the scoring heuristic at matched beam widths. If the gap does not widen with scale—or worse, if it narrows because larger models have better-calibrated probabilities that serve as adequate implicit value functions—the softmax bottleneck argument would be weakened and the practical case for explicit value functions at production scales would diminish.

Combining OREO's value function with online exploration in an actor-critic framework. OREO is purely offline: it learns from a fixed dataset and cannot explore novel states. The paper's iterative extension (Section 4.3) is a step toward online learning—each iteration collects new data from the updated policy—but the exploration is limited to the policy's own greedy or sampled outputs, without any explicit exploration bonus or uncertainty-driven action selection. A natural extension would be to use OREO's value function as the critic in an online actor-critic setup: the policy generates rollouts in an environment (or against a reward model), the value function provides credit assignment and advantage estimates, and the policy is updated using a policy gradient (e.g., PPO or A2C) with the value function as the baseline. This would combine OREO's principled credit assignment with online RL's ability to explore and correct for distributional shift. The key measurement would be: does OREO-pretrained value function accelerate online RL convergence compared to (a) training a value function from scratch online, (b) using a Monte-Carlo-rollout PRM as the critic, or (c) using a simpler baseline like average reward? The paper's results suggest that OREO's value function provides better credit assignment than alternatives; whether this translates to faster online learning is an open empirical question.

Stress-testing iterative OREO beyond three iterations to find the saturation point. Figure 2 shows OREO improving over three iterations while rejection sampling saturates, but OREO's own gains decelerate (near-flat from iteration 2 to 3 on MATH). A crucial stress test would run iterative OREO for many more iterations—10, 20, or until performance plateaus—to determine whether the sustained improvement is a transient advantage (OREO reaches a higher asymptote than rejection sampling but still saturates) or a permanent one (OREO continues to improve as long as new data is collected). This is not merely academic: if OREO plateaus after 5 iterations, the practical benefit over rejection sampling is the difference in asymptote (perhaps 2-3 percentage points, as suggested by Figure 2), and the additional complexity of training a value function must be weighed against that modest permanent gain. If OREO continues to improve at iteration 20 with no sign of saturation, the method represents a fundamentally different scaling regime for self-improvement, and the investment in value function training is overwhelmingly justified. This experiment would also reveal whether distributional shift—the policy drifting into states the value function hasn't been trained on—eventually causes degradation, and whether techniques from offline RL (ensembles, pessimism, constrained policy updates) are needed to prevent it.

Adapting OREO to domains without clean terminal reward signals. The paper's mathematical derivation and experimental validation assume sparse but objective terminal rewards: a math answer is correct or incorrect, an agent task succeeds or fails. Many important reasoning domains lack this clean signal. Code generation has partial correctness (passing some unit tests but not others), scientific reasoning has uncertain or probabilistic conclusions, and open-ended planning has multi-dimensional quality criteria. A critical extension would adapt OREO to these settings by replacing the binary terminal reward R_t in {0, 1} with a continuous or multi-dimensional reward signal. For code generation, R_t could be the fraction of unit tests passed; for scientific reasoning, it could be a learned reward model's assessment of argument quality; for dialogue, it could be human preference scores. The key question is whether the soft Bellman Equation framework—which assumes a well-defined scalar reward—degrades gracefully when rewards are noisy, continuous, or multi-dimensional, or whether the value function's regression target becomes too noisy for effective credit assignment. A concrete experiment: train OREO on a code generation dataset (e.g., MBPP or HumanEval) where each solution's reward is the fraction of test cases passed, and measure whether OREO outperforms rejection sampling and DPO as it does on MATH/GSM8K. A negative result here (OREO underperforms when rewards are continuous) would reveal a boundary condition on the method's applicability that is currently unknown.

Practical Applications and Downstream Use Cases

Cost-efficient fine-tuning for small research teams and individual practitioners. The paper's central practical promise is that OREO achieves better reasoning performance than existing offline methods without requiring the expensive infrastructure of online RL (PPO). For a small team with access to a single GPU node, the workflow is: (1) generate N candidate solutions per training problem using the base SFT model (the paper uses 10 for 1.5B models, 16 for 7B models), (2) verify correctness automatically (math grading scripts, unit tests, environment success signals), (3) run OREO training for 1-3 epochs using the provided open-source code. The result, per Table 1, is a 1.5B model that achieves 52.5% on MATH—competitive with much larger models from just a year ago—and a 7B model that achieves 85.9% on GSM8K. The value function produced as a byproduct then enables test-time beam search that provides up to an additional 17.9% relative improvement (Figure 3) at the cost of increased inference compute. This workflow requires no human annotation, no pairwise preference judgments, no reward model training, and no environment interaction during training—making it accessible to anyone who can generate and grade candidate solutions. The total cost is dominated by the initial data generation (75,000+ candidate solutions for MATH at 7,500 problems × 10 samples), which is a one-time upfront investment that can be amortized across multiple training runs, ablations, and hyperparameter sweeps.

Deploying value-guided search as a drop-in inference-time upgrade for existing fine-tuned models. The paper's finding that the explicit value function enables monotonic accuracy improvements with beam width (Figure 3) has an immediate practical implication: any model trained with OREO comes with a built-in verifier that can scale test-time performance without additional training. For a production deployment where latency and cost constraints vary by query (e.g., a math tutoring system that can spend more time on harder problems, or an agent system where some decisions are more consequential than others), the value function provides a knob for dynamically trading off compute against accuracy. Easy queries can use greedy decoding (B=1); moderate queries can use light beam search (B=3, providing perhaps a 5-8% relative boost based on Figure 3's trajectory); hard queries can use aggressive search (B=7, providing the full 17.9% boost). The value function is the same in all cases—no additional models, no separate PRM training pipeline, no complex inference orchestration. This is a substantially simpler deployment story than prior PRM-based search methods (Lightman et al., 2023; Snell et al., 2024), which require training and maintaining a separate verifier model that may drift from the policy's distribution over time.

Iterative self-improvement pipelines that don't hit early ceilings. The iterative training results (Figure 2) demonstrate that OREO continues to improve over three iterations while rejection sampling saturates. For organizations building automated self-improvement pipelines—where a model generates training data, filters it, and retrains itself in a loop—this finding has direct architectural implications: the standard practice of discarding failures is likely the bottleneck, not data quantity or diversity. Switching the training objective from success-only fine-tuning to OREO's joint policy-value optimization could extract more value from each iteration's generated data, sustaining improvement for more cycles before hitting diminishing returns. The concrete benefit, based on Figure 2, is approximately 2-3 additional percentage points of accuracy on MATH by iteration 3 compared to rejection sampling, with the gap potentially widening further in later iterations (though this is untested). The cost is the added complexity of training a value model alongside the policy—approximately double the GPU memory for the 1.5B case (both models fully fine-tuned) or a more modest increase for the 7B case (value model trained with LoRA). For organizations already running iterative self-improvement at scale, this cost may be justified by the sustained improvement trajectory.

When to Prefer This Method

The paper does not provide explicit decision rules for choosing OREO over named alternatives, nor does it include head-to-head comparisons against online RL (PPO) or PRM-based systems that would enable precise trade-off quantification. The following decision guidance is therefore synthesized from the paper's theoretical claims and empirical results, not stated by the authors as explicit recommendations.

Prefer OREO over DPO or KTO when:

  • The task has objective per-instance success/failure labels rather than requiring comparative human preference judgments (Section 3.3: "the BT model is reasonable for a general dialogue system... [but] unnecessary for multi-step reasoning tasks where a ground-truth reward exists").
  • Credit assignment matters—errors typically occur at identifiable steps within a longer correct reasoning chain, and per-step signal is needed to distinguish good steps within bad trajectories from the specific steps that caused the failure (Section 3.3, Section 5.1: the 3.1-point drop from token-level to response-level OREO on MATH quantifies the value of this granularity).
  • The dataset contains a mix of successes and failures (rather than only successes, in which case rejection sampling is simpler and nearly as effective; Table 1: rejection sampling achieves 50.3 vs. OREO's 52.5 on MATH for the 1.5B model).
  • You want a value function for test-time search without the engineering effort of training a separate PRM (Section 4.4, Figure 3: OREO provides this "for free").

Prefer rejection sampling over OREO when:

  • Simplicity of implementation is the primary concern—rejection sampling requires only standard supervised fine-tuning on filtered data, with no value model, no Bellman consistency loss, and no joint optimization.
  • The task is easy enough that the SFT model's successes already cover the space of good behavior adequately, and failures provide little additional signal (Table 1: on GSM8K with the 7B model, rejection sampling achieves 83.6 vs. OREO's 85.9—the 2.3-point gap may not justify the additional complexity for some deployments).
  • Only one iteration of training is planned, and the base model's success rate is high enough that failure trajectories are rare or uninformative (Figure 2: the gap between OREO and rejection sampling is smallest at iteration 1 and widens in later iterations).

Prefer online RL (PPO) over OREO when:

  • The offline dataset has limited coverage of the state space—the base model makes systematic errors that never produce correct trajectories, so no amount of offline learning from failures can discover what the correct action would have been (online exploration could discover fixes).
  • Maximizing absolute performance is more important than minimizing training cost, and the infrastructure for online RL is available (the paper provides no online RL baseline, so the performance gap is unknown, but the RLHF literature generally finds online methods outperform offline ones when budget is unconstrained).
  • The task requires adapting to a changing environment or reward function during training (OREO is limited to the fixed offline dataset).

These preferences are provisional and should be revisited as the research program this paper opens—scaling OREO to larger models, comparing against online RL, testing on broader task families—produces more comprehensive evidence. The paper's contribution is establishing that the soft Bellman Equation framework is a viable and principled alternative to both preference-based methods and heuristic PRM training, not that OREO in its current form is universally optimal across all settings.