Understanding the performance gap between online and offline alignment algorithms

ArXiv: 2405.08448

🎯 Pitch

Online RLHF algorithms consistently beat offline methods like DPO, but not for the reasons you’d expect—better data coverage or quality can’t close the gap. The real driver is on-policy sampling, and the only way offline training catches up is when its dataset is generated near the policy’s own current distribution.

---

1. Executive Summary

This paper systematically investigates why online RLHF algorithms consistently outperform offline alignment methods such as DPO and IPO, using T5X encoder-decoder models trained on four preference datasets (OpenAI summarization, Anthropic helpfulness, Anthropic harmlessness, and Chat arena). Through hypothesis-driven ablations, the authors test and largely invalidate intuitive explanations—including data coverage, dataset quality, and classification accuracy—establishing instead that the core mechanism is on-policy sampling (where the policy generates its own training responses and receives feedback from a proxy preference model, versus training on a static pre-collected dataset). The key finding is a clear and persistent performance gap that online algorithms achieve superior KL-budgeted peak win rates (e.g., ~5–15 percentage points higher on Anthropic helpfulness), with the gap narrowing only when the offline dataset is made more on-policy by including responses generated near the SFT policy's distribution—establishing that offline algorithms can partially recover online performance only when the training data distribution closely matches the policy's own evolving output distribution.

2. Context and Motivation

The Core Problem: The Role of On-Policy Sampling in LLM Alignment

The central question this paper investigates is deceptively straightforward yet carries profound practical implications: is online reinforcement learning truly necessary for AI alignment, or can offline methods suffice? The question emerges from a tension in the rapidly evolving landscape of language model alignment techniques. On one side stands the canonical RLHF framework (Christiano et al., 2017; Ouyang et al., 2022), which involves training a reward model from human preference data, then using online reinforcement learning—where the policy generates its own responses, receives feedback from the learned reward model, and iteratively improves. On the other side, a wave of offline alignment algorithms, most notably Direct Preference Optimization (DPO; Rafailov et al., 2023) and Identity Preference Optimization (IPO; Azar et al., 2023), has demonstrated impressive empirical results while eliminating the need for explicit reward modeling, on-policy sampling, or value function learning. These offline methods directly optimize the policy against a static dataset of pairwise preferences.

This question matters enormously for several practical reasons the paper implies throughout (Section 1):

• Cost and complexity. Offline algorithms are substantially simpler to implement and compute: they require no reward model training, no on-policy sampling infrastructure, and no separate critic networks. If offline methods truly match online performance, the path to alignment becomes dramatically simpler and cheaper.
• Scalability of alignment pipelines. Online algorithms require maintaining generation servers, reward model inference, and coordination between training and sampling loops—infrastructure that becomes increasingly expensive as models scale. Understanding whether this complexity is functionally necessary or merely historical convention has direct implications for how organizations architect their alignment pipelines.
• Conceptual foundations of alignment. At a deeper level, the answer speaks to whether alignment is fundamentally about learning from a fixed corpus of human preferences (a supervised learning problem) or whether it requires iterative, interactive refinement (a reinforcement learning problem). The theoretical commitments of offline methods suggest the former; empirical evidence for online methods would suggest the latter.

The Unfair Comparison Problem

A major obstacle to answering this question cleanly is that online and offline algorithms differ along multiple dimensions simultaneously, making it difficult to isolate the effect of on-policy sampling. The paper identifies several confounds that prior work failed to disentangle:

Different optimization budgets. Online algorithms typically involve many more gradient updates and more total computation than offline algorithms—they sample responses, query a trained reward model, and run additional forward passes. A naive comparison at "convergence" might attribute to online sampling what is actually just more computation. The paper adopts the framework from Gao et al. (2023) to calibrate comparisons using KL divergence from the supervised fine-tuned (SFT) reference policy ($\mathbb{KL}(\pi_\theta, \pi_{\text{sft}})$) as a unified measure of optimization budget. This metric tracks how far the optimized policy has drifted from its starting point, providing a hyperparameter-agnostic way to compare different algorithms on a common scale.

Different underlying algorithms, not just different sampling. Online RLHF typically uses policy-gradient methods like PPO (Schulman et al., 2017), which involve value functions, advantage estimation, and clipping. Offline methods like DPO use contrastive losses with no explicit reward modeling. Any performance difference could plausibly be attributed to these algorithmic differences rather than the online-versus-offline distinction. The paper's key methodological move is to use the IPO loss for both online and offline variants (Section 2). IPO can be instantiated as an online algorithm (where responses are sampled from the current policy $\pi_\theta$ and labeled by a trained proxy preference model) or as an offline algorithm (where the same loss is applied to a static dataset), sharing identical hyperparameters, architecture, and optimization procedure. The only difference is the source of the training responses—the sampling distribution $\mu$ in Equation (1):

$\min_\theta \mathbb{E}_{x \sim p, (y_w, y_l) \sim \mu} \left[ \left( \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{sft}}(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{sft}}(y_l|x)} - \frac{\beta}{2} \right)^2 \right]$

For online, $\mu = \pi_\theta$; for offline, $\mu$ is a fixed dataset distribution. This clean experimental design isolates on-policy sampling as the single causal variable.

Why Existing RL Theory Doesn't Settle This

A reader familiar with reinforcement learning might object: isn't it obvious that online methods outperform offline ones? The literature on offline RL extensively documents the challenges of distribution shift, overestimation bias, and poor generalization from static datasets (Levine et al., 2020; Fujimoto et al., 2019). The paper directly addresses this objection (Section 1, "Isn't it obvious..."):

The reward model is learned from the same data. In RLHF, the online algorithm's reward signal comes from a learned proxy preference model, which itself is trained on the same static pairwise dataset that the offline algorithm uses directly. This is fundamentally different from standard online RL, where the agent typically has access to a ground-truth reward function during interaction. If the reward model is the bottleneck—as argued by Gao et al. (2023)—then online interaction might not provide meaningful advantage, since the feedback signal is still constrained by a static dataset of human preferences. The offline algorithm might just be a more direct way to optimize the same signal.

The bandit formulation eliminates temporal dependence. RLHF is typically formulated as a contextual bandit problem—each prompt-response pair is evaluated independently, with no sequential dependence between actions. This eliminates many of the compounding error issues that plague offline RL (where errors in value estimation propagate through multi-step transitions). Additionally, all the algorithms studied here (IPO, DPO) include strong KL regularization toward the SFT policy, which serves as a form of implicit conservatism that should mitigate overestimation. These properties could plausibly reduce the online-offline gap to negligible levels.

Offline methods are theoretically grounded. DPO and IPO were derived from first principles as closed-form solutions to the RLHF objective, with provable optimization properties under the assumption that the offline dataset has full support over response space (Rafailov et al., 2023; Azar et al., 2023). If the theoretical assumptions hold approximately, offline methods should recover the optimal policy.

The Observation That Motivated This Study

Despite these reasons suggesting offline methods might work well, the authors' opening experiment (Figure 1) reveals a clear and systematic gap. Across four datasets—OpenAI summarization (Stiennon et al., 2020), Anthropic helpfulness and harmlessness (Bai et al., 2022), and Chat arena side-by-side (Chiang et al., 2024)—online IPO consistently achieves both:

1. Higher performance at matched KL budgets: For a fixed drift from the SFT policy, online policies achieve better win rates against the golden policy baseline.
2. Higher peak performance: Online policies' maximum achievable win rate (before over-optimization causes degradation) is substantially higher than offline policies' peak—on the order of 5–15 percentage points depending on the dataset.

The phenomenon is robust: it holds across all four tasks, over a wide range of hyperparameters (learning rates, regularization coefficients $\beta$, and training steps), and follows a classic Goodhart's law curve where proxy optimization eventually diverges from true performance. The gap is not anecdotal or small; it represents a fundamental difference in what these algorithms can achieve.

Where Prior Work Falls Short

The paper positions itself relative to several strands of prior research, identifying specific limitations that motivate the hypothesis-testing approach:

Over-optimization studies (Gao et al., 2023). This prior work established the scaling laws of reward model over-optimization—showing that larger reward models delay but do not eliminate over-optimization. However, it focused exclusively on online algorithms with policy-gradient methods and did not compare online versus offline paradigms. The current paper extends this framework by providing the first comparative analysis under a synthetic setup and introducing preference models (scalar pairwise predictions) alongside point-wise reward models.

Offline algorithm derivations (Rafailov et al., 2023; Azar et al., 2023). These works derived DPO and IPO from theoretical principles, proving that under full data coverage assumptions, the optimal offline policy equals the optimal RLHF policy. The empirical performance of these algorithms has been impressive in practice (Jiang et al., 2023; Tunstall et al., 2023). However, the theoretical guarantees rely on assumptions—particularly full support of the behavior policy—that may not hold in practice. The current paper provides an explicit empirical case where theory fails (Section 5.2, Appendix H): an offline algorithm trained on data from a high-performance policy $\mu$ cannot recover $\mu$'s performance, let alone improve upon it, because the theoretical optimal policy $\pi^* \propto \pi_{\text{sft}} \exp(\beta^{-1} p(y \succ \mu))$ requires evaluating $p(y \succ \mu)$ for responses outside $\mu$'s effective support.

Online preference optimization (Calandriello et al., 2024; Guo et al., 2024). Recent work has explored online variants of contrastive losses, demonstrating that iteratively sampling from the current policy and updating improves over offline baselines. However, these studies did not systematically investigate why online methods outperform offline ones—they observed the phenomenon without explaining its causal mechanisms. The current paper takes the next step of controlled ablation to isolate specific hypotheses.

The tandem effect in RL (Ostrovski et al., 2021). This prior work demonstrated that in standard deep RL, an agent passively observing the same data stream as an actively learning agent performs substantially worse—a phenomenon termed the "tandem effect." The current paper adapts this experimental design to RLHF (Figure 3, Figure 4) but notes that RLHF's structure (learned reward models, bandit formulation, KL regularization) could plausibly mitigate or eliminate this effect. The fact that a significant gap remains despite these structural differences suggests the problem is more fundamental than previously appreciated.

Concurrent work on online-vs-offline comparison (Xu et al., 2024; Tajwar et al., 2024). Two papers released contemporaneously also study the online-offline gap. Xu et al. focus on implementation-level improvements to PPO that close the gap with DPO, while Tajwar et al. identify conditions (such as when the reward model's peak is within offline distribution) where offline methods perform competitively. The current paper's distinctive contribution is the systematic hypothesis-testing approach: rather than proposing fixes or characterizing boundary conditions, it asks why the gap exists in the first place, testing five specific mechanistic explanations.

The Hypothesis-Testing Methodology

The paper's investigative structure represents a methodological contribution in itself. Rather than designing a new algorithm or reporting benchmark results, the authors structure their investigation around five falsifiable hypotheses, each capturing a plausible mechanistic explanation for the online-offline gap (Section 3):

1. Data coverage: Online algorithms see more diverse responses (sampled from evolving policies) than the static offline dataset.
2. Dataset quality: The offline preference dataset contains responses from sub-optimal policies; higher-quality responses would close the gap.
3. Classification accuracy: Online algorithms benefit from a separately trained proxy preference model that classifies response pairs more accurately than an offline policy can.
4. Contrastive loss pathologies: The performance gap is an artifact of contrastive loss functions (like IPO) rather than a fundamental sampling issue.
5. Model scale: Larger policy networks might bridge the gap, making the online-offline distinction irrelevant at scale.

Each hypothesis is tested through carefully controlled ablations, often using clever experimental designs (e.g., the shuffled online dataset $\mathcal{D}_{\text{online-shuffled}}$ in Section 5.1 that perfectly controls for data coverage while varying only presentation order). The fact that most hypotheses are rejected—with evidence that is often surprising or counterintuitive—makes the negative results themselves informative.

What This Paper Ultimately Argues

The paper's position, supported by the progressive elimination of alternative explanations, is that on-policy sampling is not merely helpful but fundamentally necessary for achieving strong generative performance in alignment. The key mechanism, elaborated through the classification-versus-generation analysis (Section 5.3, Figure 8), is that:

• Offline training improves discriminative ability on a static dataset (the policy gets better at classifying which of two responses is preferred), but this improvement does not translate into better generation quality.
• Online training improves generative ability by continuously shifting the sampling distribution toward higher-quality responses, even though the policy never becomes a good classifier on any fixed dataset.

This is not a story about more data, better data, or more computation—it is about a qualitatively different learning dynamic induced by sampling from the evolving policy. The implication is that offline methods face a fundamental challenge: they optimize a proxy objective (pairwise classification accuracy) that is poorly correlated with the true objective (generating responses preferred by humans), and on-policy sampling is the mechanism that aligns these two objectives.

3. Technical Approach

This is primarily an empirical analysis paper whose core idea is that on-policy sampling triggers a fundamentally different learning dynamic than training on static data, and that this dynamic—not data coverage, quality, classification accuracy, or model scale—is the primary driver of the online-offline performance gap.

3.1 Reader Orientation

The paper constructs a controlled experimental framework for comparing online and offline RLHF algorithms where the only difference is the source of training responses—the policy's own evolving output distribution versus a static pre-collected dataset. This framework isolates on-policy sampling as a single causal variable while holding the loss function, model architecture, hyperparameters, compute budget, and initial checkpoint constant, enabling the hypothesis-testing ablation approach.

The system solves the problem of fair comparison between online and offline alignment algorithms—which normally differ in loss function, architecture, reward modeling, implementation, and compute—by unifying them under the Identifiable Preference Optimization (IPO) loss with a shared KL divergence budget metric, producing a clean experimental platform where the effect of sampling can be measured in isolation.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework consists of five major components connected in two parallel pipelines (Figure 2 in the paper):

1. Golden preference model (XXL T5X, 11B parameters) — trained once on the initial pairwise preference dataset to emulate ground-truth human preferences. It serves as the evaluation oracle and dataset re-labeler for all downstream experiments.

2. Relabeled golden dataset $\mathcal{D}_{\text{golden}}$ — the initial preference dataset re-scored by the golden preference model, creating a consistent ground-truth signal for training both online proxy reward models and offline policies.

3. Online pipeline (upper path in Figure 2) — a proxy preference model (Large T5X, 770M parameters) is trained on $\mathcal{D}_{\text{golden}}$, then used to label pairs of responses sampled on-policy from the current policy $\pi_\theta$ during iterative training. The policy and proxy model interact in a loop: policy generates responses $\rightarrow$ proxy model ranks them $\rightarrow$ IPO loss updates policy $\rightarrow$ updated policy generates new responses.

4. Offline pipeline (lower path in Figure 2) — the same IPO loss is applied directly to $\mathcal{D}_{\text{golden}}$ (or ablated variants), treating the static dataset's winning and losing responses as fixed targets without any generation or proxy model interaction.

5. Evaluation infrastructure — a fixed golden policy baseline (trained online with access to the golden preference model, giving it privileged information) serves as the comparison target. All evaluated policies generate responses on 2048 deterministically sub-sampled prompts, and the golden preference model judges side-by-side win rates.

Information flows as follows: an initial preference dataset enters the system $\rightarrow$ the golden preference model re-labels it to create $\mathcal{D}_{\text{golden}}$ $\rightarrow$ the online path trains a proxy preference model from this dataset and then iteratively samples from the policy, labels with the proxy model, and updates the policy $\rightarrow$ the offline path directly optimizes the policy on $\mathcal{D}_{\text{golden}}$ without proxy model interaction $\rightarrow$ both policies are evaluated against the golden policy baseline using the golden preference model as judge $\rightarrow$ KL divergence from the SFT policy tracks optimization budget across all experiments.

3.3 Roadmap for the Deep Dive

• First, the IPO loss function and its online/offline instantiation — since this is the single algorithmic primitive used across all experiments, understanding how Equation (1) maps to different sampling distributions $\mu$ is the foundation for everything that follows.

• Second, the experimental infrastructure stack — the SFT baseline training, the golden preference model construction, the proxy preference model training and inference details (including the critical positional bias mitigation in Equation 5), and the KL divergence estimation procedure, because these shared components must be understood before any ablation result can be interpreted.

• Third, the controlled over-optimization framework — how KL divergence serves as a unified budget metric (Section 4.1), how hyperparameter sweeps populate the trade-off curves in Figure 1, and why this framework enables fair comparison across algorithms that would otherwise have incommensurable compute footprints.

• Fourth, the tandem dataset construction (Figures 3–4) — the critical experimental design where online-generated data is saved and replayed offline, because this is the methodological innovation that cleanly isolates the effect of sampling order from data coverage.

• Fifth, the classification-vs-generation analysis machinery — how policies are used as pairwise classifiers (Appendix E), how classification accuracy is measured against the golden preference model, and how the discriminative-generative trade-off is tracked across training (Figure 8), because this provides the key mechanistic insight.

• Sixth, the dataset ablation design (Figure 12) — the construction of datasets $\mathcal{D}_{\text{sft vs. 800}}$, $\mathcal{D}_{\text{800 vs. 4k}}$, and $\mathcal{D}_{\text{4k vs. 4k}}$ with systematically varied properties (proximity to SFT, response differentiation, absolute quality), because this directly tests what offline data characteristics enable recovery of online performance.

3.4 Detailed, Sentence-Based Technical Breakdown

The IPO Loss Function and Online/Offline Instantiation

The paper unifies all experiments under the Identity Preference Optimization loss from Azar et al. (2023). Unlike DPO, which uses a logistic log-loss, IPO uses a squared error between the policy's log-ratio and a target constant. The loss is reproduced in Equation (1):

$\min_\theta \mathbb{E}_{x \sim p, (y_w, y_l) \sim \mu} \left[ \left( \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{sft}}(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{sft}}(y_l|x)} - \frac{\beta}{2} \right)^2 \right]$

where $x$ is a prompt drawn from the prompt distribution $p$, $(y_w, y_l)$ is a pair of winning and losing responses drawn from the sampling distribution $\mu$, $\pi_\theta$ is the policy being optimized, $\pi_{\text{sft}}$ is the frozen supervised fine-tuned reference policy, and $\beta > 0$ is a scalar hyperparameter controlling regularization strength (larger $\beta$ means stronger KL regularization toward $\pi_{\text{sft}}$, constraining how much the policy can change).

What it computes: for each training pair, the loss measures the squared distance between (a) the policy's log-ratio difference—how much more likely it makes the winning response versus the losing response relative to the SFT baseline—and (b) the target value $\beta/2$. The policy is penalized when this difference deviates from $\beta/2$ in either direction: too small (insufficient preference for the winner) or too large (excessive preference that risks reward hacking). The expectation is taken over prompts and response pairs.

Why this form: the squared loss shape confers a specific property that the logistic loss in DPO lacks: it penalizes both under-fitting and over-fitting symmetrically around the target $\beta/2$. If the policy makes the winning response far more likely than the target specifies (large positive log-ratio), the loss increases quadratically, providing natural regularization against extreme probability shifts. This is particularly important for online training where the sampling distribution shifts over time—the squared penalty prevents the policy from exploiting transient features in the proxy preference signal. Additionally, $\beta/2$ serves as an interpretable knob: when $\beta$ is small, the target is close to zero and the policy barely differentiates winner from loser; when $\beta$ is large, the policy can place substantially more weight on winning responses.

Online versus offline instantiation: the only structural difference between online and offline variants is the sampling distribution $\mu$:

• Online IPO: $\mu = \pi_\theta$, meaning that at each training step, the current policy $\pi_\theta$ generates two responses for each prompt, the proxy preference model (a separately trained Large T5X model that takes $(x, y_1, y_2)$ and predicts a scalar preference probability) labels which response is preferred, and this freshly generated and labeled pair becomes the training data. The sampling distribution co-evolves with the policy—as $\pi_\theta$ shifts probability mass toward better responses, the training distribution shifts accordingly, creating a feedback loop.

• Offline IPO: $\mu$ is a fixed dataset distribution—specifically $\mathcal{D}_{\text{golden}}$, the golden preference model's re-labeled version of the initial pairwise preference dataset. The policy sees the same fixed pairs throughout training, with no interaction with a proxy model and no response generation. The prompts are drawn uniformly from the dataset, and the winning-losing pairs are pre-determined.

This design choice is what makes the paper's comparison possible: because the loss function, architecture, hyperparameters, and initialization are identical, any performance difference can be causally attributed to the sampling process—whether the data comes from the policy's own current distribution or from a static historical distribution.

Design rationale for choosing IPO over DPO or PPO: the paper explicitly avoids policy-gradient methods like PPO because "it is not clear how to optimize the PPO loss offline since the algorithm by design uses on-policy samples" (Section 2). DPO was originally derived as an offline equivalent of policy-gradient RLHF, making it a natural candidate. However, IPO is preferred because its squared loss eliminates the need to tune the logistic function's saturation behavior, and more practically, because recent work (Calandriello et al., 2024) had already established a working online IPO recipe. The paper notes that conclusions should transfer to DPO and other contrastive losses "due to similar empirical patterns" (Section 2, citing Tang et al., 2024), though this is not experimentally verified within the paper itself.

Hyperparameter baseline: the default configuration uses learning rate $1 \times 10^{-5}$, regularization coefficient $\beta = 0.1$, 4,000 gradient steps, and batch size 32 for Large models (16 for XL, 8 for XXL). These values were "extensively tuned on the OpenAI summarization task" (Section 4.1) and applied without further tuning to the other three datasets, though the authors note that Anthropic harmlessness "requires generally fewer steps to learn." For offline algorithm hyperparameter sweeps, learning rate is varied across $\{3 \times 10^{-6}, 1 \times 10^{-5}, 3 \times 10^{-5}\}$, $\beta$ across $\{0.1, 0.5, 1.0\}$, and training steps across $\{4\text{k}, 20\text{k}\}$. These sweeps are what populate the scatter of offline points in Figure 1—each point represents a policy checkpoint from a specific hyperparameter combination, providing broad coverage of the KL divergence vs. performance space.

The Supervised Fine-Tuning (SFT) Baseline Construction

All RLHF experiments start from a supervised fine-tuned checkpoint rather than a base pretrained model. The SFT stage is not designed to maximize downstream performance; rather, it serves a functional purpose: ensuring that "from the beginning of RLHF, the policy can already produce reasonable samples with a wide coverage" and that the starting policy distribution is "not too out-of-distribution from the pairwise preference dataset that the preference models are trained on" (Appendix C).

The training procedure is deliberately simple and resource-efficient:

Data construction: the authors repurpose all RLHF pairwise preference datasets into a single SFT dataset by pooling together responses from all four downstream tasks. For each prompt in the original preference datasets, both the winning response $y_w$ and losing response $y_l$ are treated as SFT targets—no filtering, no quality selection. This means the SFT policy is trained on responses of mixed quality, approximately matching the distribution it will encounter during offline RLHF (since $\mathcal{D}_{\text{golden}}$ is constructed from the same response pool).

Training configuration for Large models: batch size 128, learning rate $3 \times 10^{-5}$, training for 4,000 steps (roughly 3–4 epochs over the pooled dataset), with a context length of 1024 tokens and target length of 128 tokens. The choice of 4,000 steps matches the default RLHF training horizon, creating symmetry between the SFT and alignment phases.

Scaling for larger models: for XL (3B parameters) and XXL (11B parameters) models, training runs for 8,000 steps at learning rate $10^{-5}$ with batch sizes of 16 and 8 respectively—roughly 1 epoch for XL and half an epoch for XXL. The paper notes that "this amount of finetuning seems enough to improve the overall performance as the policy network scales up" (Appendix C), suggesting that the SFT stage's role diminishes with model capacity.

Why this matters for the experimental design: by using the same training data distribution for SFT as for offline RLHF, the paper eliminates a potential confound where offline algorithms might fail simply because their training distribution is far from the starting policy. The SFT policy's output distribution is, by construction, close to the distribution that generated $\mathcal{D}_{\text{golden}}$, which should theoretically make offline learning easier. If offline methods still underperform relative to online methods despite this favorable initialization, the problem is more fundamental than distribution mismatch at initialization.

The Golden Preference Model Construction and Usage

The golden preference model serves as both the ground-truth oracle for evaluation and the data re-labeler that creates $\mathcal{D}_{\text{golden}}$. Its role is central to the experimental design because it provides a consistent, automated signal that emulates human judgment while eliminating the noise, cost, and availability limitations of actual human raters.

Architecture and training: an XXL T5X encoder-decoder model (approximately 11 billion parameters) is trained as a pairwise preference classifier. Given a tuple $(x, y_1, y_2)$—a prompt and two responses—the model outputs a scalar prediction $r_\theta(x, y_1, y_2) \in [0, 1]$ representing the probability that $y_1$ is preferred to $y_2$. Training uses the initial pairwise preference dataset (with original human or dataset-curated labels) and maximizes likelihood via the loss in Equation (4):

$\max_\theta \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log r_\theta(x, y_w, y_l) \right]$

where the expectation is over the training set, and the model predicts the probability that the first response shown is the winner. The global minimizer of this objective is the true preference probability $p(y_1 \succ y_2 | x)$.

Positional bias mitigation (Equation 5): the loss in Equation (4) creates a vulnerability: the model can learn to systematically prefer whichever response is placed in the first position, regardless of content. This is a well-known issue in preference modeling. To address it, the training objective explicitly randomizes response ordering and symmetrizes the loss:

$\max_\theta \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \frac{1}{2}\log r_\theta(x, y_w, y_l) + \frac{1}{2}\log(1 - r_\theta(x, y_l, y_w)) \right]$

The first term trains the model to correctly identify the winner when the winning response appears first; the second term trains it to correctly identify the winner when the winning response appears second (by maximizing $\log(1 - r_\theta(x, y_l, y_w))$, which penalizes predicting that the loser—now in first position—is preferred). This symmetrization approximately removes positional bias.

Inference-time debiasing: however, positional bias cannot be eliminated perfectly through training. The paper found that during online RLHF, the policy would "easily exploit such remaining bias if we just use $r_\theta(x, y_1, y_2)$ to determine the preference" (Appendix A.2). The solution is a debiasing step at inference: instead of using the raw prediction $r_\theta(x, y_1, y_2)$, preference is determined by the sign of the difference between the two possible orderings:

$r_\theta(x, y_1, y_2) - r_\theta(x, y_2, y_1)$

If this difference is positive, $y_1$ is judged preferred; if negative, $y_2$ is preferred. This effectively cancels out any additive positional bias that affects both orderings equally, since such bias would appear in both predictions and subtract out. This debiasing is applied during both training (when the proxy preference model labels online-generated pairs) and evaluation (when the golden preference model judges side-by-side comparisons).

Training details: the golden model trains on the original dataset labels with batch size 16, learning rate $10^{-4}$, and linear warmup for 1,000 steps. Context length is 1024 tokens, target length 128. Training terminates when validation accuracy plateaus (no strict rule enforced). The use of an XXL model (the largest in the T5X family) reflects the need for the golden signal to be as reliable as possible—it is the ground truth against which all policies are measured.

Why this matters for fair comparison: by using the golden model to re-label the entire training dataset into $\mathcal{D}_{\text{golden}}$, the paper ensures that both the online proxy preference model (Large, 770M parameters) and the offline policy are optimized against the same underlying preference signal. Any performance difference cannot be attributed to the offline dataset having noisier or systematically different labels. Additionally, using the golden model for evaluation means the performance metric is consistent and automated, enabling the large-scale hyperparameter sweeps required to populate the KL-performance trade-off curves.

Relationship to prior work: the preference model approach differs from the point-wise reward models used in Gao et al. (2023). A point-wise reward model assigns a scalar to each response individually, and preference between two responses is determined by comparing their scores (typically via a Bradley-Terry model: $p(y_1 \succ y_2) = \sigma(r(y_1) - r(y_2))$). A pairwise preference model directly takes both responses as input, making it more expressive—it can capture interactions between responses that a point-wise model cannot (e.g., detecting that two responses have similar content but different politeness levels). This expressiveness, however, comes with the positional bias complications described above.

The Proxy Preference Model for Online Training

The online algorithm's feedback signal comes from a proxy preference model—a Large T5X model (770M parameters, the same size as the policy) trained on $\mathcal{D}_{\text{golden}}$ using the same loss and debiasing procedure as the golden model. The proxy model is the weaker (smaller) surrogate that provides the online training signal; the golden model is the stronger (larger) oracle used only for evaluation and dataset construction.

Training configuration: the proxy model trains on $\mathcal{D}_{\text{golden}}$ with batch size 32, learning rate $10^{-4}$, linear warmup for 1,000 steps, context length 1024, target length 128. The batch size of 32 is deliberately chosen to match the policy optimization batch size, creating symmetry in how both models process data. The proxy model trains once before online RLHF begins and is held fixed throughout—it does not update as the policy distribution shifts. This is a critical design choice: the paper is studying over-optimization against a fixed proxy signal, not iterated model refinement.

Interaction with the online policy: during each training step of online IPO, the policy $\pi_\theta$ generates two responses for each prompt in a batch of 32, the proxy model scores the pair using the debiased inference procedure ($\text{sign}(r(x, y_1, y_2) - r(x, y_2, y_1))$), the winning response is designated $y_w$ and the losing response $y_l$, and the IPO loss (Equation 1) is computed using this freshly labeled pair. The proxy model never generates text; it only judges pairs generated by the policy.

Why separate proxy and golden models? This two-tier structure isolates the effect of the learned signal's quality. By keeping the proxy model fixed at 770M parameters while the golden model is 11B parameters, the paper creates a realistic scenario where the training signal is imperfect—as it always is in practice (human raters are noisy, reward models are approximations). The gap between proxy and golden judgments is what enables over-optimization, and the paper's KL-performance curves directly visualize this dynamic: performance initially improves as the policy exploits the proxy signal, then degrades as it begins to exploit proxy errors that the golden model would not endorse.

The KL Divergence Budget Framework

A central methodological contribution is the use of KL divergence as a unified optimization budget metric. This choice resolves a serious comparison problem: online and offline algorithms consume different amounts of compute per gradient step (online requires generation + proxy inference + policy update; offline requires only policy update), and even within each paradigm, different hyperparameter configurations produce policies at different distances from the SFT starting point. Some metric is needed that abstracts away from raw FLOPs and captures the effective amount of optimization that has occurred.

The KL divergence from the SFT policy is defined as:

$\mathbb{KL}(\pi_\theta, \pi_{\text{sft}}) = \mathbb{E}_{x \sim \rho, y \sim \pi_\theta(\cdot|x)} \left[ \log\frac{\pi_{\theta}(y|x)}{\pi_{\text{sft}}(y|x)} \right]$

where $\rho$ is the prompt distribution (uniform over the training set), $y \sim \pi_\theta(\cdot|x)$ indicates that responses are sampled from the current optimized policy (not from the dataset), and the expectation averages over both prompts and sampled responses.

What it computes: for each token position in each generated response, the KL divergence measures how much the current policy's next-token distribution differs from the SFT policy's next-token distribution. These per-token divergences are summed across the response length and averaged across prompts. A KL of 0 means the optimized policy is identical to the SFT policy; larger values indicate the policy has shifted further from its starting point. The unit is nats (natural log units), accumulated over all token positions.

Why this form: KL divergence is the natural metric for this setting because it appears directly in the theoretical RLHF objective as the regularization penalty. The regularized RLHF objective (from which IPO is derived) maximizes:

$\mathbb{E}_{y \sim \pi_\theta}[r(y)] - \beta \cdot \mathbb{KL}(\pi_\theta, \pi_{\text{sft}})$

where $r(y)$ is the (unknown) true reward. A policy that achieves high reward with low KL is preferred over one that achieves the same reward with higher KL, because the latter is deviating more from the trusted SFT baseline. By plotting performance against KL, Figure 1 reveals each algorithm's efficiency: how much performance improvement is extracted per unit of policy drift. Online algorithms achieve steeper initial slopes and higher peaks, meaning they extract more value per unit of drift.

Estimation procedure (Appendix D): computing the KL divergence requires an unbiased estimator because the expectation is over the policy's own samples. The paper's procedure:

1. Sub-sample 256 prompts from the training set (32 prompts for XL models to reduce inference cost).
2. For each prompt $x_i$, sample a response $y_i \sim \pi_\theta(\cdot|x_i)$ from the current policy. This response has $T_i$ tokens.
3. For each token position $t$ in the response, compute the full $V$-way categorical distributions (over the vocabulary of size $V$) from both $\pi_\theta(\cdot | x_i, y_{i, <t})$ and $\pi_{\text{sft}}(\cdot | x_i, y_{i, <t})$.
4. Compute the KL divergence between these two categorical distributions in closed form: $\sum_{v \in V} \pi_\theta(v) \log(\pi_\theta(v) / \pi_{\text{sft}}(v))$.
5. Average over all token positions in all responses.

This estimator is unbiased because the responses are sampled from $\pi_\theta$, satisfying the expectation's requirement $y \sim \pi_\theta(\cdot|x)$. The per-token decomposition uses the fact that KL divergence over sequences factorizes into sums of per-step KL divergences.

What the KL sweep looks like in practice: for online algorithms, the KL naturally increases over training steps as the policy drifts from the SFT baseline. For offline algorithms, different hyperparameter combinations produce different KL trajectories—$\beta$ controls the regularization strength (higher $\beta$ constrains the policy more, leading to lower KL), and learning rate controls how quickly the policy moves. By pooling checkpoints from many hyperparameter combinations, Figure 1 shows the achievable frontier: for any given KL budget, what is the best performance any offline configuration can achieve, and how does it compare to online?

Why this matters for hypothesis testing: the KL framework enables fair comparison because it abstracts away from implementation details. An offline policy trained for 20k steps with $\beta=0.1$ might achieve the same KL as an online policy trained for 4k steps with a different configuration. By comparing them at matched KL, we ask: given the same amount of policy drift, which approach produces a better policy? If online is consistently better, the advantage cannot be explained by simply having more optimization budget—it must be something about how that budget is spent.

The Tandem Dataset Construction: Isolating Sampling from Coverage

This is the paper's most methodologically innovative contribution—the experiment that cleanly tests Hypothesis 1 (data coverage) by creating an offline dataset with identical response diversity as the online algorithm's training stream, differing only in presentation order. The design is adapted from Ostrovski et al. (2021), who studied the "tandem effect" in standard deep RL—two agents observing the same data stream, one actively generating it and one passively learning.

Construction of $\mathcal{D}_{\text{online}}$: the online algorithm is run normally with the proxy preference model on prompts from $\mathcal{D}_{\text{golden}}$. During this run, every batch of training data is saved—the prompts, the two responses sampled from $\pi_\theta$, and the proxy model's preference label. Each batch has $B = 32$ examples, indexed by step $t$. The result is a stream of data $(x_t, y_t^{(w)}, y_t^{(l)})_{t=1}^T$ preserving the exact temporal order in which the online algorithm observed and trained on these examples. This dataset is called $\mathcal{D}_{\text{online}}$.

Equivalence check: running offline IPO on $\mathcal{D}_{\text{online}}$ with the same initialization and hyperparameters should be mathematically equivalent to the original online run, because both processes start from the same SFT checkpoint, see the same sequence of batches in the same order, and apply the same updates. The paper verified this equivalence empirically (Appendix F), confirming that "the offline experiment with $\mathcal{D}_{\text{online}}$ almost perfectly matches the online experiment, in terms of the training statistics up to $\sim 0.1\%$ throughout training." This verification step is non-trivial: the authors needed to ensure token-level match of the data stream, since "even slight mismatch would cause drastic divergence in the learning curves." Sources of mismatch include string-token inconsistencies and missing batches due to compute system preemptions. This sensitivity itself is evidence of the tandem effect's strength—the learning trajectory is fragile with respect to the exact data ordering.

Construction of $\mathcal{D}_{\text{online-shuffled}}$: the critical ablative dataset is created by taking $\mathcal{D}_{\text{online}}$ and applying uniform random shuffling across all batches. The result, $\mathcal{D}_{\text{online-shuffled}}$, contains exactly the same responses, prompts, and preference labels as the online algorithm saw—identical data coverage by construction. The only difference is that examples are presented in random order rather than in the particular temporal sequence induced by the evolving policy.

What offline training on $\mathcal{D}_{\text{online-shuffled}}$ tests: if Hypothesis 1 (data coverage) were correct, offline training on $\mathcal{D}_{\text{online-shuffled}}$ should match or approach online performance, because the coverage argument is about which responses appear in training, not when they appear. If performance remains substantially below online (as the paper finds in Figure 4 for most datasets), then coverage alone cannot explain the gap. The remaining explanation must involve the temporal structure—the fact that online training sees data ordered by the policy's own evolution, where early batches contain responses from policies close to the SFT baseline and later batches contain responses from more optimized policies. This temporal alignment between the data distribution and the policy's current state is what "on-policy" means operationally.

Interesting nuance—partial shuffling experiments (Appendix G, Figure 13): the paper also experimented with partial shuffling, where only a local window of batches is randomized rather than the entire dataset. The finding is that "as the amount of shuffling increases, the performance stays constant for a while before significantly deteriorating." In other words, the learning process is robust to small amounts of off-policy noise (batches slightly out of temporal order), but breaks down when the ordering is fully random. This suggests a smooth degradation rather than a cliff—the learning dynamics can tolerate some stale data, but the on-policy structure matters in aggregate.

The Classification-vs-Generation Analysis Framework

Section 5.3 introduces a conceptual apparatus for understanding what online and offline policies are actually learning. The core insight is that while both algorithms optimize a discriminative loss (pairwise preference classification), online training produces better generative performance even though it produces worse classifications.

Policies as classifiers (Appendix E): under the IPO loss and other contrastive RLHF objectives, a policy $\pi_\theta$ can be repurposed as a pairwise preference classifier without any additional training. Given a prompt $x$ and two responses $y_1, y_2$, the scoring function is:

$f_\theta(x, y_1, y_2) = \log\frac{\pi_\theta(y_1|x)}{\pi_\theta(y_2|x)} - \log\frac{\pi_{\text{sft}}(y_1|x)}{\pi_{\text{sft}}(y_2|x)}$

If $f_\theta(x, y_1, y_2) > 0$, the policy "classifies" $y_1$ as preferred to $y_2$, and vice versa. The SFT log-ratio term serves as a baseline, following the derivation of contrastive losses as binary classification (Tang et al., 2024). This scoring function is directly implied by the IPO objective: if the policy places more probability mass on $y_1$ relative to the SFT baseline than it does on $y_2$, it implicitly predicts that $y_1$ is better.

Classification accuracy measurement: the paper evaluates this classification ability on two types of data:

1. On the static preference dataset $\mathcal{D}_{\text{golden}}$: subsample 256 prompts and pairs of responses from the dataset, compute $f_\theta$ for each pair, and measure the fraction where the sign of $f_\theta$ agrees with the golden preference model's judgment. This is reported in Figure 8 (top row) across training steps.

2. On online-generated data: subsample 64 prompts and response pairs near each of 10 evenly spaced checkpoints from an online training run, and measure classification accuracy against the golden model. This is reported in Figure 6, showing how well policies and proxy models classify data from different points in the online training trajectory.

The counterintuitive finding: offline policies become good classifiers on the static dataset, with classification accuracy rising from random guess (~50%) to 60–70% over training (Figure 8, top row). This confirms that the offline optimization process works as intended—the policy is learning to distinguish winning from losing responses in the training data. Online policies, by contrast, maintain sub-50% classification accuracy on the same static dataset despite achieving much higher generative performance (win rates against the golden baseline).

The likelihood decay phenomenon (Figure 8, bottom row): an equally important observation is that all policies decrease the likelihood of winning responses from the static preference dataset over time. The metric plotted is:

$\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_{\text{golden}}} \left[ \log\pi_\theta(y_w|x) - \log\pi_{\text{sft}}(y_w|x) \right]$

This negative (and declining) value means that relative to the SFT baseline, the optimized policy places less probability on the training data's winning responses—sometimes substantially less. The decay is most drastic for offline algorithms, which is surprising given that their loss function is designed to increase the log-ratio $\log(\pi_\theta(y_w|x) / \pi_\theta(y_l|x))$. The paper interprets this as evidence that offline optimization achieves higher classification accuracy not by making winning responses more likely, but by making losing responses even less likely than winning ones—both probabilities decrease, but the loser's probability decreases faster.

Why this matters for the online-offline gap: the interpretation (Section 5.3.3) is that offline algorithms improve through a circuitous route: they reduce probability mass on the specific losing responses in the training set, which indirectly shifts mass toward better responses not present in the training set. This is inefficient and unpredictable—there's no guarantee that decreasing probability on one particular bad response will increase probability on the right good response. Online algorithms, by contrast, shift probability mass toward responses that are actually sampled and judged to be good by the proxy model, directly improving generation quality without needing to memorize what makes a bad response bad.

Self-classification accuracy (Appendix E.1): an intriguing side finding is that no trained policy can classify its own samples better than random chance. The metric:

$\mathbb{E}_{x \sim \rho, (y_w, y_l) \sim \pi_\theta} \left[ \mathbb{I}[f_\theta(x, y_1, y_2) > 0] \right]$

measures whether the policy's implicit classifier agrees with the golden model on pairs sampled from the policy itself. Even online policies, which generate their own training data and should theoretically align their probabilities with response quality, hover near 50% throughout training. The paper derives that a surrogate for this metric is related to but not exactly optimized by the IPO loss—the loss optimizes the integrand (the log-ratio) but not the sampling distribution. Since the theoretical optimal policy would achieve near-perfect self-classification (Appendix E.1), this indicates all trained policies remain far from optimal, and that self-classification accuracy might be a useful diagnostic for tracking convergence to optimality.

The Dataset Ablation Framework

Section 6 introduces a systematic method for constructing offline datasets with controlled properties, enabling direct testing of what makes offline data effective for policy learning. Four datasets are constructed from the same prompt set (from $\mathcal{D}_{\text{golden}}$), varying in how the two sides of each response pair are generated:

1. $\mathcal{D}_{\text{golden}}$ (baseline): the standard offline dataset, where both response sides come from the original dataset's response pool. Since the SFT policy is trained on the same pool, this is approximately $\mathcal{D}_{\text{sft vs. sft}}$—both sides are in-distribution for the starting policy.

2. $\mathcal{D}_{\text{sft vs. 800}}$: winning responses are generated by the SFT policy $\pi_{\text{sft}}$, and losing responses are generated by the online IPO policy at step 800 (an intermediate checkpoint). Both sides are then scored by the golden preference model to determine the actual winner. This dataset has the property that one side is exactly on-policy for the SFT starting point, and the two sides are generated by policies with substantially different quality, creating large differentiation.

3. $\mathcal{D}_{\text{800 vs. 4k}}$: winning responses are generated by the online IPO policy at step 800, losing responses by the online policy at step 4,000 (the final checkpoint). Both sides are far from the SFT distribution, and the quality gap between them represents the improvement that online training achieves. This dataset tests whether large response differentiation alone helps, without proximity to SFT.

4. $\mathcal{D}_{\text{4k vs. 4k}}$: both sides are generated by the online IPO policy at step 4,000—the highest-quality policy available. This directly tests Hypothesis 2 (that higher absolute response quality improves offline learning), and also provides the empirical case where theory breaks down (Appendix H).

After generation, all pairs are re-labeled by the golden preference model, ensuring the ground-truth preference signal is consistent across datasets. The offline IPO algorithm is then trained on each dataset with hyperparameter sweeps to produce KL-performance trade-off curves.

What each comparison reveals:

• $\mathcal{D}_{\text{800 vs. 4k}}$ vs. $\mathcal{D}_{\text{4k vs. 4k}}$: tests the effect of response differentiation. $\mathcal{D}_{\text{800 vs. 4k}}$ has large differentiation (policies at different training stages), while $\mathcal{D}_{\text{4k vs. 4k}}$ has two responses from the same high-quality policy (small differentiation). If offline performance is driven by having clear winners and losers, the former should outperform the latter.

• $\mathcal{D}_{\text{sft vs. 800}}$ vs. $\mathcal{D}_{\text{800 vs. 4k}}$: tests the effect of proximity to SFT. Both datasets have large differentiation, but $\mathcal{D}_{\text{sft vs. 800}}$ includes SFT-generated responses that are on-policy for the initial training phase, while $\mathcal{D}_{\text{800 vs. 4k}}$ includes only off-policy (online-trained) responses. If proximity to SFT matters, the former should outperform the latter.

• $\mathcal{D}_{\text{sft vs. 800}}$ vs. $\mathcal{D}_{\text{golden}}$: tests whether SFT-proximate data with clear differentiation outperforms the baseline dataset where both sides are SFT-generated but with smaller quality gaps. If differentiation matters, $\mathcal{D}_{\text{sft vs. 800}}$ should improve over $\mathcal{D}_{\text{golden}}$.

The results (Figure 12) show that $\mathcal{D}_{\text{sft vs. 800}}$ consistently matches or exceeds $\mathcal{D}_{\text{golden}}$, while $\mathcal{D}_{\text{800 vs. 4k}}$ and $\mathcal{D}_{\text{4k vs. 4k}}$ underperform. The interpretation: proximity to SFT is the most important property, not differentiation or absolute quality. By having at least one response side generated near the SFT distribution, the offline dataset mimics the initial stage of online learning, where the policy generates responses close to its own current distribution and receives feedback. This enables a solid first improvement step, after which the policy can continue to improve even from a static dataset.

The Best-of-2 (Bo2) Loss: A Non-Contrastive Baseline

To test whether the online-offline gap is specific to contrastive losses (Hypothesis 4), the paper introduces a simpler alternative loss function:

$\min_\theta \mathbb{E}_{x \sim p, (y_w, y_l) \sim \mu} \left[ \log\pi_\theta(y_w|x) \right]$

where the algorithm simply performs supervised fine-tuning on whichever response in the sampled pair is judged to be the winner—completely ignoring the losing response. This is the "Best-of-2" (Bo2) loss, studied by Gao et al. (2023) in the one-step over-optimization case.

What it computes: for each prompt, the policy's log-probability of generating the winning response $y_w$ is maximized, with no term discouraging the losing response $y_l$. The expectation is over the same sampling distribution $\mu$ as before, meaning the online variant samples responses from the current policy and labels them with the proxy model, while the offline variant uses a fixed dataset.

Why this form is useful as a diagnostic: the Bo2 loss eliminates several potential confounds present in contrastive losses. It has no log-ratio term, no reference to the SFT policy, no penalty for the losing response, and no non-linear transformation. If the online-offline gap were primarily caused by the contrastive loss structure (e.g., pathological behavior like decreasing both winner and loser probabilities), switching to Bo2 should dramatically reduce or eliminate the gap. If the gap persists, the problem is more fundamental—rooted in the sampling process itself rather than the loss function's mathematical form.

Key design difference from IPO: whereas IPO's loss depends on the log-ratio $\log(\pi_\theta(y_w|x) / \pi_{\text{sft}}(y_w|x)) - \log(\pi_\theta(y_l|x) / \pi_{\text{sft}}(y_l|x))$, which involves both responses and the SFT reference, Bo2 depends only on $\log\pi_\theta(y_w|x)$. This means Bo2 has no explicit mechanism for reducing probability on losing responses, no KL regularization against the SFT policy (beyond what the SFT initialization provides implicitly), and no target constant $\beta/2$ to tune. It is a pure SFT-style update on filtered data.

Online-offline comparison for Bo2 (Figure 9): the online and offline instantiations differ only in sampling distribution $\mu$. Online Bo2 samples from $\pi_\theta$, labels with the proxy model, and trains on the winner. Offline Bo2 trains on the fixed $\mathcal{D}_{\text{golden}}$ winners. The tandem shuffle experiment is also replicated: offline Bo2 on $\mathcal{D}_{\text{online-shuffled}}$ tests whether data coverage explains the gap.

Model Scaling Experiments

The scaling experiments (Section 5.5) extend the comparison across three model sizes using the T5X family:

All three sizes use the same proxy preference model (Large, 770M) during online training—the proxy model does not scale with the policy. This is a deliberate choice to maintain comparability: if the proxy model also scaled up, it would become harder to attribute performance changes to policy scaling versus improved training signal. The frozen proxy model also reflects a realistic deployment scenario where the reward/preference model is a fixed component.

Why the batch sizes decrease with model size: larger models consume more memory per forward pass, so the batch size is reduced to fit within hardware constraints. The paper does not adjust learning rates or other hyperparameters to compensate for the batch size changes—the only change is the batch size itself, with all other hyperparameters held at baseline values.

Evaluation consistency across sizes: all policies are evaluated against the same golden policy baseline using the same golden preference model, regardless of the evaluated policy's size. This means a Large policy and an XXL policy are judged on the same scale—win rate against a fixed reference. The absolute win rates therefore reflect real capability differences, not artifacts of changing evaluation protocols.

The best-performance computation (Figures 11 and 14): to compare scaling trends, the paper extracts the "best possible performance" for each method × size combination—the 90th percentile win rate across all hyperparameter configurations and checkpoints. This filters out suboptimal training runs and focuses on the achievable frontier. In Figure 11, these values are normalized by the best online policy at that size, showing the relative gap. In Figure 14 (Appendix G), absolute win rates are shown.

Summary of Design Choices and Their Justifications

• IPO loss for both online and offline over DPO or PPO: enables fair comparison by keeping the loss function identical across conditions, isolating sampling as the only variable. The squared error shape provides symmetric regularization against both under-fitting and over-fitting, which is important for stable online training where the data distribution shifts.

• Golden preference model for evaluation and data re-labeling over human raters or original dataset labels: provides a consistent, automated oracle that eliminates inter-rater noise and enables the large-scale experiments required for sweeping hyperparameters and populating KL-performance curves.

• Proxy preference model for online training signal (Large, 770M) over using the golden model directly or training a point-wise reward model: creates a realistic scenario with an imperfect training signal, enabling study of over-optimization dynamics. Keeping the proxy model fixed throughout online training isolates the effect of policy distribution shift.

• KL divergence budget metric over raw compute FLOPs or training steps: abstracts away from implementation-specific compute costs and provides a unified measure of optimization progress that is comparable across algorithms, hyperparameters, and model sizes.

• Tandem dataset construction ($\mathcal{D}_{\text{online-shuffled}}$) over simply comparing different offline datasets: provides the cleanest possible test of Hypothesis 1 (data coverage) by giving the offline algorithm exactly the same data as the online algorithm, varying only presentation order.

• Controlled dataset ablations ($\mathcal{D}_{\text{sft vs. 800}}$, etc.) over searching through existing datasets: enables systematic variation of dataset properties (proximity to SFT, response differentiation, absolute quality) to identify which characteristics drive offline performance.

• Same proxy model across all policy sizes over scaling the proxy model with the policy: keeps the training signal quality constant across scaling experiments, isolating the effect of policy capacity.

• SFT on pooled responses from all datasets over separate task-specific SFT: ensures the starting policy has reasonable generation quality across all tasks while keeping the distribution close to the training data, giving offline algorithms the most favorable possible initialization.

4. Key Insights and Innovations

Innovation 1: The Online-Offline Performance Gap Is Not About Data — It's About a Qualitatively Different Learning Dynamic

The paper's most fundamental conceptual contribution is not the observation that online methods outperform offline ones — that would be incremental — but the systematic elimination of intuitive explanations that points toward a genuinely surprising mechanism. Prior work comparing online and offline alignment either attributed differences to implementation details (PPO vs. DPO use different loss functions, architectures, and compute budgets) or assumed the gap would be explained by standard RL intuitions about data coverage or reward model quality. This paper systematically tests those intuitions and shows they are wrong.

What makes this a conceptual advance rather than a benchmark result is the hypothesis-testing methodology itself. The paper constructs five falsifiable hypotheses, each capturing a plausible mechanism that should explain the gap if standard RL reasoning transfers to RLHF:

• Data coverage (Hypothesis 1): The field's default assumption is that offline methods suffer because they train on a narrow static dataset while online methods see diverse responses from an evolving policy. This is the standard diagnosis in offline RL (Levine et al., 2020). The paper's tandem experiment (Figure 4, Section 5.1) tests this directly by giving offline algorithms the exact same response diversity as online algorithms — and finds the gap persists essentially unchanged on three of four datasets. Coverage is not the answer.

• Dataset quality (Hypothesis 2): A related intuition, grounded in the theoretical derivation of DPO and IPO, is that offline methods improve upon the behavior policy that generated the dataset. If the behavior policy is weak, offline methods start from a disadvantage. The paper tests this by constructing $\mathcal{D}_{\text{4k vs. 4k}}$ — a dataset of responses from a high-performance online policy — and finds offline training on this dataset barely improves over the SFT baseline (Figure 5, Section 5.2). This is a direct empirical counterexample to existing theory, which predicts that the optimal policy $\pi^* \propto \pi_{\text{sft}} \exp(\beta^{-1} p(y \succ \mu))$ should substantially improve over $\mu$. The breakdown occurs because the theoretical guarantee requires full support of $\mu$, which fails in practice.

• Classification accuracy (Hypothesis 3): Perhaps online methods work better because they use a separately trained proxy preference model that classifies response pairs more accurately than an offline policy can. The paper confirms that proxy models are indeed better classifiers (Figure 6, 70–90% vs. offline policies' ~70%), but then demonstrates that classification accuracy on a static dataset has essentially zero correlation with generative performance (Figure 7, Section 5.3.2). Improving a policy's ability to tell which of two fixed responses is better does not make it better at producing good responses.

The elimination of these hypotheses is not a null result — it is a diagnostic finding that redirects the field's attention away from data engineering (curating better offline datasets, improving coverage, filtering for quality) and toward a more subtle question about how learning dynamics differ under on-policy versus off-policy sampling. The paper doesn't just say "online is better"; it says "the standard reasons you think online is better are wrong, so something deeper is happening."

Innovation 2: The Discriminative-Generative Trade-off as a New Diagnostic Lens

The paper introduces a novel conceptual framework for understanding what alignment algorithms actually learn: the distinction between discriminative capability (can the policy classify which of two given responses is better?) and generative capability (does the policy produce responses that humans prefer?). Prior work on DPO and IPO had noted theoretically that these algorithms implicitly train the policy as a reward model, but no one had systematically measured how these two capabilities co-evolve during training and how they relate to actual performance.

The findings are counterintuitive and carry implications beyond this paper's specific experimental setup:

Offline training produces good classifiers but mediocre generators (Figure 8). Offline IPO policies steadily improve their classification accuracy on the static preference dataset from ~50% to 60–70%, confirming the optimization is working as designed. Yet this classification improvement does not translate into better win rates against the golden baseline — in fact, the correlation between classification accuracy and generative performance is essentially flat. The policy learns to say "Response A is better than Response B" but not to produce responses like A.

Online training produces good generators but poor classifiers (Figures 6, 8). Online IPO policies never exceed ~50% classification accuracy on any fixed dataset, yet achieve substantially higher generative performance. The online training dynamic — where the sampling distribution continuously shifts — prevents the policy from converging to a good classifier on any static distribution, but simultaneously drives it to shift probability mass toward better and better responses in an absolute sense.

The likelihood decay phenomenon reveals a mechanistic explanation (Figure 8, bottom row). Both online and offline policies decrease the probability of the training data's winning responses relative to the SFT baseline — and offline policies decrease them more drastically. This means offline improvement comes through an indirect route: the policy achieves higher classification accuracy not by making winners more likely, but by making losers even less likely than winners. Both probabilities drop, but the loser drops faster, increasing the log-ratio. This is a fragile mechanism — it depends on the specific losing responses in the training set and doesn't guarantee that the freed probability mass moves to genuinely good responses. Online improvement, by contrast, comes from directly increasing probability on responses that are sampled and judged good by the proxy model, even though the policy never becomes good at classifying responses it hasn't generated.

This discriminative-generative framework is significant because it provides a unified language for diagnosing alignment algorithms. Rather than asking "Does algorithm X beat algorithm Y on benchmark Z?", one can ask "Is this algorithm improving generative capability directly, or is it improving discriminative capability in hopes that it transfers?" The paper's evidence suggests that for current contrastive losses, discriminative improvement transfers poorly — a finding with direct implications for algorithm design. If one could design a loss function where discriminative accuracy on training data does correlate with generative quality, offline methods might close the gap without needing on-policy sampling.

Innovation 3: The Tandem Effect in RLHF — Why Data Ordering, Not Just Data Content, Controls Learning

The paper adapts the "tandem effect" concept from Ostrovski et al. (2021) to RLHF and demonstrates that its severity persists even under RLHF's seemingly favorable structure. This is both a methodological contribution (the experimental design of $\mathcal{D}_{\text{online}}$ and $\mathcal{D}_{\text{online-shuffled}}$) and a conceptual one (the finding that temporal coherence between data distribution and policy state is what "on-policy" operationally means).

Prior work on offline RL has largely focused on coverage — does the dataset contain sufficient diversity of states and actions to support learning a good policy? The tandem effect literature (Ostrovski et al., 2021) showed that in standard deep RL, even with identical coverage, passive observers of an active learner's data stream perform substantially worse. But there were good reasons to think RLHF might be different: the bandit formulation eliminates temporal credit assignment, the KL regularization toward the SFT policy provides implicit conservatism, and the reward signal comes from a learned model rather than the environment. Any of these could have mitigated or eliminated the tandem effect.

The paper's key finding (Figure 4) is that the effect persists strongly in RLHF — offline training on $\mathcal{D}_{\text{online-shuffled}}$ (identical data coverage, random order) substantially underperforms online training on three of four datasets. The partial shuffling experiments (Appendix G, Figure 13) add nuance: small amounts of shuffling (local randomization within a window) are tolerated, but full randomization breaks the learning dynamic. This suggests the critical property is not strict one-step temporal alignment but rather a coarser statistical property — that the data distribution at each training stage is approximately matched to the policy's current state.

What makes this a conceptual advance is the redefinition of "on-policy" for the RLHF context. In standard RL, "on-policy" means the data was generated by the current policy. In RLHF with a fixed proxy preference model, the data is labeled by a frozen model — only the responses are on-policy. The paper's results suggest that this partial on-policy property (on-policy generation + off-policy labeling) is sufficient to capture most of the benefit of full on-policy learning. This has practical implications: it means that maintaining an up-to-date reward model may be less important than maintaining an up-to-date sampling distribution, which is a simpler infrastructure requirement.

Innovation 4: Proximity to the Starting Policy as the Key Determinant of Offline Data Quality

The dataset ablation in Section 6 (Figure 12) produces a finding that is simultaneously intuitive in retrospect and surprising given prior assumptions: having at least one response side generated near the SFT policy's distribution is the single most important property of offline RLHF data, more important than response differentiation or absolute response quality.

This finding challenges two common intuitions:

Against "better data solves offline alignment": The natural response to finding that offline methods underperform is to curate higher-quality preference data — responses from stronger models, clearer preference pairs, higher absolute quality. The paper's $\mathcal{D}_{\text{4k vs. 4k}}$ experiment (both responses from a strong online policy) and $\mathcal{D}_{\text{800 vs. 4k}}$ experiment (large quality gap between two strong policies) both perform poorly, showing that high absolute quality and clear differentiation are not sufficient.

Against the theoretical optimality of DPO/IPO: The derivations of these algorithms (Rafailov et al., 2023; Azar et al., 2023) assume the offline dataset provides sufficient coverage to evaluate $p(y \succ \mu)$ for all responses the optimal policy might assign probability to. The empirical breakdown (Appendix H) demonstrates this assumption fails in practice: when $\mu$ is a narrow high-quality policy, there are many good responses outside its support that the optimal policy cannot discover from the data. The practical fix — making the dataset more on-policy by including SFT-generated responses — is not predicted by the theory, which treats all responses with non-zero probability under $\mu$ symmetrically.

The conceptual contribution is the reframing of offline dataset construction from a coverage problem to an on-policy initialization problem. The optimal offline dataset, under this view, should not aim for maximum coverage or maximum quality, but should instead provide a bridge: responses that are close enough to the starting policy's distribution that the initial learning steps can make meaningful progress, after which the policy can continue improving even from static data. $\mathcal{D}_{\text{sft vs. 800}}$ succeeds not because it has better responses than $\mathcal{D}_{\text{4k vs. 4k}}$ (it has worse ones), but because it is aligned with the policy's initial state in a way that enables a clean first improvement step.

This has direct practical implications: when constructing preference datasets for offline alignment, one should prioritize including responses from the base model or its near variants (the SFT checkpoint, early RLHF checkpoints) over responses from stronger but distributionally distant models. It also suggests a hybrid deployment strategy: start with an offline phase using SFT-proximate data, then transition to online updates once the policy has moved far enough that the offline data becomes stale — effectively blurring the boundary between offline and online in a principled way.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Four open-source preference datasets are used: OpenAI summarization (Stiennon et al., 2020), Anthropic helpfulness (Bai et al., 2022), Anthropic harmlessness (Bai et al., 2022), and Chat arena side-by-side (Chiang et al., 2024). Only the training split is used for all experiments since the study does not investigate generalization to test-set prompts. The initial datasets contain prompts with labeled pairs of winning and losing responses.

• Base model(s). All experiments use T5X encoder-decoder models (Raffel et al., 2020) implemented within the T5X data and compute framework (Roberts et al., 2023). The primary policy and proxy preference models are Large T5X (770M parameters). Scaling experiments additionally use XL (3B parameters) and XXL (11B parameters). The golden preference model is always XXL (11B). The choice is driven by the need for a controlled, reproducible setup where model scale can be varied systematically — T5X provides a family of models spanning an order of magnitude in parameter count, enabling scaling analysis without changing architecture families.

• Metrics. The primary metric is side-by-side win rate of the learned policy against a fixed golden policy baseline, judged by the golden preference model. For a given policy $\pi_\theta$, the evaluation procedure: (1) 2,048 prompts are deterministically sub-sampled from the training set; (2) for each prompt, both $\pi_\theta$ and the golden baseline policy generate responses; (3) the golden preference model judges which response is preferred using the debiased inference procedure ($\text{sign}(r(x, y_1, y_2) - r(x, y_2, y_1))$); (4) win rate is the fraction of prompts where $\pi_\theta$'s response is preferred. A secondary metric is KL divergence $\mathbb{KL}(\pi_\theta, \pi_{\text{sft}})$, estimated unbiasedly by sampling 256 prompts (32 for XL models), generating responses from $\pi_\theta$, and computing the per-token KL between $\pi_\theta$ and $\pi_{\text{sft}}$ distributions, averaged over all token positions and prompts (Appendix D). Classification accuracy (used diagnostically in Section 5.3) is measured by subsampling 256 data points from the pairwise preference dataset and computing the fraction where the policy's implicit classifier $f_\theta(x, y_1, y_2)$ agrees with the golden preference model's judgment (Appendix E).

• Baselines. The paper uses several baselines, all instantiated within the same IPO loss framework:

  • Offline IPO: The policy is trained directly on $\mathcal{D}_{\text{golden}}$ (the golden relabeled dataset) using the IPO loss with a static sampling distribution. This is the primary baseline throughout.
  • Offline with shuffled online data ($\mathcal{D}_{\text{online-shuffled}}$): An offline variant trained on the shuffled version of the online-generated data stream, controlling for data coverage while randomizing temporal order (Section 5.1, Figure 4).
  • Online IPO: The policy is trained with on-policy sampling and proxy preference model labeling (Section 2). This serves as the upper-bound reference in most comparisons.
  • Golden policy baseline: A policy trained via online IPO with direct access to the golden preference model (rather than a proxy model), giving it privileged information. All evaluated policies are compared against this baseline for win rate computation.
  • Best-of-2 (Bo2) online and offline: Non-contrastive variants where the loss is simply $\log\pi_\theta(y_w|x)$, with online and offline instantiations (Section 5.4, Figure 9).

• Generation budget / compute accounting. The paper does not use raw FLOPs or wall-clock time as the primary budget metric. Instead, it adopts the KL divergence $\mathbb{KL}(\pi_\theta, \pi_{\text{sft}})$ as a unified measure of optimization budget (Section 2, Section 4.1), following Gao et al. (2023). This choice is explicitly motivated: "we do not prioritize compute as a main factor during comparison, and instead adopt the setting from Gao et al. which uses the KL divergence between the RLHF policy and reference SFT policy as a measure of budget." The rationale is that KL divergence measures how far the policy has drifted from the trusted SFT starting point, providing a hyperparameter-agnostic metric that is comparable across algorithms with different per-step compute costs. For offline algorithms, different hyperparameter combinations (learning rate, $\beta$, training steps) produce different KL trajectories, and the full set of checkpoints populates the KL vs. performance curves.

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The paper pools results across hyperparameter sweeps (learning rates $\{3 \times 10^{-6}, 1 \times 10^{-5}, 3 \times 10^{-5}\}$, regularization coefficients $\beta \in \{0.1, 0.5, 1.0\}$, training steps $\{4\text{k}, 20\text{k}\}$) to populate trade-off curves, but does not report confidence intervals, standard errors, or statistical tests for any comparison. The evaluation uses a fixed set of 2,048 deterministically sub-sampled prompts across all experiments, providing consistency but not enabling estimation of evaluation variance. The "best possible performance" metric (Figures 11, 14) is defined as the 90th percentile across experiments for a given method-size combination, which provides a non-parametric summary but without uncertainty quantification.

---

Main Quantitative Results

5A. Online Achieves Better KL-Performance Trade-offs Across All Datasets (Figure 1)

Figure 1 presents the central motivating result: the trade-off between KL divergence and policy performance for online and offline IPO across all four datasets. Each data point represents a policy checkpoint from a specific hyperparameter configuration, with offline points pooled across sweeps and online points from a single fixed hyperparameter set.

Key quantitative patterns:

• Anthropic helpfulness: Online algorithms achieve substantially higher peak performance and better efficiency. At matched KL divergence (approximately 5–10 nats), the online policy achieves roughly 15–20 percentage points higher win rate than the best offline configuration. The offline curve plateaus at a lower peak (approximately 55–60% win rate) compared to online (approximately 70–75% peak).

• OpenAI summarization: Similar pattern with online showing clearly superior trade-off. The offline curve peaks at roughly 55% while online reaches approximately 65–70%. At low KL budgets (2–5 nats), the gap is smaller but online still maintains advantage.

• Anthropic harmlessness and Chat arena sxs: The gap is narrower but still present. Online achieves higher peak performance (roughly 5–10 percentage points better) though the difference is less dramatic than for helpfulness and summarization. For Chat arena sxs, at high KL budgets the offline and online curves approach each other more closely.

• Over-optimization under Goodhart's law: Both online and offline curves exhibit the characteristic inverted-U shape — performance initially increases with KL divergence, reaches a peak, and then declines as the policy over-optimizes against the proxy signal (online) or the finite dataset (offline). The peak for online occurs at a higher KL and higher performance level than for offline.

The paper notes that "online algorithms are a pareto improvement over offline algorithms" — for any given KL budget, online achieves better or equal performance, and the maximum achievable performance is higher.

5B. Data Coverage Alone Does Not Explain the Gap (Figure 4, Section 5.1)

The tandem experiment tests Hypothesis 1 by comparing offline training on $\mathcal{D}_{\text{online-shuffled}}$ (identical response coverage as online, random order) against online training and baseline offline training. Figure 4 overlays these new curves on top of the Figure 1 results.

Key quantitative findings:

• OpenAI summarization: Offline with $\mathcal{D}_{\text{online-shuffled}}$ shows essentially no improvement over baseline offline with $\mathcal{D}_{\text{golden}}$. The curve lies nearly on top of the baseline offline results, remaining substantially below online (roughly 15–20 percentage points gap persists).

• Anthropic helpfulness: Similarly, $\mathcal{D}_{\text{online-shuffled}}$ provides negligible benefit over $\mathcal{D}_{\text{golden}}$. The gap to online remains large across all KL budgets.

• Chat arena sxs: This is the notable exception. Offline with $\mathcal{D}_{\text{online-shuffled}}$ nearly matches online performance, closing the gap much more than for the other two datasets. The paper notes this as a "probable exception" and cautions that "arriving at the shuffled dataset $\mathcal{D}_{\text{online-shuffled}}$ in the first place is infeasible since one requires access to the whole path of data generated by an online algorithm."

• Anthropic harmlessness: The shuffled online data provides modest improvement over baseline offline but still falls short of online, particularly at higher KL budgets.

The paper concludes that "overall, there is little improvement on the offline performance due to the change from offline to the shuffled online dataset" and that "the performance difference between online and offline algorithms cannot be explained by the difference in data coverage alone."

5C. Higher-Quality Offline Data Does Not Close the Gap (Figure 5, Section 5.2)

Hypothesis 2 is tested by constructing $\mathcal{D}_{\text{4k vs. 4k}}$ — a dataset where both response sides are generated by the online IPO policy at step 4,000 (the highest-quality checkpoint), then relabeled by the golden preference model. Offline IPO is trained on this dataset with hyperparameter sweeps.

Key quantitative findings (Figure 5):

• OpenAI summarization: The offline policy trained on $\mathcal{D}_{\text{4k vs. 4k}}$ achieves win rates that "linger around low values" — barely above the SFT baseline and substantially below even the baseline $\mathcal{D}_{\text{golden}}$ offline results. The curve for $\mathcal{D}_{\text{4k vs. 4k}}$ is flat or nearly flat across KL values, showing negligible improvement from optimization.

• Anthropic helpfulness: Similar pattern — $\mathcal{D}_{\text{4k vs. 4k}}$ underperforms $\mathcal{D}_{\text{golden}}$ and shows minimal improvement over the SFT starting point.

• Anthropic harmlessness: The same pattern holds.

The paper notes this constitutes "an empirical case where existing theories of offline algorithms fail to predict" — the theoretical optimal policy $\pi^* \propto \pi_{\text{sft}} \exp(\beta^{-1} p(y \succ \mu))$ should substantially improve over $\mu$, but cannot because $\mu$ lacks full support over response space. The finding directly challenges the theoretical foundations of DPO and IPO that assume the behavior policy's coverage is sufficient.

5D. Classification Accuracy Does Not Predict Generative Performance (Figures 6–8, Section 5.3)

Section 5.3 examines the relationship between discriminative and generative capabilities.

Proxy model classification superiority (Figure 6): Across the sequence of data generated by an online experiment, the proxy preference model achieves substantially higher classification accuracy than either online or offline policies used as classifiers. On the OpenAI summarization task, the proxy model maintains roughly 70–80% accuracy on early online data, declining to approximately 60% by the end as the data becomes increasingly out-of-distribution. Online and offline policies used as classifiers hover near 50% throughout, with offline policies slightly higher (55–60% range) but far below the proxy model. This confirms part of Hypothesis 3: proxy models are indeed better classifiers.

Zero correlation between classification and generation (Figure 7): The paper plots policy win rate against classification accuracy on $\mathcal{D}_{\text{golden}}$ for both online and offline experiments across all datasets. The key finding is that within the offline experiments, there is "little statistical correlation between the classification accuracy and model performance" — policies with higher classification accuracy on the preference dataset do not have higher win rates against the golden baseline. Offline policies achieve classification accuracy in the 60–70% range while online policies remain near 50%, yet online policies have substantially higher generative performance. This refutes the second part of Hypothesis 3: improving classification accuracy would not close the gap.

Classification accuracy evolution during training (Figure 8, top row): Offline policies steadily improve classification accuracy on $\mathcal{D}_{\text{golden}}$ from approximately 50% to 60–70% over training, with the offline-on-shuffled variant following a similar trajectory. Online policies remain near 50% (chance level) throughout. The offline optimization is working as intended — the policy is learning to classify — but this learning does not transfer to generation.

Likelihood decay for winning responses (Figure 8, bottom row): All policies show decreasing log-probability of the winning responses from $\mathcal{D}_{\text{golden}}$ over training, measured as $\mathbb{E}[\log\pi_\theta(y_w|x) - \log\pi_{\text{sft}}(y_w|x)]$. The decay is most drastic for offline algorithms, with the value dropping substantially below zero (winning responses become less likely than under the SFT baseline). Online policies also show decay but to a lesser degree. The paper interprets: "in order to obtain high classification accuracy, a frequent solution of offline optimization is to set the likelihoods of winning responses low and losing responses even lower."

Online classification accuracy over training (Figure 8, online row): While Figure 6 showed out-of-distribution classification (on the online data stream), Figure 8 shows classification on the static $\mathcal{D}_{\text{golden}}$. The online policy consistently fails to exceed random chance on this static dataset, reinforcing that online training does not produce good classifiers on any fixed distribution.

5E. Non-Contrastive Losses Show Some Gap Closure (Figure 9, Section 5.4)

The Best-of-2 (Bo2) loss replaces the contrastive IPO objective with a simple SFT-on-winner objective. Figure 9 compares online vs. offline Bo2 alongside the original IPO curves (shown faintly for reference).

Key quantitative findings:

• Chat arena sxs: The online-offline gap essentially vanishes for Bo2. Offline Bo2 achieves a trade-off curve that is on par with or slightly better than online Bo2 at high KL budgets. The two curves largely overlap across the observed KL range.

• OpenAI summarization and Anthropic helpfulness: The gap persists but is reduced in magnitude compared to the IPO case. Offline Bo2 still underperforms online Bo2, but the difference is less stark than for IPO. Additionally, offline Bo2 with $\mathcal{D}_{\text{online-shuffled}}$ closely matches online Bo2 performance, suggesting that for Bo2, the data coverage hypothesis provides a better explanation.

• Cross-dataset pattern: The paper notes that "the data coverage hypothesis might offer a better explanation" for Bo2 since offline with shuffled online data nearly recovers online performance, whereas this was not the case for IPO (except for Chat arena sxs).

The takeaway is nuanced: the online-offline gap is not purely an artifact of contrastive losses, since it persists (albeit reduced) with Bo2 on most datasets. However, the underlying mechanism may differ between loss functions — for IPO, the temporal ordering of data is critical; for Bo2, data coverage explains more of the gap.

5F. Scaling Policy Networks Narrows But Does Not Eliminate the Gap (Figures 10–11, Section 5.5)

The scaling experiments compare Large (770M), XL (3B), and XXL (11B) policy networks, all using the same proxy preference model (Large, 770M).

Key quantitative findings (Figure 10):

• Over-optimization persists at all scales: The inverted-U shape is present for both Large and XL policies, and for both online and offline variants. "This corroborates that the bottleneck is the proxy preference model, and scaling policy sizes will not make a fundamental impact" (citing Gao et al., 2023).

• Peak performance increases with scale: Both online and offline peak win rates improve as policy size increases. However, the relative gap between online and offline persists — online XL outperforms offline XL by a similar margin as online Large outperforms offline Large.

• Peak KL is similar across scales: The KL divergence at which peak performance is achieved is roughly constant across model sizes for a given method, consistent with Gao et al. (2023). The paper notes that KL values are "not comparable between the two policy sizes since they are with respect to different SFT policies."

• Chat arena sxs shows gap closure at scale: For this dataset, online and offline performance become much closer at the XL scale, with the gap nearly closing. The paper does not observe this for the other two datasets tested at scale.

Best-performance scaling trends (Figure 11): By extracting the 90th percentile win rate for each method × size combination and normalizing by the best online policy at that size, the paper tracks how the relative gap scales:

• Online vs. offline with $\mathcal{D}_{\text{online-shuffled}}$: The gap narrows substantially as model size increases (16× scaling from Large to XXL), for both OpenAI summarization and Anthropic helpfulness. This "indicates a bigger role by the data coverage hypothesis" at larger scales — bigger models can better leverage the additional coverage in the shuffled online dataset.

• Online vs. baseline offline (with $\mathcal{D}_{\text{golden}}$): The gap narrows but at a slower rate. For Anthropic helpfulness, the normalized gap decreases from roughly 0.4 to 0.3 across the 16× scale increase. For OpenAI summarization, the decrease is smaller (from roughly 0.35 to 0.25). The paper concludes that "it is also not clear whether the gap will plateau at certain point, indicating a potentially divergent gap that cannot be bridged by just scaling the policy network."

• Chat arena sxs: The gap is smaller to begin with and nearly closes at XXL scale across all offline variants.

Absolute win rates (Figure 14, Appendix G): The raw win rates show that online XXL achieves the highest absolute performance across all tasks, followed by offline with $\mathcal{D}_{\text{online-shuffled}}$ XXL, then baseline offline XXL. The ordering is consistent across model sizes.

---

Ablation Studies and Robustness Checks

• Partial shuffling of $\mathcal{D}_{\text{online}}$ (Appendix G, Figure 13): Offline IPO is trained on $\mathcal{D}_{\text{online}}$ with varying levels of shuffling — from zero (exact online order) to full uniform shuffle. At small shuffling levels, the trade-off curve is similar to online performance. As shuffling increases, given a fixed KL budget, performance degrades. The paper interprets this as robustness to slight off-policyness — "when the shuffling is small, the dataset is still largely on-policy."

• Proxy model vs. policy as classifier (Figures 6–7, Section 5.3): The proxy preference model achieves 70–90% training accuracy on $\mathcal{D}_{\text{golden}}$ while offline policies peak at ~70% and online policies near 50%. However, Figure 7 shows this classification advantage does not translate to generative performance — within offline experiments, there is no positive correlation between classification accuracy and win rate. This is both an ablation confirming the proxy model's superior discriminative capability and a demonstration that such superiority does not cause the online advantage.

• Self-classification accuracy (Appendix E.1): Policies' ability to classify their own samples — $\mathbb{E}_{x, (y_w, y_l) \sim \pi_\theta} [\mathbb{I}[f_\theta(x, y_1, y_2) > 0]]$ — is measured throughout training and found to hover near 50% (random chance) for all algorithms, including online. The paper derives that the IPO loss does not directly optimize this metric and that "all the policies we have trained are likely all far from optimal." This is a negative result showing that even the best-performing policies have poor self-assessment capability.

• Positional bias mitigation in preference models (Appendix A): The proxy and golden preference models are trained with a symmetrized loss (Equation 5) to address positional bias — the tendency to prefer whichever response appears first regardless of content. At inference, preference is determined by the difference $r_\theta(x, y_1, y_2) - r_\theta(x, y_2, y_1)$ rather than a single forward pass. The paper notes that "the online RLHF process can easily exploit" remaining positional bias if only a single ordering is used, validating the necessity of this debiasing step.

• Dataset property ablation — proximity to SFT vs. differentiation vs. absolute quality (Figure 12, Section 6): Three properties of offline datasets are varied systematically:

  • Large differentiation without SFT proximity ($\mathcal{D}_{\text{800 vs. 4k}}$): Does not improve over $\mathcal{D}_{\text{golden}}$ despite having responses from policies at different training stages (large differentiation).
  • High absolute quality without SFT proximity ($\mathcal{D}_{\text{4k vs. 4k}}$): Performs poorly, barely above the SFT baseline.
  • SFT proximity with large differentiation ($\mathcal{D}_{\text{sft vs. 800}}$): Consistently matches or exceeds $\mathcal{D}_{\text{golden}}$ across datasets, despite having the same or lower absolute response quality. This identifies proximity to SFT as the key property. The comparison $\mathcal{D}_{\text{sft vs. 800}}$ vs. $\mathcal{D}_{\text{golden}}$ ($\mathcal{D}_{\text{sft vs. sft}}$) is dataset-dependent — neither consistently dominates, as both have SFT proximity but differ in differentiation.

• SFT initialization quality check (Appendix C): The SFT stage pools all responses from all four downstream tasks and trains for 4k steps (Large) or 8k steps (XL/XXL). The paper verifies that "from the beginning of RLHF, the policy can already produce reasonable samples with a wide coverage" and that the starting distribution is "not too out-of-distribution from the pairwise preference dataset that the preference models are trained on." This is a robustness check ensuring the offline algorithms are not disadvantaged by starting far from their training distribution.

• Batch size and training step consistency across scales (Section 5.5): For scaling experiments, batch sizes decrease with model size (32 for Large, 16 for XL, 8 for XXL) due to hardware constraints, but all other hyperparameters remain at baseline values. The paper acknowledges this as a potential confound but does not ablate on batch size directly — the scaling comparison conflates model capacity with batch size effects.

• Proxy model fixed across policy sizes (Section 5.5): In all scaling experiments, the same Large (770M) proxy preference model is used for online training, regardless of policy size. This is intentional to isolate policy scaling effects, but it means the online training signal quality degrades relative to policy capacity as the policy grows — a larger policy can more easily over-optimize against a fixed-capacity proxy model.

• Hardware tandem effect verification (Appendix F): The paper empirically verified that offline training on $\mathcal{D}_{\text{online}}$ (exact online order) "almost perfectly matches the online experiment, in terms of the training statistics up to ~0.1% throughout training." This required ensuring token-level match of the data stream, as "even slight mismatch would cause drastic divergence." The discrepancy slowly increases over time due to "inherent numerical randomness in the compute hardware" — an effect the paper terms the "hardware tandem effect."

---

Critical Assessment

Does the Evidence Support the Central Claim That On-Policy Sampling Is Fundamentally Necessary?

The paper's headline claim — that the online-offline gap reveals a "fundamental necessity of on-policy learning for AI alignment" (Section 8) — is supported with carefully controlled evidence, but with boundary conditions that the paper itself acknowledges and that warrant caution.

What the experiments convincingly demonstrate: Under the specific experimental conditions — T5X encoder-decoder models, the IPO loss, four specific preference datasets, a fixed proxy preference model, and a particular SFT initialization protocol — offline algorithms consistently underperform online ones, and this underperformance cannot be attributed to data coverage, data quality, classification accuracy, contrastive loss pathologies, or insufficient model scale. The tandem experiment (Figure 4) is the strongest single piece of evidence: giving offline algorithms the exact same response diversity as online algorithms, differing only in presentation order, fails to close the gap on three of four datasets. This cleanly isolates the temporal structure of on-policy sampling — the alignment between the data distribution's evolution and the policy's evolution — as the critical factor.

The classification-vs-generation analysis (Figures 6–8) provides a mechanistic explanation that is internally consistent and makes testable predictions: offline training improves discriminative ability on a static dataset through an indirect mechanism (making losers less likely rather than winners more likely), while online training improves generative ability by directly shifting probability mass toward better responses discovered through sampling. The likelihood decay pattern (Figure 8, bottom row) provides converging evidence for this mechanism.

What the experiments do NOT demonstrate: Several boundary conditions limit the generality of the "fundamental necessity" claim:

1. Single model architecture. All experiments use T5X encoder-decoder models. It is at least plausible that decoder-only models (GPT-style), which have different inductive biases and generation patterns, might exhibit different online-offline dynamics. The paper acknowledges this limitation only implicitly.

2. Single loss function family (with one exception). The IPO loss is used throughout, with Bo2 providing a single non-contrastive comparison point. While the paper argues that results should transfer to DPO and other contrastive losses "due to similar empirical patterns" (citing Tang et al., 2024), this is an extrapolation, not a demonstrated fact. The Bo2 results (Figure 9) actually suggest the dynamics may differ between loss functions — for Bo2, data coverage explains more of the gap, and for Chat arena sxs, the gap nearly closes.

3. Fixed proxy model quality. The online algorithm's feedback signal comes from a single proxy preference model trained once and held fixed. The paper does not investigate how the online-offline gap changes when the proxy model is periodically updated, when it is trained on-policy data, or when it is scaled up alongside the policy. The finding that classification accuracy of the proxy model degrades over online training (Figure 6) suggests the proxy model becomes increasingly unreliable as the policy distribution shifts — this degradation itself contributes to online over-optimization and may not be a fundamental feature of online learning but rather an artifact of the fixed-proxy design.

4. SFT initialization quality. The paper deliberately chose an SFT stage that produces "reasonable samples with a wide coverage." How would the gap change if the SFT policy were weaker (producing low-quality samples) or stronger (already producing good responses)? The dataset ablation (Figure 12) suggests proximity to SFT is critical — implying the gap might be smaller or larger depending on SFT quality, but this is not systematically tested.

5. No dynamic or iterative offline approaches. The paper treats "offline" as a single-pass over a static dataset. However, as the paper notes (Section 8), "an offline algorithm with a repeatedly updated data stream is effectively an online algorithm." Many practical deployments of offline methods involve multiple rounds of data collection and retraining — a middle ground between purely static offline and purely sequential online that the paper does not explore. The finding that small amounts of shuffling are tolerated (Appendix G, Figure 13) suggests that iterative offline methods — where the dataset is periodically resampled from the current policy — might capture most of the online benefit while avoiding the infrastructure complexity of per-step on-policy sampling.

6. Dataset-specific effects. The gap is much smaller for Chat arena sxs than for OpenAI summarization or Anthropic helpfulness. For Chat arena sxs, offline with $\mathcal{D}_{\text{online-shuffled}}$ nearly matches online (Figure 4), and with Bo2, offline actually matches or exceeds online (Figure 9). The paper speculates about dataset properties (e.g., "for chat arena sxs, it might suffice to SFT on the winning response") but does not systematically characterize what dataset properties modulate the gap. This means the "fundamental necessity" claim may be dataset-dependent in ways the paper does not quantify.

Are the Quantitative Comparisons Fair?

The KL divergence budget metric is well-motivated but has limitations. KL divergence captures policy drift from the SFT reference, which is a meaningful measure of how much the policy has changed. However, it has two notable issues:

• KL divergence is evaluated on the policy's own samples (unbiased estimator). This means that for online policies, the KL is computed on responses from the current policy distribution, while for offline policies, different KL values come from different hyperparameter configurations that may produce policies with different sampling distributions. The KL values are not directly comparable in the sense that a KL of 10 for an online policy and a KL of 10 for an offline policy may represent different amounts of "effective optimization" if the shape of the policy shift differs.

• KL does not account for computational cost. The paper explicitly states it "does not prioritize compute as a main factor" and does not report FLOPs, wall-clock time, or sample efficiency. This is a deliberate scope limitation, but it means the reader cannot assess whether online methods' performance advantage is worth their additional computational cost. An offline method that achieves 90% of online performance at 10% of the computational cost might be preferable in practice, even if the online method is on a better KL-performance Pareto frontier.

The online hyperparameter configuration is not swept. The online results in Figure 1 come from a single hyperparameter configuration (learning rate $10^{-5}$, $\beta = 0.1$, 4k steps), while offline results are pooled across extensive sweeps. This asymmetry means the offline frontier represents the best achievable performance for any hyperparameter combination, while the online results represent the performance of a specific configuration. If the online hyperparameters were also swept, the online frontier might be even higher, potentially widening the gap. Conversely, the "best possible performance" metric (90th percentile) for online is based on a much smaller sample of configurations than for offline, potentially underestimating online's achievable peak.

The proxy model is fixed across policy scales. In the scaling experiments (Section 5.5), the online algorithm uses the same Large (770M) proxy model regardless of whether the policy is Large, XL, or XXL. This means that as the policy scales up, the quality of the training signal relative to the policy's capacity degrades — a larger policy can more easily over-optimize against a fixed-capacity proxy. The online curves for larger policies might reflect this degradation as much as they reflect fundamental scaling properties. A fairer comparison would scale the proxy model alongside the policy, which would likely increase the online advantage (as shown by Gao et al., 2023) but also increase the computational cost disparity.

Do the Hypothesis Tests Have Sufficient Statistical Power?

The paper does not report confidence intervals, standard errors, or any formal statistical tests. Several aspects of the experimental design raise questions about the reliability of the quantitative comparisons:

• Single evaluation set. All policies are evaluated on the same 2,048 deterministically sub-sampled prompts. While this ensures consistency, it provides no estimate of evaluation variance. A different set of 2,048 prompts might produce meaningfully different win rates, and the paper cannot quantify this uncertainty.

• Single training run per configuration. Each data point in the trade-off curves represents a single training run with a specific hyperparameter configuration. There is no replication to assess training variance — different random seeds might produce different KL trajectories and different peak performances.

• Small sample for certain comparisons. The classification accuracy measurements in Figures 6–8 use subsamples of 256 data points. For the self-classification accuracy metric (Appendix E.1), the paper reports that policies "cannot do statistically significantly better than random guess" but does not report the sample size or the statistical test used to reach this conclusion.

• Best-performance metric sensitivity. The "best possible performance" metric (90th percentile, Figures 11 and 14) is computed across all hyperparameter configurations for a given method-size combination. With different numbers of configurations per method (offline has more due to hyperparameter sweeps), the 90th percentile may have different statistical properties across methods. At small sample sizes (e.g., online XL with a single hyperparameter configuration), the 90th percentile may be poorly estimated.

What Experiments Would Have Strengthened the Paper?

Several experiments would have substantially strengthened the paper's conclusions:

1. Replication across random seeds. Running 3–5 seeds for a subset of configurations and reporting means with confidence intervals would address concerns about training variance and enable statistical comparison of the trade-off curves.

2. Compute-matched comparison. In addition to the KL-matched comparison, a comparison where online and offline receive the same total FLOPs budget (including proxy model training, generation, and inference costs) would address the practical question of which approach is more cost-effective. The paper's decision to "not prioritize compute as a main factor" is a legitimate scope limitation, but a FLOPs-matched comparison would substantially increase practical relevance.

3. Proxy model scaling. Testing online performance with proxy models scaled to match the policy size (XL proxy for XL policy, etc.) would disentangle the effect of policy scaling from the effect of proxy model quality. This would also connect more directly to Gao et al. (2023)'s finding that larger reward models reduce over-optimization.

4. Iterative offline baselines. A comparison between one-pass offline, multi-pass offline with periodic dataset re-collection from the latest policy, and fully online would characterize the spectrum between purely static and purely sequential approaches. This would address the paper's own observation that "the boundary between online and offline is often quite blurred" (Section 7).

5. Per-dataset characterization. Analyzing what dataset properties (response diversity, preference strength, topic distribution, prompt complexity) correlate with the size of the online-offline gap would provide actionable guidance for practitioners. The paper observes that the gap varies across datasets but does not systematically investigate why.

6. Decoder-only model replication. Replicating a subset of experiments with a decoder-only architecture (e.g., a T5 decoder-only variant or a GPT-style model) would test the generality of the findings beyond the encoder-decoder family.

Are the Negative Results (Rejected Hypotheses) Reliable?

The rejection of Hypotheses 1 (data coverage) and 2 (dataset quality) is the strongest part of the paper's empirical case. The tandem experiment (Figure 4) cleanly isolates data coverage by giving offline algorithms identical response diversity to online algorithms. The high-quality dataset experiment (Figure 5) directly tests and refutes the theoretical prediction that offline methods should improve over their behavior policy. Both experiments have clear, interpretable results with large effect sizes.

The rejection of Hypothesis 3 (classification accuracy) is slightly less clean — Figure 7 shows no correlation within offline experiments, but the overall pattern (offline has higher classification accuracy but lower generative performance) could be consistent with a more complex relationship where classification accuracy matters but is dominated by other factors. The paper's conclusion that "classification accuracy alone cannot explain the performance gap" is well-supported, but the stronger claim that improving classification accuracy "is unlikely to yield significant performance gains" assumes the observed flat correlation in the current setup would persist under different training procedures or loss functions, which is untested.

The rejection of Hypothesis 4 (non-contrastive losses) is the most qualified. The Bo2 results (Figure 9) show that the gap persists but also that data coverage becomes a more important explanation for Bo2 than for IPO. This suggests the gap's mechanism is partially loss-function-dependent — a nuanced finding that the paper handles appropriately but that complicates the "fundamental necessity" narrative.

The rejection of Hypothesis 5 (scaling fixes everything) is supported by the observation that the gap narrows but does not close over a 16× increase in parameters, with the rate of closure slow enough to suggest a potentially persistent gap. However, the limited scale range (770M to 11B, or roughly 14×) is modest compared to the orders-of-magnitude scaling studied in other contexts, and the gap might close at 100× or 1000× scale in ways that are not predictable from the 14× trend.

Overall Assessment

The paper's experiments strongly support a narrower claim: under the specific conditions studied (T5X models, IPO loss, four preference datasets, fixed proxy model, one-pass offline training), on-policy sampling provides a substantial and consistent advantage that cannot be explained by data coverage, data quality, classification accuracy, loss function choice, or model scale. The experiments are carefully designed, the hypothesis-testing methodology is principled, and the negative results are informative.

The paper's broader claim — that these findings reveal a "fundamental necessity of on-policy learning for AI alignment" — is plausible but not definitively established by the experiments alone. The demonstration that specific intuitive explanations fail does not prove that no alternative explanation exists, and the paper's own findings (Chat arena sxs shows a small gap, Bo2 changes the mechanism, proximity to SFT improves offline performance, partial shuffling is tolerated) suggest the phenomenon is more nuanced than a simple "online is necessary" conclusion. The paper's contribution is better characterized as shifting the burden of proof: having eliminated the most obvious explanations for why offline methods might close the gap, the paper establishes that on-policy sampling plays a causal role that is not trivially substitutable, and that practitioners who rely on purely offline alignment should be aware of a performance ceiling that may not be surmountable through better data curation or larger models alone.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Numbers

The compute-optimal framework rests on the ability to estimate prompt difficulty before allocating test-time compute. The method used — generating 2,048 samples per question and averaging either oracle correctness or PRM final-answer scores — is extraordinarily expensive. The authors acknowledge this directly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence is that the reported 4× efficiency gains over best-of-N (Figures 4, 8) are computed after difficulty is known, without amortizing the cost of learning it. The difficulty estimation step alone — generating and scoring 2,048 samples per question — consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. The 4× figure should therefore be understood as an upper bound on achievable efficiency, not a realized deployment gain. A practitioner deploying this system would need to either (a) pay this estimation cost per question, which negates much of the claimed savings, or (b) develop a cheaper alternative that the paper does not provide.

What evidence exists in the paper: The authors flag this explicitly in Section 3.2, framing it as an "exploration-exploitation tradeoff" and noting it as "a key avenue for future work." However, no experiment measures the total cost including difficulty estimation, and no comparison is made against a baseline that spends the same total compute (estimation + solving). Figure 4 shows that predicted (PRM-based) difficulty bins perform similarly to oracle bins, which means the method works without ground-truth labels — but it still requires the full 2,048-sample estimation cost.

Mitigation status: Not addressed experimentally. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) and hints at adaptive approaches where difficulty is assessed from initial samples, but no such method is developed or evaluated. The limitation is acknowledged but unresolved.

6.2 All Results Come from a Single Benchmark and a Single Model Family

Every experiment in the paper uses the MATH benchmark (500 test questions, high-school competition math problems) with PaLM 2-S* (Codey) as the base model. The authors state:

"we believe this model is representative of the capabilities of many contemporary LLMs" (Section 4)

The consequence is that the paper cannot distinguish which findings are fundamental properties of test-time compute scaling and which are artifacts of the specific model, task, or dataset. Several aspects of the results could be model- or domain-specific:

• The PRM's over-optimization behavior (Figure 3, right) depends on PaLM 2-S*'s output distribution and error patterns. A model with different calibration properties or different failure modes might exhibit different difficulty-dependent scaling curves — for instance, a model whose errors are primarily arithmetic slips rather than logical mistakes might benefit more from sequential revisions and less from beam search.
• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (some models are far better at self-correction from examples than others).
• The MATH benchmark consists exclusively of problems with unambiguous correct answers and structured multi-step solutions. The difficulty-dependent patterns identified (beam search hurting easy problems, revisions helping easy problems, no method helping the hardest problems) may not generalize to code generation, logical reasoning, scientific question-answering, or open-ended generation tasks.

What evidence exists in the paper: The paper provides no cross-model or cross-domain replication. The test set of 500 questions is split into five difficulty quintiles of approximately 100 questions each, then further split by two-fold cross-validation — meaning the compute-optimal policy is selected based on roughly 50 questions per fold per bin. The authors do not report confidence intervals on the compute-optimal scaling curves, making it impossible to assess whether the observed gains are statistically robust given this sample size.

Mitigation status: Not addressed. The authors acknowledge the single-model limitation only implicitly (by stating their belief that PaLM 2-S* is representative) and do not discuss the potential for domain-specific effects. No experiments on code, logic, science, or other reasoning domains are reported or proposed as future work.

6.3 The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following what the authors call the LLaMA paradigm. The authors explicitly acknowledge this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Additionally, the 14× larger model is evaluated with greedy decoding — no best-of-N, no majority voting, no search, no revisions.

The consequence is that the FLOPs-matched comparison systematically favors test-time compute. A Chinchilla-optimal model (Hoffmann et al., 2022) trained with 14× more total FLOPs, scaling both parameters and data, would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger than what is tested. Moreover, giving the larger model even a modest test-time compute budget (e.g., best-of-8 or a short revision chain) would create a fairer comparison — one that asks whether test-time compute is more valuable when applied to a smaller model versus a larger model, rather than whether any test-time compute beats a larger model with no test-time compute. The reported advantages (e.g., "+27.8% on easy questions at $R \ll 1$") may shrink substantially or reverse against properly optimized baselines.

What evidence exists in the paper: Section 7 and Figure 9 present the FLOPs-matched results, and the paper is transparent about the parameter-only scaling choice. However, there is no ablation showing how the comparison changes under Chinchilla-optimal pretraining or when the larger model receives a modest test-time compute budget.

Mitigation status: Acknowledged but not resolved. The authors frame the compute-optimal pretraining comparison as future work and justify their choice as "representative of a canonical approach" (Section 7). A practitioner interested in whether to invest in a larger model or better inference strategies would need results against stronger baselines than the paper provides.

6.4 Hard Problems Are Effectively Unsolvable by Test-Time Compute Scaling — No Path Forward Is Offered

Across all methods studied — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show essentially zero improvement regardless of how much test-time compute is allocated. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In Figure 9, the bin 5 scaling line is essentially flat near 0–5% for all $R$ values.

The consequence is that test-time compute can amplify existing capability — making a model more likely to find a correct solution it already has some non-trivial probability of generating — but cannot create capability from nothing. If the base model's pass@1 is near zero on a problem class (genuinely novel reasoning, out-of-distribution tasks, problems requiring knowledge or skills not acquired during pretraining), no amount of search or revision can help, because there are no correct solutions in the proposal distribution to find or refine. This is a fundamental and possibly inescapable limitation: the approach offers no path forward for problems that exceed the base model's competence.

What evidence exists in the paper: The flat or near-flat bin 5 curves across Figures 3, 7, and 9 provide consistent evidence. The authors are candid about this in the Section 7 takeaway box, noting that "on the hardest problems, the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

Mitigation status: None. The paper does not propose any approach for addressing hard problems — this is presented as a fundamental boundary condition rather than a solvable limitation. The practical implication is that organizations deploying this approach need a fallback strategy (human review, routing to larger models, or accepting failure) for problems falling outside the base model's capability range.

6.5 Sequential Revision Strategies Incur Latency Costs Not Captured by the Generation Budget Metric

The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial — each revision depends on the output of the previous one — while parallel best-of-N can be executed simultaneously given sufficient hardware. A compute-optimal strategy that allocates 128 generations as 64 sequential revisions in a single chain (fully sequential) takes approximately 64× longer wall-clock time than one that runs 128 parallel independent samples simultaneously, even though both consume the same FLOPs.

The consequence is that for latency-sensitive applications — interactive assistants, real-time decision-making, customer-facing chatbots — the sequential-heavy strategies favored by the compute-optimal policy on easy-to-medium problems may be impractical regardless of their FLOPs efficiency advantage. The paper's finding that fully sequential revisions are optimal or near-optimal for easy problems (Figure 7, bins 1–2) and that moderate sequential-to-parallel ratios are optimal for medium problems directly conflicts with latency requirements. A practitioner deploying this system interactively might find that the 4× efficiency gain comes with a 10–50× increase in user-perceived wait time.

What evidence exists in the paper: The paper provides no latency analysis. The generation budget metric ($N$) is the sole resource accounting. The hybrid sequential-parallel approach (Figure 5, Appendix I) partially addresses this by allowing some parallelism, but the optimal ratios found in Figure 7 still heavily favor sequential depth — and the latency of the sequential component is never quantified.

Mitigation status: Not addressed. This is a complete omission — the paper does not mention latency, wall-clock time, or throughput as relevant concerns. Given that one of the paper's motivating scenarios is "on-device deployment" (Section 1), where latency is often critical, this gap is significant.

6.6 The Revision Model's 38% Correct-to-Incorrect Reversion Rate Is a Substantial Reliability Problem

The paper reports that approximately 38% of correct answers produced during a revision chain get incorrectly "revised" into wrong answers in the subsequent step (Section 6.1). This is a direct consequence of the training data construction: the model only sees sequences where all in-context answers are incorrect followed by a correct target. At test time, when the model encounters a correct answer in its own context, it has no training signal for what to do — and its learned behavior is to produce a revision, which it does incorrectly, undoing the correct answer.

The consequence is that the revision process is inherently unstable: making the chain longer increases the chance that a correct answer appears early (good) but also increases the chance that it gets subsequently overwritten (bad). The paper mitigates this with majority voting or verifier-based selection across the entire chain — picking the best answer from any revision step rather than always using the last revision. However, these are imperfect patches that rely on the correctness signal being reliably recoverable from the chain. In practice, the 38% reversion rate means that approximately 4 out of 10 correct solutions get destroyed by the revision process, and the system must hope that either the correct answer gets re-generated later in the chain or that the verifier correctly identifies it before it is overwritten.

What evidence exists in the paper: Section 6.1 reports the 38% figure. Figure 6 (left) shows pass@1 gradually improving across revisions, which confirms that correct answers do appear in the chain — but this improvement is net of the destruction rate, meaning the gross improvement per revision step is higher than observed (some gains are lost to reversion). The ReST$^{EM}$ experiment (Appendix K, Figure 16) further highlights fragility: attempting to further optimize the revision model with on-policy RL caused sequential revision performance to degrade substantially, suggesting the revision approach is sensitive to training methodology in ways that are not fully understood.

Mitigation status: Partially addressed. The paper applies majority voting and verifier-based within-chain selection to recover correct answers that get overwritten. These are effective (Figure 6, right, shows sequential revisions still outperform parallel sampling despite the reversion rate), but they are patches rather than solutions — the underlying model behavior is still to sometimes destroy its own correct work. The paper does not propose training the model to recognize when no revision is needed (i.e., to output a no-change prediction), which would be a more direct fix.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around RLHF alignment from an algorithm-centric debate ("DPO vs. PPO") toward a more fundamental question about the learning dynamics induced by on-policy sampling versus static data. Prior work had established that online RLHF works (Ouyang et al., 2022) and that offline methods can approximate it (Rafailov et al., 2023), but the field lacked a mechanistic understanding of why these approaches differ and when the difference matters. This paper fills that gap by establishing that on-policy sampling is not merely a convenient way to collect training data — it triggers a qualitatively different optimization process that cannot be replicated by improving the static dataset.

The most significant conceptual shift is the reclassification of offline alignment from a data problem to a dynamics problem. Before this work, the natural response to offline underperformance was to curate better data: higher-quality responses, clearer preference pairs, broader coverage. The paper's systematic rejection of these hypotheses — through the tandem experiment (Figure 4, identical data coverage fails to close the gap), the high-quality dataset experiment (Figure 5, responses from strong policies do not help), and the classification accuracy analysis (Figure 7, being able to recognize good responses doesn't make you generate them) — eliminates these comfortable explanations. The implication is that no static dataset, however carefully constructed, can replicate the learning trajectory induced by on-policy sampling as long as the offline method remains a single-pass procedure.

This is not a paradigm shift in the sense of introducing a new theoretical framework or algorithmic primitive, but it is a significant reframing of what the alignment community should optimize for. Rather than asking "How do we build better offline datasets?", the paper redirects attention toward "How do we make the training process itself responsive to the policy's evolving distribution?" This reframing elevates concepts like iterative data collection, on-policy distillation, and online-offline hybridization from implementation details to first-class research objectives.

The paper also reconciles conflicting intuitions in the literature. On one side, the impressive empirical results of DPO and IPO in practice (Jiang et al., 2023; Tunstall et al., 2023) suggested offline methods might be sufficient. On the other side, the persistent advantage of online methods observed by practitioners and in controlled studies (Calandriello et al., 2024; Xu et al., 2024) suggested online sampling matters. The paper's resolution — that offline methods are effective at improving discriminative capability but this translates poorly to generative improvement (Section 5.3, Figure 8) — explains both observations. Offline methods can produce policies that look good when evaluated on classification-style metrics (preference prediction accuracy) or when fine-tuned on tasks where the SFT policy is already strong, but they hit a performance ceiling that online methods do not because they improve through an indirect mechanism (suppressing losers rather than promoting winners).

The paper identifies verifier over-optimization as a primary bottleneck for online methods (Figure 1, the inverted-U shape; Figure 6, the proxy model's declining accuracy) and discriminative-generative misalignment as the bottleneck for offline methods (Figure 8). This refocuses research attention in both directions: for online methods, the priority is building more robust proxy preference models that remain calibrated under distribution shift; for offline methods, the priority is designing loss functions or training procedures where discriminative improvement on the training data actually correlates with generative quality.

What becomes more attractive as a research direction:

• Iterative offline methods where the dataset is periodically refreshed with on-policy samples, blurring the boundary between online and offline. The paper's finding that small amounts of shuffling are tolerated (Appendix G, Figure 13) and that proximity to SFT is key (Figure 12) suggests that even coarse periodic updates might capture much of the online benefit.
• Hybrid architectures that separate discriminative and generative capabilities — training a strong preference classifier independently and using it to guide a separate generation policy, rather than forcing the same network to excel at both.
• Understanding the loss function properties that correlate discriminative training with generative improvement, since the paper shows this correlation is near-zero for IPO (Figure 7) but might exist for other losses.

What becomes less attractive:

• Purely static, single-pass offline alignment as a complete solution for generative quality improvement. The paper demonstrates a clear performance ceiling across four diverse datasets that cannot be breached by data curation or hyperparameter tuning.
• Explanations of the online-offline gap that appeal exclusively to data coverage or data quality. The paper provides concrete counterexamples that should shift the burden of proof to those claiming these factors are sufficient.

Follow-Up Research This Work Enables

Characterizing the data properties that control the online-offline gap. The paper observes that the gap varies substantially across datasets — larger for OpenAI summarization and Anthropic helpfulness, smaller for Anthropic harmlessness and Chat arena sxs (Figure 1). However, the paper does not systematically characterize what dataset properties modulate the gap. A strong follow-up study would construct synthetic preference datasets where properties like response diversity (measured by embedding variance), preference strength (measured by golden model preference margin), prompt complexity (measured by response length or number of reasoning steps), and topic coherence are independently varied. The specific prediction is that datasets with weaker preference signals (responses are nearly equally good, making classification harder) will show larger online-offline gaps because offline methods' indirect improvement mechanism (suppressing losers) becomes less effective when losers are nearly as good as winners. This would explain the smaller gap on Chat arena sxs (where responses may be more clearly differentiated) and generate actionable guidance: practitioners deploying offline alignment should measure their preference dataset's "gap strength" to predict whether offline methods will suffice.

Scaling the proxy preference model alongside the policy. The paper's scaling experiments (Section 5.5) keep the proxy model fixed at 770M parameters regardless of policy size, which confounds policy scaling with proxy signal quality degradation. Gao et al. (2023) showed that larger reward models reduce over-optimization; the natural extension is to test whether online methods maintain their advantage over offline methods when the proxy model scales proportionally. Concretely, replicate Figure 10 with XL proxy models for XL policies and XXL proxy models for XXL policies. The hypothesis — consistent with the paper's framework but untested — is that a larger proxy model extends the online over-optimization peak to higher KL values and higher absolute performance, potentially widening the online-offline gap at scale. If instead the gap narrows because both methods become bottlenecked by the same fundamental limitation (the golden preference signal itself), that would suggest a ceiling beyond which neither online sampling nor larger proxy models help.

Training policies to recognize when no revision is needed. The paper identifies a 38% correct-to-incorrect reversion rate in the revision model (Section 6.1) but does not experiment with teaching the model to output a "no change" token when the current answer is already correct. A concrete follow-up would modify the revision training data to include trajectories where a correct answer appears in the context followed by the same correct answer as target (not a different correct answer), teaching the model that the appropriate response to correctness is to preserve it. This is challenging because the base training procedure (Section 6.1, edit-distance-based pairing) specifically selects incorrect answers — a balanced dataset would need to include ~40% no-change examples to neutralize the reversion rate. The evaluation would measure whether the reversion rate decreases from 38% toward zero without degrading the improvement rate on incorrect answers, and whether within-chain selection (the current mitigation) becomes unnecessary.

Testing whether discriminative-generative correlation can be engineered through auxiliary losses. The paper's central mechanistic finding is that classification accuracy on a static dataset has near-zero correlation with generative performance for both IPO and Bo2 losses (Figure 7). A constructive follow-up would explore whether adding an auxiliary loss that explicitly ties classification accuracy to generative quality changes this dynamic. For example, adding a KL divergence penalty between the policy's output distribution and the SFT policy's output distribution, conditioned on correctly classified training pairs — effectively telling the policy "stay close to the SFT distribution when you can't distinguish good from bad, but diverge when you can." The experiment would train offline IPO with this auxiliary loss, measure the correlation between classification accuracy and win rate at each checkpoint (analogous to Figure 7), and test whether the offline KL-performance trade-off curve (analogous to Figure 1) shifts upward. A positive result would suggest that the discriminative-generative misalignment is not fundamental to contrastive losses but rather a consequence of the particular loss formulation, opening the door to offline methods that actually translate classification improvement into generation improvement.

Investigating whether decoder-only architectures exhibit the same gap. All experiments use T5X encoder-decoder models. A replication study using decoder-only architectures (GPT-style, Llama-style) with the same experimental design — golden preference model, proxy model, $\mathcal{D}_{\text{golden}}$, IPO loss, four datasets, KL-performance trade-off curves — would test the generality of the paper's central claim. Encoder-decoder models process the prompt and response jointly in the encoder, potentially giving them an inductive bias toward classification-style tasks that decoder-only models lack. The specific hypothesis is that decoder-only models might show an even larger online-offline gap because their autoregressive generation objective is less naturally aligned with pairwise classification. Alternatively, if the gap is similar or smaller, the paper's findings would be validated as architecture-independent, strengthening the "fundamental necessity" interpretation.

Practical Applications and Downstream Use Cases

Iterative data collection for offline alignment in resource-constrained deployments. Many organizations fine-tune open-source models on curated preference datasets without the infrastructure for online RLHF (generation servers, reward model inference loops). The paper's key practical finding — that proximity to the SFT policy is the most important property of offline data (Section 6, Figure 12) — suggests a concrete workflow improvement: rather than collecting a single high-quality preference dataset (which may be far from the starting policy's distribution), practitioners should plan for 2–3 rounds of data collection interspersed with training. Round 1 uses data generated near the SFT policy's distribution (from the SFT model itself or very early checkpoints). Round 2 collects fresh data from the improved policy after Round 1 training. Round 3 (optional) repeats. The paper shows that $\mathcal{D}_{\text{sft vs. 800}}$ (data with one side from SFT) matches or exceeds $\mathcal{D}_{\text{golden}}$ performance despite lower absolute response quality, and Appendix G Figure 13 shows robustness to small amounts of off-policy noise. This iterative approach captures the temporal coherence between data distribution and policy state that the paper identifies as the critical mechanism, without requiring the infrastructure complexity of per-step on-policy sampling.

Design of cost-effective proxy preference models for alignment infrastructure. Organizations building RLHF pipelines face a resource allocation decision: invest compute in a larger preference/reward model or in more policy training steps. The paper's proxy model analysis (Figures 6–7) provides empirical guidance. The proxy preference model (Large, 770M) maintained 70–80% classification accuracy on early online data but degraded to ~60% as the policy distribution shifted (Figure 6). This degradation directly limits online training — the policy begins over-optimizing against an increasingly unreliable signal. The practical implication is that periodically retraining or updating the proxy model (e.g., after every $K$ policy updates, using fresh on-policy data labeled by human raters or a larger model) may be more cost-effective than training a single larger proxy model upfront. The paper doesn't test this directly, but the declining accuracy curve in Figure 6 provides the diagnostic signal: practitioners should monitor proxy model accuracy on recent policy samples and trigger retraining when it drops below a threshold (e.g., 65–70%). This operationalizes the "exploration-exploitation tradeoff" the paper discusses in a different context (Section 3.2).

Offline alignment as a warm-start phase before online fine-tuning. The paper's finding that $\mathcal{D}_{\text{sft vs. 800}}$ — a dataset where one response side is generated near the SFT policy — enables solid initial improvement in offline mode (Figure 12) suggests a hybrid deployment strategy. An organization could perform offline alignment on SFT-proximate data to efficiently reach a strong baseline, then transition to online fine-tuning only for the final performance push. At the offline stage, the policy would improve from (say) 50% to 60% win rate against the golden baseline at low computational cost. The online stage would then push from 60% to 70–75%, which is the regime where the paper shows online methods' advantage is largest (Figure 1, the gap between online and offline curves widens at intermediate KL values). This hybrid approach avoids the cost of running online RLHF from scratch (when the policy's samples are still low-quality and the proxy model feedback is least useful) while still capturing the peak performance gains that only online methods can achieve. The paper's Figure 1 provides the calibration: offline methods achieve good trade-offs at low KL (2–5 nats), while online methods pull away at moderate KL (5–15 nats). A practitioner would set the offline-to-online transition point at the KL where the offline curve begins to plateau relative to the online curve — approximately 5–10 nats on most datasets in Figure 1.