RL Training2023~114 minarXiv:2309.06657

Statistical Rejection Sampling Improves Preference Optimization

score 9Tianqi Liu, Yao Zhao, Rishabh Joshi +4↑ 15Sep 13, 2023cs.CL

🎯 Pitch

Existing direct preference optimization methods suffer a critical distribution mismatch: they estimate the optimal policy on data sampled from the suboptimal SFT model, not from the target policy itself. RSO fixes this with statistical rejection sampling, using an explicit reward model to generate on-policy preference pairs from the estimated optimal policy, and the resulting policy beats DPO by a large margin, with human raters preferring RSO over 2× more often.

1. Executive Summary

This paper introduces Statistical Rejection Sampling Optimization (RSO), a framework that improves preference optimization for language models by sourcing preference pairs from the estimated target optimal policy rather than from a suboptimal sampling distribution. Using T5-large (770M) and T5-XXL (11B) models on Reddit TL;DR summarization and AnthropicHH dialogue, RSO combines a pairwise reward-ranking model with statistical rejection sampling to generate response pairs from the approximated optimal policy π_rψ (accepting or rejecting SFT-policy samples according to their reward scores under a tunable KL-regularization parameter β), then trains the policy on these pairs via either sigmoid loss—DPO's logistic regression—or the paper's improved hinge-norm loss—an SVM counterpart to DPO derived from SLiC's hinge loss. On Reddit TL;DR, RSO with sigmoid-norm loss achieves 71.86% AutoSxS win rate versus DPO's 67.72%, and in human evaluation RSO is chosen as preferred more than 2× as often as DPO (48% vs. 21%), establishing that using an explicit reward model to sample preference pairs from a distribution closer to the optimal policy yields substantially better alignment than directly optimizing on off-policy human preference data.

2. Context and Motivation

The Core Problem: Preference Data Is Almost Always Off-Policy

The fundamental gap this paper addresses is a mismatch between statistical theory and practical reality in preference-based alignment of language models. When we frame preference optimization as density estimation—learning the optimal policy π* that maximizes a KL-regularized reward objective—the maximum likelihood estimator (MLE) requires preference pairs sampled from π* itself. In practice, however, the human preference data we actually have (D_hf) is collected from a mixture of unknown, suboptimal policies: the SFT model, previous RLHF checkpoints, other agents, or human-written responses that don't come from any single policy at all.

This isn't a minor implementation detail. It means that DPO and SLiC—the two leading offline preference optimization methods at the time of this paper—are fitting their loss functions on data that is statistically mismatched to the distribution they're trying to estimate. The paper frames this as the central problem:

"DPO uses the collected human preference data from other policies directly in all the experiments and lacks a study on the effect of sampling. Although they propose to sample pairs from the SFT policy and get them labeled by human, it is still not strictly MLE for the preference model due to the mismatch between the sampling distribution and π*."

This is a distribution shift problem at the level of the training objective itself. The consequence is that the learned policy π_θ is an imperfect estimate of the true optimal policy π*, with error that depends on how far the data-generating distribution is from π*.

Why This Matters: Alignment Quality Depends on the Data Distribution

The practical significance of this gap is substantial. RLHF—which RL-free methods like DPO aim to replace—is notoriously complex: it requires simultaneously maintaining a policy model, a value model, a reward model, and a reference model, which is memory-intensive, unstable during training, and limits the maximum feasible model size. Offline methods like DPO and SLiC solve these engineering problems by eliminating the need for online sampling and value function estimation, making preference optimization dramatically simpler and more scalable.

But if these simpler methods are fundamentally limited by the quality of their training data distribution—if they cannot estimate π* as accurately as they could with on-policy data—then we're trading away alignment quality for engineering convenience. The paper's central claim is that this tradeoff is unnecessary: we can have the simplicity of offline methods while still training on preference pairs that come from a distribution close to π*, by using a reward model to guide rejection sampling from the SFT policy.

This is theoretically significant because it connects the practical algorithm design to basic principles of statistical estimation. The Bradley-Terry preference model—the foundation of both DPO and RLHF—assumes that the preference probability between two responses depends only on their true reward values. But the estimation of those rewards from finite preference data depends on the sampling distribution. If you sample pairs from a policy that rarely produces good responses, you'll have few examples of high-quality outputs in your training data, and your estimate of π* will be imprecise in the regions that matter most.

Where DPO Falls Short

DPO (Rafailov et al., 2023) made a significant contribution by showing that the optimal policy for the KL-regularized reward maximization objective can be expressed in terms of only the policy and reference model, without an explicit reward function:

$p^*(y_1 \succ y_2 \mid x) = \frac{1}{1 + \exp\left(\beta \log \frac{\pi^*(y_2|x)}{\pi_{\text{sft}}(y_2|x)} - \beta \log \frac{\pi^*(y_1|x)}{\pi_{\text{sft}}(y_1|x)}\right)}$

This allowed DPO to train directly on human preference data using a simple sigmoid loss on normalized likelihood ratios, eliminating the reward model entirely. The paper's famous framing is that "the language model is secretly a reward model"—you can read off the implicit reward from the policy's output probabilities.

But this framing papers over a critical statistical subtlety that RSO identifies. Equation (2) is a specification of the preference probability in terms of π*, but it does not specify what distribution the training pairs should come from. If you fit this model using pairs drawn from some other policy π_unk, you are not performing MLE for π*; you are solving a misspecified estimation problem. The estimator is still consistent under certain conditions, but its finite-sample efficiency—how many samples you need to get a good estimate—can be arbitrarily worse when the data distribution is far from π*.

The paper makes this point explicitly:

"Statistically speaking, since we are estimating π*(y|x), it is desired to draw samples from π*(y|x)."

DPO's experiments use either the original human preference data (collected from unknown policies) or, in the proposed extension, pairs sampled from the SFT policy and labeled by humans. Both are off-policy relative to π*. DPO never studies what happens if you could actually sample from the optimal policy, leaving open the question of how much performance is left on the table due to this distribution mismatch.

An additional limitation: DPO has no mechanism to incorporate reward model information beyond what is implicitly encoded in the preference labels. If you have a well-trained reward model that can evaluate arbitrary response pairs, DPO provides no way to use it for generating new training data. The reward model's knowledge is locked inside the preference labels it was trained on, and the policy cannot benefit from the reward model's ability to compare novel response pairs.

Where SLiC Falls Short

SLiC (Zhao et al., 2022; 2023) takes a different approach: it uses a contrastive ranking calibration loss with a margin δ:

$L(\theta) = \max(0, \delta - \log \pi_\theta(y_w|x) + \log \pi_\theta(y_l|x)) - \lambda \log \pi_\theta(y_{\text{ref}}|x)$

SLiC can operate either directly on human preference data or on pairs sampled from the SFT policy and ranked by a reward model (the "sft-sample-rank" approach). However, both strategies share the same fundamental limitation as DPO: the preference pairs come from a suboptimal proposal distribution.

The paper identifies two specific issues with SLiC:

First, the loss function itself lacks a clear statistical interpretation. While DPO's sigmoid loss corresponds to logistic regression under the BT model, SLiC's hinge loss with SFT regularization was proposed more heuristically. This makes it hard to understand what exactly SLiC is optimizing or how its hyperparameters should be set. The paper addresses this by showing that SLiC's loss is almost equivalent to a support vector machine with hinge loss on normalized likelihood ratios, providing a principled statistical interpretation that had been missing.

Second, SLiC's "sft-sample-rank" approach—sampling response pairs from the SFT policy and ranking them via a reward model—is still off-policy. The SFT policy produces responses that are generally lower-quality than what the optimal policy would produce, so the training pairs are concentrated in a region of the output space where the optimal policy has relatively low probability density. This means the learned policy may be inaccurate precisely where it matters most—in the high-quality response region that π* concentrates on.

The Reward Model Question: Is Explicit Better Than Implicit?

Underlying both DPO's and SLiC's limitations is a deeper question that the paper engages with: Is it better to fold the reward model into the policy (as DPO does) or to maintain an explicit reward model (as SLiC does)?

DPO's position is that the language model is the reward model, so no separate reward model is needed. This is elegant—one less model to train and maintain—and it eliminates the possibility of mismatch between the reward model and the policy.

The RSO paper pushes back on this, arguing that comparing between two responses (reward discrimination) is fundamentally easier than generating high-quality responses (policy generation), and therefore a separate reward model can provide useful signal that gets lost when it's absorbed into the policy. The paper states this directly:

"DPO claims that the language model is secretly a reward model, we show that the language model learns better from an explicit reward model because comparing between two responses (reward) is easier to learn than generating high quality responses (policy)."

This is a claim about the relative difficulty of different learning problems. A pairwise reward-ranking model only needs to learn to discriminate—it outputs a single token ("A" or "B") and can achieve high accuracy with a relatively modest number of parameters. A language model policy needs to generate entire sequences autoregressively, which is a much harder learning problem. By keeping the reward model explicit, RSO can use a powerful 11B-parameter T5-XXL for reward modeling while training a 770M-parameter T5-large for the policy, getting better reward discrimination than the policy could achieve internally.

This argument has important practical implications. If discrimination is easier than generation, then for a fixed compute budget, it may be better to invest in a strong reward model (which then guides a weaker policy via rejection sampling) rather than trying to make the policy simultaneously good at both generation and self-evaluation.

The "Rejection Sampling" Terminology Gap

The paper also identifies a terminological confusion that has obscured the relationship between statistical rejection sampling and common RLHF practices. In the RLHF literature (Bai et al., 2022; Touvron et al., 2023; Stiennon et al., 2020), "rejection sampling" typically refers to best-of-N: sample N completions from a policy, score them with a reward model, and return the highest-scoring one. This is a form of extreme exploitation—it selects purely based on the reward model's ranking with no regularization toward the SFT policy.

The RSO paper clarifies that this best-of-N approach is actually a special case of statistical rejection sampling with β → 0—where β is the temperature parameter in the KL-constrained reward maximization objective. When β → 0, the acceptance probability in Algorithm 1 becomes 1 only for the maximum-reward response and 0 for all others, reducing to best-of-N. When β → ∞, all samples are accepted, reducing to the SFT policy. The key insight is that best-of-N trusts the reward model too much ("reward hacking") because it has no regularization, and the paper's statistical rejection sampling allows tuning β to balance between reward exploitation and staying close to the SFT policy.

This conceptual reframing is significant because it connects a heuristic practice (best-of-N) to a principled statistical framework (rejection sampling from π_r as defined by the KL-regularized objective) and reveals the hyperparameter β as the crucial knob controlling this tradeoff. Prior work had not made this connection explicit, treating best-of-N as a separate technique rather than as a limiting case of a broader sampling methodology.

How RSO Positions Itself

The paper positions RSO not as a replacement for DPO or SLiC, but as a better data generation strategy that can be combined with either loss function. The core insight is that the loss function (sigmoid-norm for logistic regression, hinge-norm for SVM) and the preference data distribution are independent axes of the preference optimization problem, and prior work had only explored suboptimal points in this space.

Specifically, the paper maps out a 3×3 grid of design choices:

Loss function: sigmoid-norm (DPO's logistic regression), hinge (original SLiC), or hinge-norm (improved SVM-style SLiC)
Preference data source: direct (human data from unknown policies), sft-sample-rank (SFT policy samples ranked by reward model), or rso-sample-rank (samples from the approximate optimal policy π_rψ via rejection sampling, ranked by reward model)

DPO occupies one cell in this grid (sigmoid-norm + direct), and SLiC occupies another (hinge + sft-sample-rank, approximately). RSO argues that these are not the best cells, and that moving to the rso-sample-rank column yields consistent improvements regardless of which loss function is used. The paper's contribution is therefore both methodological (the statistical rejection sampling algorithm for generating on-policy preference pairs) and analytical (the unified framework revealing the independence of loss and sampling distribution).

The relationship to online RLHF is also worth clarifying. Online RLHF samples from the current policy during training, which is approximately π* if training is converging well. So in a sense, online RLHF already has the benefit of on-policy preference data. But it pays for this with the complexity and instability of RL. RSO aims to get the benefit of on-policy data without the cost of online RL, using a frozen reward model and one round of rejection sampling to approximate π*-distributed pairs offline. This is a middle ground: better data than DPO/SLiC's off-policy pairs, simpler training than online RLHF.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

RSO is a pipeline for generating better training data for preference-based alignment of language models. The system takes a base SFT policy model, a human preference dataset collected from some unknown mixture of policies, and a set of prompts, then produces preference pairs that are statistically closer to what the optimal policy would generate, which are subsequently used to train the final aligned policy via either logistic regression (DPO-style) or SVM-style loss.

The core problem it solves is a distribution mismatch in preference optimization: DPO and SLiC train on preference pairs that come from suboptimal policies (either the SFT model or unknown human data sources), but the statistical theory underlying both methods assumes pairs are sampled from the optimal policy π* itself. RSO's solution shape is: train an explicit pairwise reward model → use it to guide rejection sampling from the SFT policy (accepting responses with probability proportional to their reward, controlled by a temperature β) → construct preference pairs from the accepted samples → train the policy on these on-policy-like pairs. The reward model never directly updates the policy; instead, it acts as a filter that reshapes the SFT policy's output distribution into something closer to π*.

3.2 Big-picture architecture (diagram in words)

The RSO system has four major components connected in a linear pipeline:

Supervised Fine-Tuned Policy (π_sft) — a T5-large (770M) or T5-XXL (11B) language model fine-tuned on task-specific supervised data. This serves as both the proposal distribution for sampling response candidates and the starting point (reference policy) for preference optimization. It generates n_c = 64 response candidates per prompt at temperature 0.7 with top-k=40.
Pairwise Reward-Ranking Model (ρ_ψ) — a T5-XXL (11B) text-to-text model trained on the human preference dataset D_hf to discriminate between response pairs. Given a prompt x and two responses y_1, y_2, it outputs either "A" or "B" to indicate which is preferred. Its decoding probability for "A" estimates P(y_1 ≻ y_2 | x). From this pairwise model, a pointwise reward score r_ψ(x, y) is derived by comparing any response y against a randomly chosen baseline from the SFT policy (which is assigned a reward of 0 by convention). This component has 73.23% validation accuracy on summarization and 69.75% on dialogue.
Statistical Rejection Sampling Module — a sampling algorithm that takes the n_c = 64 SFT-generated candidates, scores each with the derived pointwise reward r_ψ(x, y), and sub-selects n_d = 8 responses by accepting each candidate with probability exp((r - r_max) / β), where r_max is the maximum reward among not-yet-accepted candidates and β = 0.5 controls the tradeoff between reward exploitation and staying close to the SFT distribution. This produces response sets distributed approximately according to the KL-regularized optimal policy π_rψ.
Preference Optimization Module — constructs preference pairs from the 8 accepted responses via first-round ranking (4 pairs per prompt, labeled by the reward model), then trains the policy model π_θ on these pairs using either sigmoid-norm loss (logistic regression, same form as DPO) or hinge-norm loss (SVM-style, the paper's improved version of SLiC). Training uses Adafactor optimizer with learning rate 1e-5, batch size 32, and early stopping based on the reward model's win rate against SFT targets.

Information flow: Human preference data → train pairwise reward model → SFT policy generates 64 candidates per prompt → reward model scores all candidates against a baseline → rejection sampling selects 8 responses proportionally to exp(r/β) → reward model labels 4 chosen-vs-rejected pairs → policy model trains on these pairs via sigmoid-norm or hinge-norm loss → final aligned policy.

3.3 Roadmap for the deep dive

First: The KL-constrained reward maximization objective and its optimal solution — because the entire RSO framework is built on solving this objective, and understanding what π_r represents is prerequisites for the rejection sampling algorithm.
Second: The pairwise reward-ranking model — how it is trained, how pointwise rewards are derived from pairwise comparisons, and why an explicit reward model is maintained rather than absorbed into the policy.
Third: The statistical rejection sampling algorithm — the core methodological contribution, including the acceptance criterion, the role of β, the sampling-without-replacement procedure, and the proof that accepted samples follow the target distribution.
Fourth: Loss functions and the unified preference optimization framework — how sigmoid-norm (DPO) and hinge-norm (improved SLiC) are derived from the BT preference model, and why they represent logistic regression and SVM, respectively, on normalized likelihood ratios.
Fifth: The three preference data sources (direct, sft-sample-rank, rso-sample-rank) — what each represents in terms of statistical estimation quality, and why rso-sample-rank is closest to the MLE-optimal sampling distribution.
Sixth: The full training recipe and hyperparameter configuration — bringing all components together into the end-to-end pipeline with all specific numbers and design choices.

3.4 Detailed, sentence-based technical breakdown

This is primarily a methodological paper whose core idea is that preference optimization can be significantly improved by using an explicit reward model to sample training pairs from a distribution closer to the optimal policy, and that this improvement is independent of whether one uses DPO's logistic regression loss or an improved SVM-style loss.

The KL-Constrained Reward Maximization Objective

The foundation of RSO—and indeed of DPO, RLHF, and most modern alignment methods—is a single objective function that formalizes what we want from an aligned language model.

The objective: Given a reward function r(x, y) that scores how good a response y is for a prompt x, and a reference policy π_sft(y|x) (the supervised fine-tuned model that we want to stay close to), RLHF and DPO both optimize:

$\max_\pi \mathbb{E}_{x \sim \mathcal{P}, y \sim \pi} \left[ r(x, y) \right] - \beta \, D_{\text{KL}} \left( \pi(y|x) \, \| \, \pi_{\text{sft}}(y|x) \right)$

where P is the distribution over prompts, π(y|x) is the policy being optimized, and D_KL is the Kullback-Leibler divergence measuring how far π diverges from π_sft.

What it computes: This objective maximizes a weighted sum of two competing goals. The first term E[r(x,y)] drives the policy toward responses with high reward—it wants the policy to generate outputs that humans (or a reward model) prefer. The second term -β D_KL(π || π_sft) penalizes the policy for straying too far from the reference SFT model—it ensures the policy doesn't collapse to a few high-reward but degenerate outputs, preserving the diversity and fluency the SFT model learned from supervised data. The hyperparameter β controls the tradeoff: when β → 0, reward dominates and the policy puts all probability mass on the single highest-reward response (pure exploitation); when β → ∞, the KL penalty dominates and the optimal policy is identical to π_sft (pure exploration with no alignment).

Why this form: The KL penalty serves as both a regularizer and a constraint. Without it, the maximization is unbounded—the policy could assign infinite probability to the highest-reward response, which would be a degenerate distribution. The KL penalty keeps the policy in a neighborhood of π_sft, ensuring that aligned outputs remain fluent and diverse rather than becoming reward-hacked nonsense. This specific combination of expected reward minus KL divergence has a closed-form optimal solution (shown below), which makes it analytically tractable in a way that other regularization schemes are not. The parameter β maps directly to the temperature in the optimal policy expression, giving it a clear operational meaning.

The optimal policy: DPO's key contribution (which RSO builds on) is proving that this objective has a closed-form solution. The policy π_r that maximizes the above objective for a given reward function r is:

$\pi_r(y|x) = \frac{1}{Z(x)} \pi_{\text{sft}}(y|x) \exp\left( \frac{1}{\beta} r(x, y) \right)$

where Z(x) = Σ_y π_sft(y|x) exp(r(x,y)/β) is the partition function—a normalization constant ensuring π_r sums to 1 over all possible responses.

What it computes: This equation says: to get the optimal policy, start with the SFT policy's probability for each response y, then re-weight it by exp(r(x,y)/β). High-reward responses get their probability multiplied by a large factor; low-reward responses get their probability multiplied by a small factor. The partition function Z(x) then rescales everything so that probabilities sum to 1. The result is a new distribution π_r that is proportional to the SFT distribution times an exponential reward boost—responses that the SFT model already liked AND that have high reward get the most probability mass.

Why this form matters for RSO: This equation is the bridge between reward models and policy optimization. It says that if you can compute reward scores r(x,y) for any response y, you can in principle construct π_r by re-weighting π_sft. RSO's statistical rejection sampling is exactly a way to generate samples from π_r without needing to compute the intractable partition function Z(x)—by using π_sft as a proposal distribution and accepting/rejecting samples based on exp(r(x,y)/β). The equation also makes precise what the reward model needs to provide: not an absolute score, but scores that can be compared across responses via their exponentiated values.

The inverse relationship—reward from policy: Rearranging the optimal policy equation yields DPO's key insight:

$r(x, y) = \beta \log \frac{\pi_r(y|x)}{\pi_{\text{sft}}(y|x)} + \beta \log Z(x)$

where Z(x) is the partition function from the forward direction.

What it computes: This equation reads a reward score out of any policy π_r by comparing its log-probability for a response against the SFT model's log-probability, plus an additive constant that depends only on the prompt x. The term β log(π_r/π_sft) is the core signal: it measures how much the aligned policy up-weights or down-weights a response relative to the base SFT model.

Why this form matters: This is what enables DPO to eliminate the explicit reward model entirely. Since the Bradley-Terry preference model only depends on reward differences between two responses, the β log Z(x) term cancels out—it's the same for both y_1 and y_2 because it depends only on x. This means preference probabilities can be expressed purely in terms of policy probability ratios, giving Equation (2) from the introduction. RSO does NOT eliminate the reward model—it keeps it explicit—but this inverse relationship is what allows RSO to frame the preference optimization step as fitting a policy whose implicit rewards match the reward model's explicit judgments.

The Pairwise Reward-Ranking Model: Training and Pointwise Score Derivation

RSO maintains an explicit reward model rather than folding it into the policy (as DPO does). This section covers how this model is built and how it generates the scalar reward scores needed for rejection sampling.

Model architecture and task formulation: The reward model is a T5-XXL (11B parameter) text-to-text transformer. Rather than outputting a scalar reward, it is trained as a pairwise discriminator: given a prompt x and two responses y_1, y_2, it outputs either the token "A" (preferring y_1) or "B" (preferring y_2). The model takes formatted text input:

For summarization: "[CONTEXT] {x} [SUMMARY A] {y1} [SUMMARY B] {y2}"
For dialogue: "[CONTEXT] {x} [RESPONSE A] {y1} [RESPONSE B] {y2}"

The probability that the model decodes "A" (versus "B") is treated as the estimated preference probability ρ_ψ(x, y_1, y_2) ≈ P(y_1 ≻ y_2 | x). To remove positional bias—where the model might systematically favor response A regardless of quality—response pairs and their associated labels are randomly flipped during training.

Training data and procedure: The reward model is trained on the human preference dataset D_hf = {(x^(i), y_w^(i), y_l^(i))}, where y_w is the preferred (winning) response and y_l is the losing response according to human raters. This is the same dataset that DPO would train on directly. The model is trained via standard sequence-to-sequence cross-entropy, learning to output the correct token ("A" if y_w is in position A, "B" otherwise). The paper reports validation accuracy of 73.23% on Reddit TL;DR and 69.75% on AnthropicHH.

Deriving pointwise rewards from pairwise comparisons: The rejection sampling algorithm requires a scalar reward for each individual response, not a pairwise preference. The paper derives these pointwise scores using a baseline trick. Let y_b be an arbitrary baseline response (in practice, a randomly chosen sequence from the SFT policy). By convention, assign r(x, y_b) = 0. Then for any response y, from the Bradley-Terry model:

$r_\psi(x, y) = \text{logit}(\rho_\psi(x, y, y_b)) = \log\left( \frac{\rho_\psi(x, y, y_b)}{1 - \rho_\psi(x, y, y_b)} \right)$

where ρ_ψ(x, y, y_b) is the reward model's estimated probability that y is preferred over the baseline y_b, and logit(p) = log(p/(1-p)) is the log-odds transform.

What it computes: This converts a pairwise comparison probability into a pointwise reward score on the log-odds scale. When ρ_ψ = 0.5 (response y is equally preferred to baseline y_b), logit(0.5) = 0, giving r_ψ = 0—the baseline's own score. When ρ_ψ > 0.5 (y is preferred over baseline), r_ψ is positive. When ρ_ψ < 0.5, r_ψ is negative. The logit transform maps the (0,1) probability range to (-∞, +∞), producing an unbounded reward score.

Why this form: The logit transform is the correct mapping because the Bradley-Terry model specifies that the log-odds of preference equals the difference in rewards. Specifically, logit(P(y ≻ y_b)) = r(x,y) - r(x,y_b) = r(x,y) - 0 = r(x,y). Using the raw probability ρ_ψ directly as a reward (without the logit transform) would incorrectly compress reward differences—a 0.99 vs. 0.90 probability gap represents a much larger true reward difference than a 0.55 vs. 0.46 gap, but this is only captured on the log-odds scale. The baseline choice y_b sets the zero point of the reward scale; any consistent baseline produces reward scores that differ by an additive constant, and additive constants cancel out in the rejection sampling acceptance criterion (which depends only on reward differences, specifically r - r_max).

Why maintain an explicit reward model: The paper argues for explicit reward modeling on the grounds that discrimination is easier than generation. An 11B-parameter T5-XXL reward model only needs to decode a single "A" or "B" token—it can be highly accurate at judging response pairs because it focuses all its capacity on a simple binary classification problem. The policy model, by contrast, must generate entire multi-token sequences autoregressively, which is a far harder learning problem at equivalent scale. By keeping the reward model separate and larger than the policy (11B vs. 770M), RSO gets high-quality reward discrimination that then guides the policy through data construction, rather than asking the policy to simultaneously learn both generation and self-evaluation. This is a form of asymmetric scaling—spend more parameters on the discriminator than the generator because discrimination is the easier problem with higher marginal returns to scale.

Statistical Rejection Sampling: The Core Algorithm

This is the central methodological contribution of the paper. Statistical rejection sampling (Neal, 2003) is a general technique for generating samples from a target distribution π_target using samples from a simpler proposal distribution π_proposal and an acceptance criterion. RSO applies this to sample from π_rψ (the approximate optimal policy) using π_sft as the proposal.

The theoretical setup: Recall from the optimal policy equation that the target distribution is:

$\pi_{r_\psi}(y|x) = \frac{1}{Z_\psi(x)} \pi_{\text{sft}}(y|x) \exp\left( \frac{1}{\beta} r_\psi(x, y) \right)$

where Z_ψ(x) = Σ_y π_sft(y|x) exp(r_ψ(x,y)/β) is the (intractable) partition function. This equation defines the target distribution up to the unknown constant Z_ψ(x). Rejection sampling is perfectly suited for this situation—it can sample from π_rψ using only the unnormalized density ratio π_rψ(y|x) / π_sft(y|x) = exp(r_ψ(x,y)/β) / Z_ψ(x).

The acceptance criterion: To sample from π_rψ, the standard rejection sampling procedure requires finding a constant M such that M · π_sft(y|x) ≥ π_rψ(y|x) for all y. The optimal (smallest) such M for the not-yet-accepted set D_x is:

$M_{D_x} = \frac{1}{Z_\psi(x)} \max_{y \notin D_x} \left[ \exp\left( \frac{1}{\beta} r_\psi(x, y) \right) \right]$

The acceptance probability for a candidate y is then:

$\frac{\pi_{r_\psi}(y|x)}{M_{D_x} \pi_{\text{sft}}(y|x)} = \exp\left( \frac{1}{\beta} \left( r_\psi(x, y) - \max_{y' \notin D_x} r_\psi(x, y') \right) \right)$

What this computes: For each candidate response y sampled from π_sft, compute its reward r_ψ(x,y), find the maximum reward r_max among all candidates not yet accepted, and accept y with probability exp((r_ψ(x,y) - r_max) / β). The highest-reward candidate (where r_ψ = r_max) is always accepted with probability exp(0) = 1. Candidates with rewards below r_max are accepted with probability less than 1, decreasing exponentially with the gap r_max - r_ψ(x,y). The divisior Z_ψ(x) cancels out in the ratio, meaning the algorithm never needs to compute the intractable partition function.

Why this form: The acceptance probability depends only on the difference between the candidate's reward and the maximum available reward, scaled by 1/β. This means the algorithm is agnostic to the absolute reward scale—only relative reward differences matter. The subtraction of r_max in the exponent follows from using the tightest possible bound M_{D_x}, which minimizes the rejection rate. Without adjusting M to exclude already-accepted responses, the bound would be looser and more candidates would be rejected. The autoregressive update of r_max as responses are accepted ensures that in every round, at least one candidate (the current maximum) is accepted with certainty, guaranteeing the algorithm makes progress.

The role of β: The hyperparameter β controls how aggressively the sampling favors high-reward responses:

When β → 0: The exponent (r - r_max)/β diverges to negative infinity for any r < r_max, making the acceptance probability effectively zero for all but the maximum-reward response. This reduces to best-of-N rejection sampling (also called top-1-over-N)—generate N candidates, pick the one with highest reward. This is "pure exploitation" that the paper warns leads to reward hacking because it trusts the reward model completely.
When β → ∞: The exponent approaches 0 for all responses, making acceptance probability approximately 1 for every candidate. This reduces to sampling directly from the SFT policy with no reward-based filtering—"pure exploration" with no alignment.
At intermediate β: The acceptance probability smoothly interpolates between these extremes, accepting higher-reward responses more often while still occasionally accepting lower-reward ones. This provides regularization against reward model errors—even if the reward model mistakenly assigns an inflated score to a bad response, the expected number of times that response gets through the filter is bounded by β.

Why this form for controlling the tradeoff: The exponential form exp(r/β) in the optimal policy equation directly traces back to the KL-constrained objective. The KL penalty -D_KL(π || π_sft) in the objective creates exactly this exponential re-weighting structure. The β in the rejection sampling algorithm is the same β from the objective—it has the same interpretation and the same effect on the resulting distribution. This consistency means that tuning β for the sampling procedure is equivalent to tuning the exploration-exploitation tradeoff in the underlying optimization problem, giving it principled meaning rather than being an arbitrary algorithmic knob.

The algorithm in practice (Algorithm 1): The paper's implementation (Algorithm 1 in Appendix A.1) works as follows for each prompt x:

Sample n_c = 64 response candidates from π_sft(y|x) at temperature 0.7 with top-k=40.
Score each candidate with the derived pointwise reward r_ψ(x, y) using the baseline trick.
Initialize an empty set of accepted responses and a dictionary mapping each candidate to its reward.
While fewer than n_d = 8 responses have been accepted:
- Compute r_max = max(r) over remaining (not-yet-accepted) candidates.
- For each remaining candidate y with reward r:
  - Draw u ~ Uniform[0, 1].
  - If u < exp((r - r_max) / β), accept y and remove it from the candidate pool.
- If the desired number n_d is reached, stop.
Return the set of accepted responses.

Critical implementation detail—sampling without replacement: Once a response is accepted, it is removed from the candidate pool before the next round. This has two effects. First, it ensures the algorithm produces n_d distinct responses—without replacement, the same high-reward response could be accepted repeatedly. Second, it forces the algorithm to explore beyond the single highest-reward candidate. After the top candidate is removed, the next round's r_max becomes the second-highest reward, creating a new acceptance landscape. This produces a set of responses that collectively represent the high-reward tail of π_sft's output distribution, rather than a single point estimate.

Expected acceptance rate (Theorem 1): The paper proves that as the number of candidates goes to infinity, the expected acceptance rate is:

$\mathbb{E}_{y \sim \pi_{\text{sft}}(y|x)} \left[ \exp\left( \frac{1}{\beta} (r_\psi(x, y) - r_{\max}) \right) \right]$

What this computes: The average probability that a randomly sampled SFT response gets accepted, where the expectation is taken over the SFT distribution. This depends on how concentrated the reward distribution is near the maximum. If many SFT responses have rewards close to r_max, the acceptance rate is high (the SFT policy naturally generates high-quality responses). If most SFT responses have rewards far below r_max, the acceptance rate is low (the SFT policy rarely generates good responses, so most get rejected).

Why this is important: The acceptance rate determines the computational efficiency of the algorithm. If the SFT policy's pass@1 for high-reward responses is very low, the expected number of SFT samples needed to get n_d accepted responses grows exponentially—the acceptance probability goes to zero as (r - r_max)/β diverges. This is why the paper samples 64 candidates to get 8 accepted responses—the 8x oversampling factor empirically provides enough candidates to ensure the rejection sampling doesn't reject everything. The acceptance rate formula also clarifies that β controls not just the quality of accepted responses but also the sampling efficiency: larger β (less selective) yields higher acceptance rates but lower average quality; smaller β (more selective) yields lower acceptance rates but higher average quality.

Connection to prior "rejection sampling" in RLHF: The paper explicitly connects its statistical rejection sampling to the best-of-N approach called "rejection sampling" in the RLHF literature (AnthropicHH, Llama2):

"If β → 0, only the highest reward response will be accepted and all other responses will be rejected. This is the rejection sampling (top-k-over-N) referred by AnthropicHH and Llama2."

The key difference is that prior work's best-of-N has β implicitly set to 0—it trusts the reward model absolutely and selects only the top-ranked response. This is vulnerable to reward hacking because the reward model may assign high scores to responses that exploit its blind spots (e.g., overly long outputs, repetitive phrases, sycophantic responses). RSO's tunable β provides a principled regularization against this: by accepting some lower-reward responses with non-zero probability, the sampling maintains some of the SFT model's diversity and reduces dependence on the reward model's potentially flawed highest-reward predictions. In practice, the paper finds β = 0.5 to be optimal (Figure 3b).

Loss Functions and the Unified Preference Optimization Framework

With the sampled preference pairs from rso-sample-rank, the policy π_θ is trained to align its implicit reward (via the inverse relationship r_θ = β log(π_θ/π_sft)) with the preference labels. The paper presents two loss functions that differ in how they penalize preference violations, unifying DPO and SLiC within a common statistical framework.

The Bradley-Terry preference probability in terms of policy: The starting point for both loss functions is Equation (2), which expresses the true preference probability p*(y_1 ≻ y_2 | x) in terms of the unknown optimal policy π* and the known reference policy π_sft:

$p^*(y_1 \succ y_2 | x) = \frac{1}{1 + \exp\left( \beta \log \frac{\pi^*(y_2|x)}{\pi_{\text{sft}}(y_2|x)} - \beta \log \frac{\pi^*(y_1|x)}{\pi_{\text{sft}}(y_1|x)} \right)}$

What this computes: The probability that y_1 is preferred over y_2, expressed as a logistic function of the difference in normalized log-probability ratios. The term log(π*(y)/π_sft(y)) is the implicit reward (up to scale β) that the optimal policy assigns to response y. The larger π*'s probability of y_1 relative to π_sft's probability of y_1, compared to the same ratio for y_2, the higher the preference probability for y_1.

Why this form enables policy optimization without a reward model: The partition function Z(x) cancels out in the difference, meaning the preference probability depends only on the policy probabilities themselves. If we have training pairs (x, y_w, y_l) where y_w ≻ y_l, we can directly optimize π_θ to make p_θ(y_w ≻ y_l | x) close to 1, without ever computing an explicit reward. This is DPO's key insight. RSO extends it by noting that while the loss form doesn't require a reward model, the sampling distribution for the training pairs still matters for estimation quality.

Loss Function 1: Sigmoid-Norm (Logistic Regression / DPO's Loss)

This is the loss function used by DPO, which the paper reframes as logistic regression on normalized likelihood ratios:

$L_{\text{sigmoid-norm}}(\pi_\theta | \pi_{\text{sft}}, D_p) = -\mathbb{E}_{(x, y_w, y_l) \sim D_p} \left[ \log \sigma\left( \gamma \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{sft}}(y_w|x)} - \gamma \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{sft}}(y_l|x)} \right) \right]$

where σ(z) = 1/(1 + exp(-z)) is the logistic sigmoid function, D_p is the preference dataset of (x, y_w, y_l) triples, and γ is a temperature hyperparameter (equivalent to β in DPO's original formulation, but the paper decouples γ from the sampling β to allow independent tuning of the loss sharpness and the sampling selectivity).

What it computes: For each preference pair (x, y_w, y_l) in the training set, compute the difference in normalized log-probability ratios between the winner and loser: Δ = γ log(π_θ(y_w)/π_sft(y_w)) - γ log(π_θ(y_l)/π_sft(y_l)). If y_w truly deserves higher reward, then π_θ should assign it higher probability relative to π_sft than it assigns to y_l, making Δ > 0. Pass Δ through the sigmoid to get the predicted preference probability for y_w, then take the negative log—this is the standard logistic regression loss for binary classification where the target is P(y_w ≻ y_l) = 1. The expectation averages this loss over all training pairs.

Why this form—logistic regression interpretation: The term inside the sigmoid, γ log(π_θ/π_sft), is exactly the policy's implicit reward estimate (scaled by γ/β). The logistic regression is fitting a linear decision boundary in this implicit reward space: it tries to make the reward difference r_θ(y_w) - r_θ(y_l) large and positive for all training pairs. The logistic loss is a smooth, convex upper bound on the 0-1 classification error—it heavily penalizes confident mistakes (when Δ is very negative, -log σ(Δ) grows approximately linearly with |Δ|) while providing relatively gentle gradients for correctly classified examples with large margins. The hyperparameter γ controls the steepness of the sigmoid: larger γ makes the loss more sensitive to small reward differences, effectively increasing the penalty for misclassified examples. The paper finds γ = 0.05 to be optimal (Figure 3a).

Why decouple γ from β: DPO sets γ = β because, in its derivation, β is both the KL penalty coefficient and the inverse temperature in the optimal policy. But RSO uses β only for rejection sampling—it controls the distribution of training pairs. The loss temperature γ controls how aggressively the policy fits those pairs. Decoupling them allows independent optimization: β determines the quality-vs-diversity tradeoff in the training data, while γ determines the training dynamics given that data. This is a practical insight that the DPO derivation obscured.

Loss Function 2: Hinge-Norm (SVM / Improved SLiC)

The paper proposes an improved version of SLiC's loss, framed as the support vector machine counterpart to DPO's logistic regression:

$L_{\text{hinge-norm}}(\pi_\theta | \pi_{\text{sft}}, D_p) = \mathbb{E}_{(x, y_w, y_l) \sim D_p} \left[ \max\left( 0, 1 - \left( \gamma \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{sft}}(y_w|x)} - \gamma \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{sft}}(y_l|x)} \right) \right) \right]$

where the max(0, ·) is the hinge function, and the margin is 1 (compared to SLiC's original margin δ).

What it computes: For each preference pair, compute the same implicit reward difference Δ = γ log(π_θ(y_w)/π_sft(y_w)) - γ log(π_θ(y_l)/π_sft(y_l)). If Δ ≥ 1, the loss is zero—the prediction is "correct enough" with at least a unit margin. If Δ < 1, a linear penalty 1 - Δ is incurred. The loss only cares about violations of the margin; once the margin is satisfied, no further optimization pressure is applied.

Why this form—SVM interpretation: The hinge loss creates a hard margin classifier: it doesn't care about making Δ arbitrarily large, only about pushing it above the threshold 1. This is exactly the SVM loss for binary classification where the target is y_w ≻ y_l with label +1. The logistic loss (sigmoid-norm) by contrast continues to provide gradients even for correctly classified examples—it always wants the margin to be larger, pushing π_θ(y_w) up and π_θ(y_l) down further. The hinge loss is more robust to outliers because once the margin is satisfied, outliers don't influence the gradient. The margin value 1 is conventional for SVMs; SLiC's original margin δ plays the same role but is treated as a tunable hyperparameter (the paper's 1/γ corresponds to SLiC's δ).

Improvement over original SLiC loss: The paper notes two subtle differences between their hinge-norm loss and SLiC's original loss (Equation 1). First, SLiC includes a regularization term -λ log π_θ(y_ref | x) that encourages the policy to maintain high probability on the SFT target response, but the paper's ablation (Table 5, Appendix A.6) shows this regularization "does not show significant improvement" across multiple λ values, so they drop it to align better with DPO's setting. Second, SLiC uses tournament-style ranking to construct preference pairs from a ranked list of responses, while RSO uses first-round ranking (pairing adjacent responses in the ranked list). These modifications make the hinge-norm loss a purer SVM counterpart to DPO, removing heuristics that weren't empirically justified.

Connecting losses to the Bradley-Terry model: Both loss functions can be understood as fitting a binary classifier where the logit is γ log(π_θ/π_sft), which corresponds to the policy's implicit reward estimate. Under the BT model, the true log-odds of preference should be (r*(y_w) - r*(y_l)) / β. The loss functions are training π_θ so that its implicit reward differences match the observed preference labels, with the only difference being how discrepancies are penalized: sigmoid-norm uses logistic (smooth, always non-zero gradient); hinge-norm uses hinge (hard margin, zero gradient beyond margin).

The Three Preference Data Sources

The paper evaluates three strategies for constructing the preference dataset D_p used in the loss functions. These differ in how close the sampling distribution is to the target optimal policy π*.

Source 1: Direct Use the human preference data D_hf directly as D_p, without any reward model or additional sampling. This is what DPO does in all its experiments.

What it represents statistically: The human preference data D_hf was collected from "mixed unknown policies"—the SFT model, previous RLHF checkpoints, policies from other agents (as in Llama2's data collection), or even human-written responses. The sampling distribution π_unk is not under the experimenter's control and is almost certainly not π*. This means the training pairs are off-policy: they come from a distribution that may be very different from the distribution the policy should concentrate on after alignment. Statistically, this is not MLE for π* because the pairs are not i.i.d. from π*. The resulting estimator may still be consistent under regularity conditions, but its finite-sample efficiency is degraded—you need more data to get the same quality of estimate, and the degradation is worse the further π_unk is from π*.

Source 2: SFT-Sample-Rank For each prompt x in the SFT training set, sample n_d = 8 response candidates from π_sft(y|x), rank them using the pairwise reward model ρ_ψ, and construct preference pairs from the ranked list. This is approximately what SLiC does in its "sample-rank" variant.

What it represents statistically: The sampling distribution is the SFT policy π_sft, which is at least a known and controlled distribution. However, π_sft assigns high probability to responses that are fluent and on-topic (because it was supervised fine-tuned on good examples), but not necessarily to responses that maximize human preference. The training pairs are concentrated in the "average quality" region of response space—the region where π_sft has high density—rather than the "high quality" region where π* has high density. As a result, the policy may learn to discriminate well among mediocre responses but have poor estimates of reward differences in the high-quality tail, which is exactly where the deployed policy should operate.

Source 3: RSO-Sample-Rank For each prompt x, sample n_c = 64 responses from π_sft, apply statistical rejection sampling with the reward model and β = 0.5 to select n_d = 8 responses distributed approximately according to π_rψ(y|x), then rank these accepted responses using the reward model to construct preference pairs. This is RSO's proposed approach.

What it represents statistically: The sampling distribution is π_rψ, which is the optimal policy induced by the trained reward model r_ψ according to Equation (4). Since r_ψ is trained to approximate the true human preference function r*, π_rψ should be close to the true optimal policy π*. The training pairs are therefore approximately on-policy—sampled from a distribution that concentrates on high-quality responses, which is where π* has high density. This makes the preference optimization a much better approximation to MLE for π*, because the pairs are drawn from (an estimate of) the target distribution itself.

Why this matters—the statistical argument in full: The Bradley-Terry model specifies P(y_w ≻ y_l | x) = σ(r*(x, y_w) - r*(x, y_l)). This formulation does not depend on the distribution of (y_w, y_l) given x—it's a statement about the functional form of preferences, not about sampling. However, when we estimate r* (or equivalently π*) from finite data, the quality of the estimate depends on where the data points are located. If most training pairs involve responses far from the high-quality region, the estimate of r* in the high-quality region will be imprecise because it's extrapolating from distant observations. By sampling pairs from π_rψ ≈ π*, RSO concentrates training data where the policy needs to be most accurate—in the region of high-quality responses that the aligned policy should produce. This is the same principle that makes on-policy data collection in RLHF effective: you want to evaluate and improve the policy on exactly the kinds of responses it's likely to generate, not on arbitrary responses from a different distribution.

The paper's claim about this ordering: The paper asserts a clear ordering: rso-sample-rank > sft-sample-rank > direct in terms of statistical estimation quality for π*. The experimental results consistently support this ordering across all metrics, loss functions, and tasks (Table 1): within each loss function column, moving from direct to sft-sample-rank to rso-sample-rank monotonically improves performance.

The Full Training Recipe

Phase 0: Supervised Fine-Tuning A T5-large (770M) or T5-XXL (11B) model is fine-tuned on task-specific supervised data to produce the SFT policy π_sft and the reference model. For Reddit TL;DR, the SFT data contains 117k training examples of forum posts and their human-written TL;DR summaries. For AnthropicHH, the positive (chosen) responses from the helpfulness preference data serve as SFT targets.

Phase 1: Reward Model Training A separate T5-XXL (11B) model is fine-tuned as a pairwise reward-ranking model ρ_ψ on the human preference data D_hf. No architecture modifications are needed—it's a standard T5 model that outputs "A" or "B" tokens. Training uses the same human preference pairs that would be used directly by DPO.

Phase 2: Candidate Generation For each prompt x in the SFT training set, generate n_c = 64 response candidates from the SFT policy π_sft(y|x) using temperature sampling with temperature 0.7 and top-k=40. The paper doesn't specify the maximum decoding length, but it's presumably set to match the task's typical response length.

Phase 3: Rejection Sampling For each prompt:

Score all 64 candidates using the derived pointwise reward r_ψ(x, y) = logit(ρ_ψ(x, y, y_b)) where y_b is a randomly chosen baseline response from the SFT policy (reward score 0 by convention).
Apply Algorithm 1 with β = 0.5 to select n_d = 8 responses via statistical rejection sampling.
The 8 accepted responses are approximately distributed as π_rψ(y|x) with β = 0.5.

Phase 4: Preference Pair Construction For each prompt, construct n_d / 2 = 4 preference pairs from the 8 accepted responses. The paper's default approach is "first-round-rank": rank the 8 responses by their reward scores and pair adjacent responses (1st vs. 2nd, 3rd vs. 4th, etc.), labeling the higher-ranked one as preferred. The paper also experiments with "tournament-rank" (full elimination tournament producing 7 pairs from 8 responses) but finds first-round-rank is optimal for AutoSxS while tournament-rank inflates proxy reward without improving true quality (Table 2)—evidence that tournament ranking introduces a bias the reward model exploits but humans don't agree with.

Phase 5: Policy Optimization Train the policy model π_θ (initialized from π_sft) on the constructed preference pairs using either sigmoid-norm or hinge-norm loss. Key hyperparameters:

Optimizer: Adafactor (Shazeer & Stern, 2018)
Learning rate: 1e-5
Batch size: 32
γ (loss temperature): 0.05 (optimal from Figure 3a)
Checkpoint selection: pick the checkpoint with the highest reward-ranking model win rate against the SFT target responses (not validation loss, which can be misleading for preference optimization because it doesn't directly measure alignment quality).

Phase 6: Inference At inference time, the trained policy π_θ generates responses using standard decoding (the paper doesn't specify exact inference hyperparameters, but the evaluations use the reward model and AutoSxS to compare generated responses against SFT targets, so presumably decoding parameters are standard).

Scaling to T5-XXL: When scaling the policy from T5-large (770M) to T5-XXL (11B), RSO follows the same recipe with the loss fixed as sigmoid-norm. The reward model remains a separate T5-XXL (i.e., T5-XXL reward model guiding T5-XXL policy in this case). The results (Table 3) show RSO scales well, improving AutoSxS over DPO by 1.1% on Reddit TL;DR and 33.1% on AnthropicHH.

Comparison with RAFT and ReST baselines: The paper includes two additional rejection-sampling-related baselines:

RAFT (Dong et al., 2023): After SFT, select the best decoded sequence (highest reward) as new SFT target, then continue SFT training. This is essentially best-of-1 with no KL regularization.
ReST (Gulcehre et al., 2023): Normalize reward scores to [0, 1], pick sequences with reward > 0.7 as new SFT targets, then continue SFT. This is one round of "grow and improve" with a hard threshold.

Both RAFT and ReST use the reward model to filter SFT responses, but they then train with standard cross-entropy on the selected responses rather than with preference-based losses. They lack the KL-regularized rejection sampling structure (β control) and the preference discrimination training signal. The results (Table 1) show RSO substantially outperforms both: on Reddit TL;DR, RAFT achieves 53.77% AutoSxS vs. RSO's 71.86%; ReST achieves 34.36% vs. RSO's 71.86%.

Why RSO outperforms RAFT and ReST: RAFT and ReST discard information. RAFT only keeps the single best response per prompt, throwing away potentially useful signal from the other 63 candidates. ReST keeps responses above a hard threshold but throws away the fine-grained reward ranking among kept responses—all responses above 0.7 are treated as equally good SFT targets. RSO preserves more information: it uses the full reward distribution via rejection sampling (accepting some lower-reward responses probabilistically), AND it constructs explicit preference pairs that teach the policy about reward differences between responses via the sigmoid-norm or hinge-norm loss. This combination—on-policy sampling PLUS preference discrimination training—is what drives the gains.

Computational efficiency: The paper addresses efficiency concerns in Appendix A.10. The key points:

RSO requires additional SFT inference (64 decodes per prompt) and reward model inference (64 comparisons for scoring + 4 comparisons for labeling = 68 calls per prompt), but this is fully parallelizable over the training set and accounts for "less than 10% of the total training time."
Like DPO, RSO only needs one policy network during training (vs. PPO's four networks: policy, value, reward, reference).
Reward model inference is fast because it only decodes a single token ("A" or "B").
SFT decoding benefits from prompt caching (same prompt used for all 64 candidates) and batched inference.
The statistical rejection sampling itself (Algorithm 1) uses sampling without replacement and the r_max recalculation to ensure at least one acceptance per round, making it computationally negligible compared to the neural network forward passes.

4. Key Insights and Innovations

Innovation 1: Preference Optimization Is Fundamentally a Distribution Shift Problem, Not a Loss Function Problem

The paper's most significant conceptual contribution is reframing the core challenge of offline preference optimization. Before RSO, the field tacitly assumed that the primary axis of improvement was the loss function — DPO's sigmoid loss was an improvement over PPO's complexity, SLiC's hinge loss was an alternative to DPO's logistic loss, and so on. The dominant question was: "what loss function best translates human preference data into a good policy?" RSO argues that this framing misses the deeper issue. The real bottleneck is not how you fit the data, but which data you fit — specifically, whether your training pairs come from a distribution close to the optimal policy π* or from some suboptimal, off-policy source.

This is a diagnostic move, not a metric optimization. It says: the community has been optimizing the wrong variable. The evidence is in Table 1, which shows that within any loss function (sigmoid-norm, hinge, or hinge-norm), moving from "direct" to "sft-sample-rank" to "rso-sample-rank" — that is, moving the sampling distribution closer to π* — yields consistent, substantial improvements. On Reddit TL;DR with sigmoid-norm, the AutoSxS win rate jumps from 67.72% (direct, i.e., DPO on human data from unknown policies) to 69.02% (sft-sample-rank) to 71.86% (rso-sample-rank). The same pattern holds across both tasks and all three loss functions. Meanwhile, comparing across loss functions within the same sampling column (e.g., sigmoid-norm vs. hinge-norm, both with rso-sample-rank) shows much smaller differences: 71.86% vs. 70.84% AutoSxS on Reddit TL;DR, 40.98% vs. 38.58% on AnthropicHH. The loss function matters, but the sampling distribution matters more.

This reframing has a sharp implication that goes unstated but is clear in the paper's structure: DPO's elimination of the reward model was solving a complexity problem but creating a statistical one. By absorbing the reward model into the policy, DPO made the training pipeline simpler but also made it impossible to sample from anything close to π* — you can't rejection-sample without a reward function to evaluate candidates. DPO's "sft-sample-rank" extension (sampling from the SFT policy and getting humans to label pairs) is the closest it can get, but it's still sampling from π_sft, not π*. RSO's key insight is that keeping the reward model explicit — at the cost of training and maintaining a separate 11B-parameter model — provides a capability (guided rejection sampling toward π*) that no amount of loss function engineering can replicate when the reward signal is absorbed into the policy.

This is a fundamental shift in how to think about the problem, not an incremental refinement. It changes the optimization objective from "find a better loss" to "find a better data distribution," which opens up a different research direction: what other ways can we construct on-policy-like preference data without running full online RL?

Innovation 2: A Unified Statistical Framework Revealing DPO as Logistic Regression and SLiC as SVM

The paper provides a clean statistical reinterpretation that had been missing from both the DPO and SLiC lines of work. DPO was presented as a clever algebraic manipulation that eliminates the reward model; SLiC was presented as a contrastive calibration heuristic. RSO shows that both are instances of the same statistical estimation problem — fitting the Bradley-Terry preference model — and differ only in their choice of classification loss: DPO uses logistic regression (sigmoid loss on normalized log-probability ratios), while SLiC is essentially a support vector machine (hinge loss on the same normalized log-probability ratios, once the SFT regularization term is dropped).

This unification is not merely taxonomic. It provides principled guidance for hyperparameter selection and loss comparison that was previously ad hoc. By framing DPO as logistic regression with logit γ log(π_θ/π_sft), the role of γ becomes clear: it's the inverse temperature of the sigmoid, controlling how sharply the loss penalizes preference violations. DPO originally set γ = β because β appears in both the KL-regularized objective and the optimal policy expression, but RSO's reframing reveals these as independent quantities — β controls the exploration-exploitation tradeoff in sampling, while γ controls the training dynamics given fixed data. Decoupling them (section 3.2, studied in Figure 3a) is a practical improvement that follows directly from the statistical interpretation.

Similarly, reframing SLiC's hinge loss as an SVM on implicit reward differences explains why it's more robust to outliers (the margin creates a region where correctly classified examples contribute zero gradient) and why it might be more prone to reward hacking (the hard threshold means the loss doesn't penalize the policy for making the margin arbitrarily large, which can lead to extreme probability ratios that exploit the reward model). The paper's improved "hinge-norm" variant — applying the hinge to normalized likelihood ratios log(π_θ/π_sft) rather than raw log-probabilities log π_θ — is a direct consequence of this reframing: it's the SVM counterpart to DPO's logistic regression on the same normalized quantities. The improvement over the original SLiC hinge loss is visible in Table 1: on Reddit TL;DR, hinge-norm with rso-sample-rank achieves 70.84% AutoSxS vs. hinge (original SLiC-style) with rso-sample-rank at 69.26%.

This contribution is incremental in mechanism but fundamental in conceptual clarity. The loss functions themselves aren't new (logistic regression and hinge loss are standard), but the paper's identification of what they're fitting — a Bradley-Terry preference model on implicit rewards defined by normalized policy probability ratios — provides the missing theoretical grounding that enables systematic comparison and improvement. Before this work, researchers choosing between DPO and SLiC were comparing heuristics; after this work, they're comparing well-understood statistical estimators with known properties.

Innovation 3: Statistical Rejection Sampling as a Unifying Principle Connecting Best-of-N to KL-Regularized Policy Optimization

The paper makes a theoretical connection that had been obscured by terminological confusion in the RLHF literature. In prior work (AnthropicHH, Llama2, ReST), "rejection sampling" referred to a specific heuristic: generate N candidates from a policy, score them with a reward model, and keep the best one (or top-k). This was treated as a distinct technique — a simple trick for improving outputs without any formal connection to the RLHF objective.

RSO shows that this best-of-N "rejection sampling" is actually a limiting case of a more general statistical rejection sampling procedure that generates samples from the KL-regularized optimal policy π_r. Specifically, best-of-N corresponds to setting β → 0 in Algorithm 1: when β is infinitesimally small, the acceptance probability exp((r - r_max)/β) becomes 1 for the maximum-reward response and 0 for all others, reducing exactly to selecting the top-1 candidate by reward. The general algorithm with β > 0 interpolates between best-of-N (β = 0, pure exploitation) and SFT sampling (β = ∞, pure exploration), with β playing the same role as in the KL-constrained objective: it controls the tradeoff between pursuing high reward and staying close to the reference policy.

This connection is conceptually fundamental because it reveals best-of-N's vulnerability as a specific parameter choice rather than an inherent limitation of rejection sampling. Best-of-N is not a flawed technique per se; it's rejection sampling with β set to a value that maximizes reward model exploitation and minimizes regularization. The reward hacking that best-of-N is known to suffer from (selecting responses that score highly under the reward model but are actually poor) is therefore a symptom of the β → 0 limit — trusting the reward model too absolutely — rather than an indictment of the sampling approach. By setting β = 0.5 (the optimal value found in Figure 3b), RSO achieves substantially better true quality (AutoSxS and human evaluation) than what best-of-N would produce, because it maintains some of the SFT model's diversity and avoids over-optimizing the reward model's potentially flawed judgments.

This reframing also clarifies why prior "rejection sampling" approaches like RAFT and ReST underperform RSO in Table 1 despite using similar ingredients (SFT policy + reward model). RAFT selects only the single best response per prompt (β → 0 with no KL regularization), discarding the information in the other 63 candidates and providing only one SFT target rather than structured preference pairs. ReST uses a hard reward threshold (equivalent to a discontinuous acceptance function rather than the smooth exponential form required by the KL-constrained objective), which doesn't correspond to any valid β in the statistical rejection sampling framework. RSO's unified formulation makes these design choices legible as approximations to the KL-regularized optimal policy, revealing why they fall short and how to fix them.

The evidence for this insight's practical importance is in Figure 3b: sweeping β from 0 to 5 on Reddit TL;DR shows that β = 0.5 achieves the highest proxy reward win rate, with β = 0 (best-of-N) substantially lower (roughly 86% vs. 92%), and β = 5 (near-SFT) also lower (roughly 87%). The optimal β is at an intermediate value that balances reward exploitation and regularization — exactly what the theoretical connection to the KL-constrained objective predicts.

Innovation 4: The Asymmetric Difficulty of Discrimination vs. Generation as a Design Principle for Alignment Systems

The paper articulates and empirically validates a design principle that has implications beyond the specific RSO algorithm: maintaining an explicit reward model that is larger and more capable than the policy model is a better allocation of compute than trying to make the policy self-evaluating. The paper states this directly:

"DPO claims that the language model is secretly a reward model, we show that the language model learns better from an explicit reward model because comparing between two responses (reward) is easier to learn than generating high quality responses (policy)."

This is a claim about the relative statistical difficulty of two learning problems: pairwise preference discrimination (outputting "A" or "B" given two responses) versus autoregressive sequence generation (outputting a full response token-by-token). Discrimination requires learning a decision boundary in representation space — a binary classification problem. Generation requires learning a high-dimensional conditional distribution over sequences — a density estimation problem over an exponentially large output space. For a fixed parameter budget, the discriminator can achieve higher accuracy on its (simpler) task than the generator can achieve implicit discrimination accuracy as a byproduct of its (harder) task.

The paper operationalizes this principle through asymmetric scaling: the reward model is an 11B-parameter T5-XXL, while the policy is a 770M-parameter T5-large (in the main experiments). The reward model gets 14× more parameters than the policy, reflecting the judgment that reward discrimination has higher marginal returns to scale. This is not an arbitrary engineering choice — it follows from the asymmetry claim. If discrimination and generation were equally hard, you'd want them at equal scale; if generation were harder (as DPO implicitly assumes by absorbing reward into the policy), you'd want the policy to be larger. The paper's choice of a larger discriminator than generator is a bet on the asymmetry claim, and the strong performance of RSO (Table 1) validates that bet.

This principle is fundamental rather than incremental because it challenges a core premise of the DPO approach. DPO's elegance comes from proving that you don't need a separate reward model — the policy can serve as its own critic. The RSO paper argues that while this is mathematically possible, it's statistically inefficient: you can make the policy do double duty, but you get better results by offloading the discrimination task to a specialized, larger model that faces an easier learning problem. The analogy is to actor-critic methods in RL, where the critic (value function) is often simpler to learn than the actor (policy), so maintaining them separately with different architectures can be more efficient than forcing the actor to estimate values internally.

The evidence is in the consistent gap between DPO (sigmoid-norm + direct) and RSO (sigmoid-norm + rso-sample-rank) across all metrics: on Reddit TL;DR, DPO achieves 67.72% AutoSxS vs. RSO's 71.86%; in human evaluation, RSO is chosen as preferred 48% of the time vs. DPO's 21%. Both use the same loss function (sigmoid-norm) and the same policy architecture (T5-large). The only difference is that RSO leverages an explicit 11B reward model to construct better training data. The gap quantifies the value of explicit, asymmetric reward modeling over implicit, self-contained policy evaluation — and it's substantial.

When the policy is scaled to T5-XXL (matching the reward model's size, Table 3), RSO still outperforms DPO (86.01% vs. 85.03% AutoSxS on Reddit TL;DR, 70.26% vs. 52.80% on AnthropicHH), but the gap narrows on summarization while remaining large on dialogue. This suggests the asymmetry principle may be domain-dependent: for tasks where the policy can achieve good implicit discrimination when given enough capacity (summarization), the explicit reward model's advantage shrinks; for tasks where discrimination remains hard even at scale (dialogue, where the DPO-direct baseline is a dismal 52.80% vs. RSO's 70.26%), the explicit reward model is crucial. This is a nuanced finding that complicates the simple "discrimination is easier" claim and points toward a more contextual principle: the value of explicit reward modeling depends on how much harder generation is than discrimination for the specific domain.

5. Experimental Analysis

Evaluation Methodology

Datasets. The paper uses two primary datasets: Reddit TL;DR summarization (Stiennon et al., 2020) and AnthropicHH dialogue (Bai et al., 2022). Reddit TL;DR contains 117k/6k/6k examples in train/validation/test splits for the SFT phase, plus D_tldr_hf consisting of 93k human preference pairs on responses decoded from multiple models. AnthropicHH uses the helpfulness slice D_helpful_hf with 161k/9k examples in train/test splits; the positive (chosen) responses serve as SFT targets. A cross-task generalization experiment uses CNN/DailyMail (Hermann et al., 2015), which contains only SFT data (287k/13k/11k examples in train/validation/test splits) and no human preference data—the model is trained only on Reddit TL;DR preference data and evaluated on CNN/DailyMail test targets.
Base models. The SFT policy for the main experiments is T5-large (770M parameters). The pairwise reward-ranking model is a separate T5-XXL (11B parameters). The paper argues this asymmetric scaling is deliberate: "comparing between two responses (reward) is easier to learn than generating high quality responses (policy)," so the reward model gets more capacity. Scaling experiments additionally use a T5-XXL policy model (11B, matching the reward model's size) to test whether the RSO advantage persists when the policy is as large as the discriminator. All models are initialized from pre-trained T5 checkpoints and then fine-tuned on task-specific data.
Metrics. Four metrics are used, listed in order of increasing reliability and cost:
- Proxy Reward Model win rate: The trained T5-XXL pairwise reward-ranking model evaluates whether the policy's generated response is preferred over the SFT target response. This is the cheapest metric to compute but is vulnerable to reward hacking—the policy may learn to exploit blind spots in the same reward model used for both data construction and evaluation.
- Gold Reward Model win rate: A separate PaLM 2-S model (Anil et al., 2023) trained on the same human preference data serves as a held-out reward evaluator. This is robust to overfitting on the training reward model but is still an automated metric. The Gold Reward Model achieves 76.07% validation accuracy on summarization and 70.18% on dialogue.
- AutoSxS win rate: A PaLM 2-L model used in few-shot in-context learning mode (8 decoded samples with 4 order-flipped response pairs, requiring average score magnitude > 0.35 to declare win/loss) compares the policy's response against the SFT target. This uses a larger, more capable model as judge and is the primary automated evaluation.
- Human evaluation: Amazon Mechanical Turk raters assign a pointwise overall quality score (1–5) to each response and choose the best among three anonymized systems ("direct," "sft-sample-rank," "rso-sample-rank"), with each task replicated 3 times and aggregated via average (pointwise) and majority vote (preference). In total 47 different raters participated with a median of 16 tasks per rater. All metrics are reported as win rates against the SFT target response (for summarization: the human-written TL;DR; for dialogue: the positive response from the helpfulness data).
Baselines. The paper compares against five baselines:
- DPO (Rafailov et al., 2023): sigmoid-norm loss with "direct" preference data—the original DPO recipe using human preference pairs from unknown policies without any additional sampling or reward model.
- SLiC-direct (Zhao et al., 2023): hinge loss applied directly to the human preference data, approximately the "SLiC-HF-direct" variant from the original paper.
- SLiC-sample-rank (Zhao et al., 2023): hinge loss on preference pairs sampled from the SFT policy and ranked by the reward model via tournament ranking. This is labeled "SLiC_sft-sample-rank" in tables.
- RAFT (Dong et al., 2023): After SFT, select the single highest-reward decoded sequence as the new SFT target and continue cross-entropy training.
- ReST (Gulcehre et al., 2023): Normalize reward scores to [0, 1], select decoded sequences with reward > 0.7 as new SFT targets, and continue cross-entropy training. This is one round of "grow and improve." Additionally, the paper evaluates all combinations of three loss functions (sigmoid-norm, hinge, hinge-norm) with three preference data sources (direct, sft-sample-rank, rso-sample-rank), yielding a 3×3 grid where DPO = sigmoid-norm-direct, SLiC-sample-rank ≈ hinge-sft-sample-rank, and RSO = {sigmoid-norm or hinge-norm} + rso-sample-rank.
Generation budget / compute accounting. The paper does not use a unified "compute budget" abstraction like generation count for search. Instead, cost is compared indirectly through inference demands. For each prompt, RSO requires: (1) n_c = 64 SFT decodes for candidate generation, (2) n_c = 64 reward model inferences for scoring those candidates (single-token outputs), (3) statistical rejection sampling (computationally negligible), and (4) n_d / 2 = 4 additional reward model inferences for labeling the constructed preference pairs. This totals 64 policy decodes and 68 reward model inferences per prompt. By comparison, DPO-direct requires zero additional inference beyond policy training. The paper notes that this additional sampling "accounts for less than 10% of the total training time" (Appendix A.10) due to parallelization, prompt caching, and the reward model's single-token decoding. For RAFT and ReST, the inference cost is n_c = 64 SFT decodes plus n_c - 1 reward model comparisons for scoring all candidates. All methods use the same batch size 32 and learning rate 1e-5 for the policy optimization step itself.
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the adaptive allocation policies in the example paper on test-time compute scaling). Instead, hyperparameters (β = 0.5, γ = 0.05, n_c = 64, n_d = 8) are selected via sweeps over Reddit TL;DR (Figures 3a and 3b) and then applied identically to AnthropicHH and CNN/DailyMail. For each training run, checkpoint selection uses the highest Proxy Reward Model win rate against SFT targets on the validation set—not AutoSxS, Gold Reward, or human evaluation, which preserves those as unbiased held-out metrics. The 95% confidence intervals in Figure 3 are computed but not specified by method; given the replication structure they are likely bootstrapped or based on standard errors across evaluation examples. Human evaluation uses 3-way replication per task with majority-vote aggregation.

Main Quantitative Results

The 3×3 Grid: Loss Functions × Preference Data Sources (Table 1)

Table 1 presents the paper's central empirical result across the 3×3 design space (loss functions: sigmoid-norm, hinge, hinge-norm; data sources: direct, sft-sample-rank, rso-sample-rank), with three metrics (Proxy Reward, Gold Reward, AutoSxS) on both tasks. The headline finding is that rso-sample-rank dominates within every loss function column, and the pattern is monotonic across data sources.

On Reddit TL;DR with sigmoid-norm loss:

direct (DPO): 84.35% Proxy, 76.09% Gold, 67.72% AutoSxS
sft-sample-rank: 88.63% Proxy, 78.14% Gold, 69.02% AutoSxS
rso-sample-rank (RSO): 92.37% Proxy, 82.22% Gold, 71.86% AutoSxS

The improvement over DPO is +4.14 percentage points AutoSxS (67.72% → 71.86%), which is a 6.1% relative improvement. The pattern replicates with hinge-norm: rso-sample-rank achieves 70.84% AutoSxS vs. direct at 66.63% (+4.21 points) and sft-sample-rank at 68.46% (+2.38 points).

On AnthropicHH with sigmoid-norm:

direct (DPO): 51.63% Proxy, 36.13% Gold, 24.01% AutoSxS
sft-sample-rank: 85.09% Proxy, 58.65% Gold, 39.56% AutoSxS
rso-sample-rank (RSO): 86.94% Proxy, 59.15% Gold, 40.98% AutoSxS

Here the jump from direct to sft-sample-rank is dramatic (+15.55 AutoSxS points), while rso-sample-rank provides a further +1.42 points. The enormous gap between direct and sft-sample-rank suggests that the human preference data for AnthropicHH is particularly off-policy—the unknown data-generating policies produce responses very different from what the SFT model generates. The DPO-direct baseline (24.01% AutoSxS) is barely above chance, while simply resampling from the SFT policy and ranking with a reward model (sft-sample-rank, 39.56%) nearly doubles the win rate. The additional gain from rejection sampling toward π_rψ (rso-sample-rank, 40.98%) is smaller but consistent.

Comparing across loss functions: Within the rso-sample-rank column, sigmoid-norm achieves 71.86% AutoSxS on Reddit TL;DR vs. hinge-norm's 70.84%—a small difference (1.02 points). On AnthropicHH, sigmoid-norm achieves 40.98% vs. hinge-norm's 38.58% (2.40 points). The hinge loss (original SLiC style) underperforms both on AutoSxS: 69.26% on Reddit TL;DR, 32.56% on AnthropicHH. The sigmoid-norm and hinge-norm losses perform similarly, with sigmoid-norm slightly favored. This suggests the primary gains come from sampling (rso-sample-rank over sft-sample-rank over direct), not from the choice between logistic regression and SVM loss.

A noteworthy anomaly: hinge loss with rso-sample-rank shows the highest Proxy Reward (93.36% on Reddit TL;DR) but lower AutoSxS (69.26%) than sigmoid-norm rso-sample-rank (92.37% Proxy, 71.86% AutoSxS). This is evidence of reward hacking—the hinge loss finds policies that exploit the training reward model (inflating Proxy Reward) without genuinely improving quality as measured by the held-out PaLM 2-L AutoSxS judge. The normalized hinge (hinge-norm) partially mitigates this: 92.80% Proxy, 70.84% AutoSxS.

Comparison to RAFT and ReST baselines: RAFT achieves only 53.77% AutoSxS on Reddit TL;DR and 24.99% on AnthropicHH—substantially below all rso-sample-rank variants. ReST is even worse: 34.36% on Reddit TL;DR, 15.58% on AnthropicHH. Both methods use the reward model to filter SFT responses but then train with cross-entropy on selected targets rather than preference-based losses. The large gap between these and RSO confirms that (1) preference discrimination training (sigmoid-norm or hinge-norm loss on explicit pairs) is more effective than cross-entropy on filtered targets, and (2) the probabilistic acceptance of rejection sampling preserves useful training signal that hard-threshold filtering (ReST) or top-1 selection (RAFT) discard.

Human Evaluation (Table 4)

Human evaluation on Amazon Mechanical Turk provides the most rigorous comparison, testing the three sampling strategies (direct, sft-sample-rank, rso-sample-rank) under both sigmoid-norm and hinge-norm losses. On Reddit TL;DR with sigmoid-norm:

direct: chosen as preferred 21% of the time, average quality score 3.84/5
sft-sample-rank: chosen 10%, quality 3.74
rso-sample-rank: chosen 48%, quality 4.02

RSO is chosen more than 2× as often as DPO (48% vs. 21%), with a quality score improvement of 0.18 points. The sft-sample-rank result (10%, quality 3.74) is surprisingly lower than direct—this is the only metric where sft-sample-rank underperforms direct, and the paper does not discuss this anomaly. One possibility is that the small sample size (47 raters, median 16 tasks) introduces noise; another is that sft-sample-rank responses are genuinely worse for some failure mode that the automated metrics don't capture.

On AnthropicHH with sigmoid-norm:

direct: chosen 15%, quality 3.04
sft-sample-rank: chosen 22%, quality 3.21
rso-sample-rank: chosen 31%, quality 3.37

RSO is chosen roughly 2× as often as DPO (31% vs. 15%), with a quality improvement of 0.33 points. The ordering here is monotonic (rso > sft > direct), unlike Reddit TL;DR.

With hinge-norm loss, the patterns are similar: rso-sample-rank is chosen 46% of the time on Reddit TL;DR and 33% on AnthropicHH, compared to 21% and 13% respectively for direct. The proportions do not sum to 100% because raters could declare ties across all three systems. Overall, human evaluation confirms the automated metric trends: rso-sample-rank > sft-sample-rank > direct for both loss functions and tasks, with the improvement from rso-sample-rank being substantial (roughly 2× preference rate over DPO).

Scaling to T5-XXL Policy (Table 3)

When the policy model is scaled from T5-large (770M) to T5-XXL (11B)—matching the reward model's parameter count—RSO continues to outperform DPO, though the margin depends on the task. On Reddit TL;DR, with both policy and reward model at T5-XXL scale:

DPO (direct): 94.04% Proxy, 85.03% AutoSxS
sft-sample-rank: 97.50% Proxy, 85.66% AutoSxS
RSO (rso-sample-rank): 98.29% Proxy, 86.01% AutoSxS

The AutoSxS improvement over DPO is +1.1 percentage points (85.03% → 86.01%). This is smaller than at T5-large scale (+4.14 points), suggesting that as the policy becomes more capable, the benefit of explicit reward-guided sampling diminishes—a larger policy may learn better implicit reward discrimination, making DPO's approach more competitive.

On AnthropicHH, the pattern is dramatically different:

DPO (direct): 76.84% Proxy, 52.80% AutoSxS
sft-sample-rank: 94.91% Proxy, 66.79% AutoSxS
RSO (rso-sample-rank): 97.54% Proxy, 70.26% AutoSxS

The AutoSxS improvement over DPO is +17.46 percentage points (52.80% → 70.26%), a 33.1% relative improvement. This is even larger than at T5-large scale (+16.97 points absolute, from 24.01% to 40.98%). The DPO-direct baseline at T5-XXL scale (52.80%) is still substantially below RSO at T5-large scale (40.98% with sigmoid-norm)—i.e., RSO with a 770M policy outperforms DPO with an 11B policy on this task. This is a striking result: better data (on-policy preference pairs from rejection sampling) can be more valuable than 14× more policy parameters when training with off-policy human data.

The cross-task gap between Reddit TL;DR and AnthropicHH in how much RSO helps at T5-XXL scale (+1.1 points vs. +17.46 points AutoSxS) is informative. It suggests the value of explicit reward modeling depends on how hard preference discrimination is relative to generation for the specific domain. On Reddit TL;DR summarization, a T5-XXL policy can apparently learn good implicit rewards even from off-policy human data (DPO achieves 85.03% AutoSxS), so RSO's additional sampling provides only a small boost. On AnthropicHH dialogue, even an 11B policy struggles to learn preferences from the available human data (DPO: 52.80%), and explicit reward-guided sampling is crucial.

Cross-Task Generalization to CNN/DailyMail (Table 6, Appendix A.7)

The CNN/DailyMail experiment tests whether RSO-trained policies transfer to a new summarization domain without any target-domain preference data. The SFT model is trained on Reddit TL;DR D_tldr_sft; preference optimization uses Reddit TL;DR preference data D_tldr_hf only; evaluation measures quality against CNN/DailyMail reference summaries.

With sigmoid-norm loss:

direct (DPO on Reddit TL;DR preferences only): 61.31% Proxy, 37.36% AutoSxS on CNN/DailyMail targets
sft-sample-rank (with prompts from CNN/DailyMail training set): 62.72% Proxy, 38.63% AutoSxS
rso-sample-rank: 69.38% Proxy, 39.71% AutoSxS

RSO outperforms DPO-direct by +2.35 AutoSxS points. With hinge-norm loss, the gap is larger: rso-sample-rank achieves 42.18% AutoSxS vs. direct at 33.91%—an 8.27-point improvement. This demonstrates that the benefit of on-policy-like preference pairs generalizes to new domains: by sampling from the SFT policy on CNN/DailyMail prompts (not used during preference training) and filtering via rejection sampling with the Reddit TL;DR-trained reward model, RSO constructs higher-quality preference pairs than using off-policy human data or SFT-policy pairs alone. The reward model's discrimination ability transfers across summarization domains even though the policy's generation distribution shifts.

Ablation Studies and Robustness Checks

Effect of γ in the loss function (Figure 3a, Reddit TL;DR): Sweeping γ = 0.005, 0.05, 0.5 with β fixed at 0.5 shows that γ = 0.05 provides the optimal Proxy Reward win rate for all three loss functions. At γ = 0.005 (very flat sigmoid/shallow hinge), the loss is too permissive—the policy is not penalized enough for preference violations, and win rates drop substantially (roughly 4–8 percentage points below optimum across loss functions). At γ = 0.5 (very steep sigmoid/sharp hinge), the loss over-penalizes small differences, leading to overfitting and slightly degraded performance compared to γ = 0.05. The optimal γ is the same for sigmoid-norm, hinge, and hinge-norm losses, suggesting this temperature is a property of the optimization problem (the scale of normalized log-probability ratios) rather than the specific loss functional form.
Effect of β in rejection sampling (Figure 3b, Reddit TL;DR): Sweeping β = 0, 0.05, 0.5, 5 with γ fixed at 0.05 and sigmoid-norm loss shows β = 0.5 achieves the optimal Proxy Reward win rate (approximately 92%). The β = 0 case (best-of-N, or "top-1-over-N": accept only the maximum-reward response) performs substantially worse (roughly 86%), confirming that pure reward exploitation without KL regularization leads to reward hacking—the accepted responses score highly on the training reward model but produce a policy that generalizes poorly. The β = 0.05 case also underperforms β = 0.5 (roughly 88–89%), showing that too little regularization is nearly as bad as none. The β = 5 case drops to roughly 87%, approaching the performance of sft-sample-rank (shown as a horizontal line in the figure)—when β is too large, rejection sampling barely filters SFT responses, making it nearly equivalent to sampling from π_sft directly. The sft-sample-rank baseline is shown as a horizontal reference line (since it does not use rejection sampling and its β is not defined), sitting at roughly 88.63%—RSO with optimal β = 0.5 improves over this by about 3.7 points.
Preference pair sampling and ranking strategies (Table 2, Reddit TL;DR): Comparing "first-round-rank" (pair adjacent responses in the ranked list) versus "tournament-rank" (full elimination bracket producing n-1 pairs from n responses) across different candidate sources (sft vs. rso) and candidate counts (8 vs. 64). Key findings:
- Tournament ranking inflates Proxy Reward but not AutoSxS: With sft-8-sample, tournament-rank achieves 90.69% Proxy vs. first-round-rank's 88.63% (+2.06), but AutoSxS is essentially identical (68.57% vs. 68.51%). With rso-8-sample, tournament-rank achieves 93.35% Proxy vs. first-round-rank's 92.37% (+0.98), but AutoSxS decreases from 71.86% to 71.69%. This is evidence that tournament ranking introduces a bias the reward model can exploit during training—the policy learns to win more proxy comparisons without genuinely improving. First-round-rank is the optimal choice (highest AutoSxS: 71.86% with rso-8-sample).
- Sampling more candidates (64 vs. 8) without rejection sampling helps less than rejection sampling: sft-64-sample-first-round-rank achieves 68.84% AutoSxS vs. sft-8-sample-first-round-rank at 68.51%—a negligible gain from 8× more candidates. The bottleneck is the SFT sampling distribution itself: generating more candidates from π_sft doesn't help if they all come from an off-policy distribution. Rejection sampling (rso-8-sample-first-round-rank: 71.86%) substantially outperforms raw SFT sampling at any scale.
- rso-8-sample-first-round-rank is the optimal configuration by AutoSxS (71.86%), outperforming all tournament variants and all larger-sample variants.
Regularization loss in SLiC (Table 5, Appendix A.6, Reddit TL;DR): SLiC's original loss includes a regularization term -λ log π_θ(y_ref | x) encouraging the policy to maintain probability on the SFT target. Sweeping λ = 0, 0.5, 5, 50, 500 with hinge loss and sft-sample-rank preference pairs shows Proxy Reward win rates all in the 90.06–90.83% range and AutoSxS in the 67.34–67.84% range—no significant trend and negligible variation. The paper drops this regularization in the hinge-norm formulation, noting it "does not show significant improvement." This is an informative negative result: the SFT regularization that seemed theoretically motivated (stay close to the supervised target) provides no empirical benefit when training on preference pairs, likely because the preference discrimination signal already anchors the policy close to reasonable outputs.
Hinge vs. hinge-norm loss on reward hacking: Across Table 1, the original hinge loss (SLiC style, without normalization by SFT probabilities) consistently shows higher Proxy Reward but lower AutoSxS than hinge-norm, particularly with rso-sample-rank. On Reddit TL;DR: hinge + rso-sample-rank = 93.36% Proxy, 69.26% AutoSxS; hinge-norm + rso-sample-rank = 92.80% Proxy, 70.84% AutoSxS. The 1.58-point AutoSxS gap with hinge-norm despite lower Proxy Reward suggests hinge-norm is more resistant to reward model exploitation. On AnthropicHH, the gap is even larger: hinge + rso-sample-rank = 82.21% Proxy, 32.56% AutoSxS; hinge-norm + rso-sample-rank = 84.44% Proxy, 38.58% AutoSxS (+6.02 AutoSxS points). The improved AutoSxS of hinge-norm over hinge with rso-sample-rank pairs is an important robustness finding: normalizing by the SFT policy's probabilities (the log(π_θ/π_sft) term) regularizes the optimization and prevents the policy from learning extreme probability ratios that exploit the reward model.
Scale of the policy model (Table 3 vs. Table 1): RSO's advantage over DPO is task-dependent at scale. On Reddit TL;DR at T5-XXL: RSO (86.01% AutoSxS) beats DPO (85.03%) by +1.1 points. On AnthropicHH at T5-XXL: RSO (70.26%) beats DPO (52.80%) by +17.46 points. This asymmetry suggests RSO is most valuable when the base task is hard enough that even a large policy cannot learn good preferences from off-policy data alone (AnthropicHH) or when the human preference data is particularly far from the optimal policy distribution (as the enormous direct-to-sft-sample-rank gap of 24.01% → 39.56% at T5-large scale indicates for AnthropicHH). When the task is relatively easy and the base SFT policy already produces high-quality outputs (Reddit TL;DR at T5-XXL scale), the additional sampling provides diminishing returns.

Critical Assessment

The experiments demonstrate a clear and consistent empirical pattern: training on preference pairs sampled closer to the optimal policy (rso-sample-rank) yields better aligned policies than training on off-policy pairs (direct) or SFT-policy pairs (sft-sample-rank). This pattern holds across two tasks, three loss functions, two policy scales, and four evaluation metrics (including human evaluation). The monotonic improvement from direct → sft-sample-rank → rso-sample-rank within every loss function column of Table 1 is the paper's central empirical result, and it is robust.

However, there are important limitations in what the experiments actually demonstrate versus what the paper claims.

On the claim that RSO is closer to on-policy estimation of π*: The paper's theoretical argument is that rso-sample-rank generates preference pairs from π_rψ, the optimal policy induced by the trained reward model r_ψ, and that this is closer to the true optimal policy π* than either π_sft or the unknown human data distribution π_unk. The experiments provide strong evidence that rso-sample-rank is better than the alternatives, but they provide no direct evidence that this is because of proximity to π*. The reward model r_ψ has 73–70% accuracy on the human preference validation sets. If the reward model is systematically biased in certain regions of response space, then π_rψ could be further from π* than π_sft in those regions, even if it's better on average. The paper does not decompose performance by difficulty, topic, or any other stratification that might reveal where the reward model's guidance helps versus hurts. This is a missing analysis: do the gains from rso-sample-rank come uniformly from all prompts, or are they concentrated on prompts where the reward model is highly accurate?

On the claim that the language model "learns better from an explicit reward model": The experiments support this when comparing sigmoid-norm-direct (no reward model, 67.72% AutoSxS) against sigmoid-norm-rso-sample-rank (with explicit reward model, 71.86% AutoSxS). But this comparison confounds two changes: (1) using a reward model vs. not, and (2) sampling from π_rψ vs. using off-policy human data. The sft-sample-rank condition uses the reward model but samples from π_sft (not π_rψ), achieving 69.02% AutoSxS—an improvement of only +1.3 points over DPO-direct. This suggests that simply having a reward model is not the dominant factor; it's the combination of the reward model with rejection sampling toward π* that drives the gains. The claim that discrimination is "easier to learn" than generation is not directly tested—the paper never compares the reward model's discrimination accuracy against the policy's implicit discrimination accuracy. A direct test would be: compute the implicit reward β log(π_θ/π_sft) from DPO's trained policy and measure its agreement with human preference labels, then compare against the explicit T5-XXL reward model's agreement. No such comparison is made.

On the generalization of findings: All experiments use the T5 model family. The paper argues (Section 5.2, Table 3) that RSO scales to T5-XXL, but this is within the same architecture. The paper provides no evidence with decoder-only models (GPT-style), which dominate contemporary LLM development. The reward-ranking model and policy model are both encoder-decoder T5 variants; whether the RSO recipe transfers to architectures where policy and reward models are structurally different (e.g., a GPT policy with a T5 reward model, or a purely decoder-based reward model) is untested. This is a significant gap given that the paper's proposed deployment uses asymmetric model capacities specifically because "discrimination is easier than generation"—if the asymmetry claim is architecture-dependent, the general recipe may not hold.

On the missing RLHF baseline: The paper explicitly excludes RLHF (PPO) as a baseline, stating "we lack expertise on RLHF and DPO shows it to be a competitive alternative." While RSO's goal is to improve upon offline methods (DPO, SLiC), the central claim that rso-sample-rank provides "on-policy-like" data is a claim about approximating the benefits of online RLHF without its complexity. Without comparing RSO against an actual online RLHF baseline (PPO with the same reward model, policy, and reference model), we cannot assess how close RSO gets to the "true" on-policy performance. It's possible that RSO with one round of rejection sampling substantially underperforms multi-step PPO that updates the policy and resamples iteratively. The paper's "future work" mentions "online variants," acknowledging this gap implicitly.

On the cost accounting: The paper claims (Appendix A.10) that the additional sampling for RSO accounts for "less than 10% of the total training time." This figure is unverified by any timing measurements or formal analysis. Moreover, the 10% figure depends on implementation details: parallelization across many TPU/GPU servers, prompt caching hitting rates, and batch sizes. In a less optimized setting (e.g., a research lab without massive inference infrastructure), the additional 64 SFT decodes + 68 reward model inferences per training prompt could dominate the cost. The paper's cost analysis assumes "batch decoding is scalable and efficient with many optimizations" and cites Pope et al. (2023), but provides no empirical wall-clock measurements. A fair cost comparison would include the reward model training cost as well—DPO-direct needs no reward model at all, while RSO requires training an 11B T5-XXL reward model. The paper never accounts for this one-time cost.

On the test set and evaluation reliability: Reddit TL;DR has 6k test examples; AnthropicHH has 9k. However, AutoSxS evaluation (which serves as the primary automated metric) uses only 8 decoded samples per comparison with a 0.35 threshold for declaring wins—this is a relatively coarse-grained evaluation. The paper does not report the number of "tie" outcomes in AutoSxS or how many comparisons fall below the 0.35 threshold. Human evaluation has a small sample (47 raters, median 16 tasks each, 3-way comparison) relative to the number of systems being compared. The finding that sft-sample-rank scores lower than direct in human evaluation on Reddit TL;DR (Table 4: 10% chosen vs. 21%, quality 3.74 vs. 3.84) is an unexplained reversal of the automated metric trend and raises questions about the reliability of the human evaluation sample. The paper does not discuss this anomaly.

On the β and γ hyperparameter optimization: Both hyperparameters were tuned on Reddit TL;DR using Proxy Reward as the optimization criterion (Figures 3a, 3b). Optimizing β based on Proxy Reward is concerning because Proxy Reward is measured by the same reward model used in rejection sampling—it's an in-domain evaluation that could reward overfitting to the reward model. The paper then uses these same β and γ values on AnthropicHH without re-tuning, implying transferability. Figure 3b shows that β = 0.5 is optimal for Reddit TL;DR, but the optimal β for AnthropicHH might differ—the paper doesn't check. If the optimal β is domain-dependent (which is plausible given that the reward model's accuracy and the SFT policy's output quality both differ across tasks), then the AnthropicHH results may actually understate RSO's potential.

Experiments that would have strengthened the paper:

Multiple rounds of rejection sampling: RSO currently does one round (SFT → rejection sample → train). Iterating this process (train policy, use trained policy as proposal for new rejection sampling, repeat) would test whether RSO can approach online RLHF performance through repeated offline rounds.
Direct on-policy comparison: Generate preference pairs from an RLHF-trained policy (which is approximately π* if RLHF converges well) and use them to train a new policy via DPO loss. If this achieves similar performance to RSO, it would validate the claim that data distribution is the key factor. If it substantially outperforms RSO, it would suggest RSO's approximation to π* is still imperfect.
Reward model accuracy stratification: Split the test set by the reward model's validation accuracy on similar examples and report RSO vs. DPO performance stratified by reward model reliability. This would clarify whether the RSO gains depend on reward model quality—a crucial practical consideration.
Ablation on reward model scale: In the main experiments, the reward model (11B) is always larger than the policy (770M). What if the reward model is the same size or smaller than the policy? This would test whether the implicit-vs-explicit reward claim is truly about task difficulty or simply about parameter count.
KL divergence measurement: Compute the actual KL divergence between the RSO-trained policy and the SFT policy, and compare with DPO's KL divergence at the same β. The theory predicts that RSO achieves a better reward-KL tradeoff (higher reward at the same KL), but the paper never measures the KL actually achieved by the trained policies.

On the claim about reward hacking resistance: The paper argues that RSO's tunable β provides regularization against reward hacking compared to best-of-N (which sets β = 0 implicitly). Figure 3b supports this: β = 0 performs worse than β = 0.5 by ~6 percentage points of Proxy Reward. However, even at the optimal β = 0.5, the paper still observes reward hacking—hinge loss reports higher Proxy Reward but lower AutoSxS than hinge-norm (Table 1). And within first-round-rank vs. tournament-rank (Table 2), tournament-rank inflates Proxy Reward without improving AutoSxS. So β tuning mitigates but does not eliminate reward hacking. The paper provides no systematic study of how the reward hacking margin changes with reward model accuracy, β, or the number of candidate samples n_c—all of which are theoretically relevant.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For in Headline Gains

The assumption or constraint. RSO's core improvement over DPO and SLiC depends on sampling preference pairs from π_rψ, the approximate optimal policy, via statistical rejection sampling. This requires generating n_c = 64 response candidates per training prompt from the SFT policy, scoring all 64 with the reward model, and then constructing n_d = 8 pairs for training—all before the preference optimization step even begins. The paper acknowledges this cost explicitly in Appendix A.10:

"RSO needs additional computation on sampling from the SFT policy and ranking from the pairwise reward model, but the additional cost is empirically minor compared to policy training."

The paper estimates this overhead at "less than 10% of the total training time," but provides no timing measurements, wall-clock comparisons, or FLOPs accounting to substantiate this claim. The figure depends heavily on implementation details: parallelization across many model servers, prompt caching hit rates, and the fact that reward model inference decodes only a single token ("A" or "B"). In a more resource-constrained setting—a research lab without massive inference infrastructure, or a deployment where reward model serving must compete with other workloads—the 64× SFT decoding factor could dominate end-to-end cost.

The consequence. The headline gains of RSO over DPO (e.g., +4.14 AutoSxS points on Reddit TL;DR, +16.97 points on AnthropicHH at T5-large scale) are reported without amortizing the cost of the additional inference that makes those gains possible. A fair comparison against DPO-direct would account for the fact that DPO requires zero additional inference beyond policy training—it uses the human preference data as-is. If the 64× SFT decoding per prompt adds, say, 50% to total training time in a non-optimized setting (rather than the claimed 10%), the efficiency-adjusted improvement per unit of compute would be substantially smaller than the raw accuracy numbers suggest. Practitioners making resource allocation decisions need to know whether spending that additional inference budget on RSO-style data construction yields better returns than, for instance, simply training DPO for more steps, using a larger policy model, or collecting more human preference data.

Furthermore, the reward model itself must be trained—an 11B-parameter T5-XXL fine-tuned on the human preference dataset. DPO-direct needs no reward model at all. The paper never accounts for this one-time training cost in the comparison. While it amortizes over many policy training runs if the reward model is reused, for a one-off alignment task the cost of training an 11B discriminator is substantial and should be factored into the total resource budget.

What evidence exists in the paper. Appendix A.10 (Table 7) provides a theoretical efficiency comparison showing that RSO requires n_c + n_c - 1 + 0.5 * n_d reward model inferences per prompt versus zero for DPO-direct, but this is a count of operations, not a cost measurement. The gap between DPO (0 SFT inference, 0 reward model inference) and RSO (64 SFT inferences, ~66 reward model inferences per prompt) is acknowledged as a table entry, but never translated into actual time or FLOPs numbers. The paper appeals to batch parallelism and prompt caching for mitigation but provides no empirical backing.

Mitigation status. The paper acknowledges the cost but dismisses it as minor, claiming the overhead is under 10% without measurement. No experiments vary the number of SFT candidates n_c to find a cost-quality Pareto frontier (e.g., would n_c = 16 achieve 90% of the gains at 25% of the cost?). The paper provides no guidance on how to tune n_c for a given computational budget. This is left entirely to future work.

All Results Are on a Single Model Family (Encoder-Decoder T5) Without Decoder-Only Validation

The assumption or constraint. Every experiment in the paper uses T5 models (Raffel et al., 2020): T5-large (770M) for the policy, T5-XXL (11B) for the reward model, with the scaling experiment using T5-XXL for both. T5 is an encoder-decoder architecture that was state-of-the-art in 2020 but has since been largely superseded by decoder-only architectures (GPT, LLaMA, PaLM) for language generation tasks. The paper makes no claim about architecture-specificity, but neither does it acknowledge this as a limitation. The paper assumes its findings transfer:

"we believe this model is representative of the capabilities of many contemporary LLMs"

though this statement appears in the context of model capability, not architecture.

The consequence. Decoder-only models differ from encoder-decoder models in ways that directly affect RSO's pipeline. Encoder-decoder models process the full input (prompt) in the encoder, then generate autoregressively from the decoder—the encoder representations are bidirectional and fixed during decoding. Decoder-only models process the concatenated prompt+response causally, with self-attention over the full sequence. This affects: (1) how the SFT policy generates the 64 candidates (decoder-only models may have different sampling dynamics, especially at temperature 0.7 with top-k=40), (2) how the reward model scores candidates (the reward-ranking model in RSO is also T5, using encoder-decoder processing of the concatenated context and two responses—a decoder-only reward model would process this differently), and (3) most critically, how the policy optimizes the sigmoid-norm or hinge-norm loss, since DPO's derivation and the normalized likelihood ratio log(π_θ/π_sft) depend on token-level autoregressive factorization which is architecturally identical between T5 and decoder-only models for the decoder component, but may interact differently with the encoder conditioning.

The absence of decoder-only experiments is particularly significant given that the dominant alignment method for decoder-only models as of 2024 is DPO, and the paper positions RSO as a direct improvement over DPO. If a practitioner is using LLaMA-2 or Mistral and wants to adopt RSO, the paper provides no evidence that the gains observed with T5 will transfer. There could be architectural interactions: decoder-only models might learn implicit rewards better (narrowing the DPO-RSO gap, as seen on Reddit TL;DR at T5-XXL scale where the gap is only +1.1 AutoSxS points) or might benefit even more from explicit reward guidance (widening the gap, as on AnthropicHH at T5-XXL). The paper provides no way to predict which.

What evidence exists in the paper. None. All models are T5 variants. The cross-task generalization experiment (Reddit TL;DR → CNN/DailyMail, Table 6) provides some evidence of domain transfer within the T5 family, but says nothing about architectural transfer. The paper does not discuss architecture as a variable at all.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not suggest decoder-only experiments as future work, and makes no claims about architectural generality. A practitioner adapting RSO to a decoder-only model would be operating without any direct evidence from this paper.

RSO Never Demonstrates What Happens with Multiple Rounds of Sampling and Training

The assumption or constraint. RSO is presented as a single-round procedure: start with the SFT policy, use the reward model to rejection-sample preference pairs from π_rψ, train a new policy on those pairs, and stop. The paper explicitly notes this is one round and that multi-round iteration is possible:

"Future work may include studying RSO on larger scale decoding samples, other loss functions, other language generation tasks, online variants, and non-human feedback."

The paper does not iterate: it never uses the RSO-trained policy as the proposal distribution for a second round of rejection sampling and retraining.

The consequence. This limitation matters because the distinction between RSO and online RLHF is precisely about iteration. Online RLHF (PPO) repeatedly samples from the current policy, evaluates with the reward model, and updates—each round moves the sampling distribution closer to π*. RSO approximates one step of this process: it samples from π_sft and rejection-filters toward π_rψ, but π_rψ is the optimal policy for the trained reward model, not necessarily for the true human preference function r*. A single round of RSO cannot correct for systematic errors in the reward model—if r_ψ assigns high scores to a particular kind of superficially good but actually flawed response, RSO will acceptance-sample those responses, construct preference pairs favoring them, and the trained policy will amplify those flaws.

Multiple rounds could potentially help: after round 1, the trained policy might produce responses that the reward model scores even higher than the SFT policy's best, and a second round of rejection sampling from this improved policy could yield even better training pairs. Alternatively, multiple rounds could hurt: if the reward model has blind spots, each round could amplify policy exploitation of those blind spots, similar to how repeated RLHF training can lead to reward over-optimization (Gao et al., 2023). The paper provides no evidence either way.

The connection to online RLHF is the key missing comparison. RSO's selling point is that it achieves on-policy-like data without the complexity of online RL. But without comparing to actual online RLHF, we don't know how much of the gap to "true" on-policy training RSO closes. If one round of RSO achieves 90% of what multi-round PPO would achieve, that's a strong result. If it achieves 50%, the single-round limitation is severe.

What evidence exists in the paper. The paper includes RAFT and ReST as baselines, both of which can be seen as single-round methods using the reward model to filter SFT responses. RSO substantially outperforms both, showing that its rejection sampling + preference loss combination is better than simple filtering + cross-entropy for one round. But neither RAFT nor ReST is an iterative method in this comparison (RAFT is described as a single round of best-of-1 selection; ReST as one round of "grow and improve"). The paper provides no iterative baseline.

The T5-XXL scaling results (Table 3) are suggestive: the RSO vs. DPO gap on Reddit TL;DR shrinks from +4.14 points (T5-large) to +1.1 points (T5-XXL). This could indicate that as the policy gets stronger, the marginal benefit of one round of rejection sampling diminishes—the SFT policy at 11B already produces high-quality responses, so rejection sampling doesn't add much over what DPO can learn from human data. If this trend continues, a second round of RSO at T5-XXL scale might provide negligible additional gain, or it might break through a plateau. The paper cannot distinguish these possibilities.

Mitigation status. The paper lists "online variants" as future work, which implicitly acknowledges the single-round limitation. No experiments address it. The optimal β = 0.5 found in Figure 3b is tuned for one round and might not be optimal for iterative application (later rounds might need different β as the policy distribution shifts).

Human Evaluation Results Contain an Unexplained Reversal That Undermines the Data Distribution Narrative

The assumption or constraint. The paper's central claim is a monotonic ordering: rso-sample-rank > sft-sample-rank > direct in terms of policy quality, because each step moves the training data distribution closer to π*. This ordering holds consistently across all automated metrics (Proxy Reward, Gold Reward, AutoSxS) on both tasks (Table 1). The paper extends this claim to human evaluation without qualification.

The consequence. The human evaluation results in Table 4 break this monotonicity on Reddit TL;DR. Under sigmoid-norm loss, human raters chose sft-sample-rank as preferred only 10% of the time, compared to direct (DPO) at 21%—a reversal of the automated metric trend where sft-sample-rank outperforms direct (69.02% vs 67.72% AutoSxS). Under hinge-norm loss, the same reversal appears: sft-sample-rank is chosen 11% of the time vs. direct at 21%. The quality scores show the same pattern: sft-sample-rank scores 3.74/5 (sigmoid-norm) and 3.68/5 (hinge-norm), both below direct at 3.84/5 and 3.80/5 respectively.

This is a significant anomaly. If sft-sample-rank (sampling preference pairs from π_sft and ranking with the reward model) actually produces worse responses than direct (training on the original off-policy human data) as judged by humans, then the monotonic improvement from moving the sampling distribution toward π* is not as robust as the automated metrics suggest. It raises the possibility that the reward model introduces systematic biases that automated metrics (themselves based on models trained on similar data) fail to detect, but humans notice. The paper provides no analysis of what kinds of errors sft-sample-rank makes that direct does not, or why humans prefer the direct-trained policy over the sft-sample-rank-trained policy despite automated metrics favoring the latter.

What evidence exists in the paper. Table 4 contains the anomalous numbers. The paper never discusses them. The text in Section 5.3 states:

"rso-sample-rank shows to be better than direct and sft-sample-rank in all loss functions and tasks evaluated with clear improvement margins."

This statement is true for rso-sample-rank—it dominates all conditions in human evaluation. But it elides the fact that sft-sample-rank underperforms direct in human evaluation on Reddit TL;DR, which contradicts the paper's own narrative that sft-sample-rank is an intermediate step on the path from direct to rso-sample-rank.

The human evaluation has a small sample: 47 raters with a median of 16 tasks each, 3-way comparisons. With only ~150 total judgments (16 tasks × 3 replicas = 48 judgments per rater × ~3 raters per task), the confidence intervals on the 10% vs. 21% preference rates are wide. It's possible the reversal is statistical noise. But the paper doesn't report confidence intervals or test for statistical significance of the differences in Table 4, making it impossible to assess whether the anomaly is real or an artifact of the small sample.

Mitigation status. Not addressed. The paper does not acknowledge this reversal, does not discuss it, and does not provide the statistical detail needed to assess whether it is a genuine effect or noise. This is a gap in the paper's self-critical analysis: the flagship human evaluation contains a result that directly contradicts the paper's explanatory framework, and it goes unexamined.

The Reward Model's Accuracy Is Modest, and RSO Provides No Guidance on How Reward Model Quality Affects the Approach

The assumption or constraint. RSO's entire pipeline depends on the pairwise reward-ranking model ρ_ψ to (1) score SFT candidates for rejection sampling, and (2) label the constructed preference pairs for policy training. The paper reports this model's validation accuracy as 73.23% on Reddit TL;DR summarization and 69.75% on AnthropicHH dialogue. These are moderate accuracies—the reward model is wrong about human preferences roughly 27–30% of the time.

The paper treats the reward model as a fixed, given component and never studies how RSO's performance varies with reward model quality. There is no ablation where the reward model is artificially degraded (trained on less data, reduced in size) or improved (trained on more data, ensembled) to map out the relationship between discriminator accuracy and final policy quality. The paper's claim that "comparing between two responses (reward) is easier to learn than generating high quality responses (policy)" is stated as a general principle, but the experiments only test one reward model at one accuracy level.

The consequence. A practitioner considering RSO needs to know: how good does my reward model need to be for RSO to outperform DPO? If the reward model has only 60% accuracy, does RSO still help, or does the noisy acceptance sampling introduce more harm than benefit? If the reward model has 85% accuracy, do the gains continue to scale, or do they saturate?

This is particularly important because RSO uses the reward model in two distinct ways that might have different accuracy requirements. The rejection sampling step (accepting/rejecting based on exp((r - r_max)/β)) uses the reward model to reshape the sampling distribution—this is sensitive to the reward model's ranking quality (is the ordering of responses by reward correct?), not just its binary preference accuracy. The preference labeling step (constructing (winner, loser) pairs) uses the reward model as a substitute for human labels—this is sensitive to the reward model's pointwise preference accuracy (does it correctly identify which of two specific responses is better?). A reward model with good ranking but mediocre pointwise accuracy might excel at rejection sampling but introduce label noise in training. A reward model with good pointwise accuracy but poor calibration might provide clean training labels but fail to correctly identify the highest-reward responses for rejection sampling. The paper provides no decomposition of these two roles.

The 73%/70% accuracy numbers also mean that ~27–30% of the reward model's preference judgments in the rejection sampling score derivation and pair labeling are incorrect. The paper provides no analysis of whether these errors are concentrated on particular types of prompts or responses where RSO might actually be harmful.

What evidence exists in the paper. Table 1 provides indirect evidence: the gap between RSO and DPO-direct is much larger on AnthropicHH (where the reward model is less accurate, 69.75%) than on Reddit TL;DR (where the reward model is more accurate, 73.23%). At T5-large scale, the AutoSxS gap is +16.97 points on AnthropicHH vs. +4.14 on Reddit TL;DR. At T5-XXL scale, +17.46 vs. +1.1 points. These numbers suggest RSO helps more when the reward model is less accurate—a counterintuitive result that the paper does not discuss. The likely explanation is that the human preference data on AnthropicHH is so off-policy that even a noisy reward model provides a better signal than the original human labels, while on Reddit TL;DR the human data is already relatively on-policy, so the noisy reward model adds less. But this interpretation is speculative without direct reward-model-quality ablations.

The Gold Reward Model (a separate PaLM 2-S trained on the same data) has accuracy of 76.07% on summarization and 70.18% on dialogue—similar to the training reward model's accuracy. This confirms that the accuracy numbers are not due to overfitting but reflect the inherent difficulty of the preference discrimination task. It also suggests that simply training a larger or differently-architected reward model might not substantially improve accuracy on this data, limiting how much better RSO could get from reward model improvements alone.

Mitigation status. The paper does not address this limitation. It does not ablate reward model quality, does not discuss minimum accuracy requirements for RSO to be beneficial, and does not provide error analysis of the reward model's mistakes. The treatment of β as the key hyperparameter controlling trust in the reward model (Section 3.2) is the paper's implicit acknowledgment that the reward model is imperfect—β > 0 provides regularization against reward model errors. But the paper only sweeps β for one reward model at one accuracy level; it provides no guidance on how β should be set as a function of reward model accuracy. A practitioner with a 60%-accurate reward model has no way to know whether β = 0.5 (optimal for the paper's 73%-accurate model) is appropriate.

The Paper Provides No Comparison to Online RLHF, Leaving the "On-Policy" Claim Incompletely Validated

The assumption or constraint. RSO is motivated by the claim that training on preference pairs from a distribution closer to π* improves estimation of the optimal policy. The paper frames rso-sample-rank as providing "on-policy-like" data that approximates what online RLHF achieves through iterative sampling:

"Our statistical rejection sampling refers to the one in the statistical field... we show that top-k-over-N is a special case of our statistical rejection sampling and it is critical to balance between the reward exploitation and regularization towards the SFT policy."

But the paper never compares RSO against an actual online RLHF baseline. It explicitly excludes RLHF:

"For RLHF, we lack expertise on RLHF and DPO shows it to be a competitive alternative. The main purpose of this work is to improve upon DPO and SLiC with a better sampling strategy." (Appendix A.8)

The consequence. Without an RLHF comparison, the paper cannot substantiate the claim that rso-sample-rank is closer to on-policy in a way that matters. Online RLHF with PPO iteratively samples from the current policy, evaluates with the reward model, and updates—this process can, in principle, converge to π* if the reward model is accurate and the optimization is stable. RSO does one round of rejection sampling from the SFT policy. We don't know how much of the gap between offline (DPO on off-policy data) and online (PPO on iteratively updated on-policy data) RSO closes. It could close 90% of the gap, making the additional complexity of PPO unjustified. Or it could close 30%, leaving substantial room for improvement.

This matters for practitioners choosing between deployment strategies. If a team already has RLHF infrastructure (PPO training loop, value model, etc.), switching to RSO would simplify training but might degrade final quality. If a team is building alignment infrastructure from scratch, RSO might be the right choice for simplicity, but the paper provides no evidence about how much quality is sacrificed relative to full online RL. The paper's silence on this comparison makes it an incomplete guide for system design.

The gap between one-round RSO and multi-round online RLHF is likely to be largest when the SFT policy's output distribution is far from π*. In that case, one round of rejection sampling can only accept the best of what the SFT policy produces—it cannot create entirely new kinds of high-quality responses that the SFT model would never generate. Multi-round RLHF, by updating the policy and resampling, can progressively shift the generation distribution into regions the original SFT model had near-zero probability of reaching. On tasks where the SFT model is already strong (e.g., Reddit TL;DR at T5-XXL scale, where DPO achieves 85% AutoSxS), one round of RSO might nearly match online RLHF because the SFT distribution already covers the high-quality region. On tasks where the SFT model is weak, the single-round limitation could be severe.

What evidence exists in the paper. The T5-XXL scaling results (Table 3) provide indirect evidence. On Reddit TL;DR, the DPO baseline already achieves 85.03% AutoSxS—only 1.1 points below RSO. If online RLHF could push performance further (say, to 90%), then RSO is leaving ~4 points on the table relative to online methods. On AnthropicHH, DPO achieves only 52.80% AutoSxS while RSO reaches 70.26%—a 17.46-point gap. This could mean RSO is much closer to the online RLHF ceiling on AnthropicHH than DPO is, or it could mean that even 70.26% is far below what online RLHF could achieve, and RSO's single round is insufficient. The paper provides no way to distinguish.

The RAFT and ReST baselines are both single-round methods (like RSO) and RSO substantially outperforms them, showing that RSO's specific combination of rejection sampling and preference loss is the best single-round method tested. But this doesn't answer the multi-round question.

Mitigation status. The paper lists "online variants" as future work but provides no analysis of the single-round vs. multi-round tradeoff. It does not discuss what properties of a task determine whether one round is sufficient or whether multiple rounds are needed. The β = 0.5 hyperparameter is optimized for one round; the paper provides no guidance on how β should change (if at all) across multiple rounds as the proposal distribution shifts from π_sft to π_θ^(1) to π_θ^(2). This is a significant open question for any practitioner wanting to apply RSO iteratively.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a methodological reframing rather than a paradigm shift. It does not introduce a fundamentally new class of algorithms—statistical rejection sampling, logistic regression, and SVMs are all well-established techniques. Rather, it identifies that the core bottleneck in offline preference optimization is not the loss function (the axis the field had been optimizing along, with DPO's sigmoid loss and SLiC's hinge loss as competing alternatives) but the sampling distribution of the training data. The reframing is this: DPO and SLiC are both solving the same statistical estimation problem—fitting a Bradley-Terry preference model where the logit is the policy's normalized log-probability ratio γ log(π_θ/π_sft)—and they differ only in choice of classification loss (logistic regression vs. SVM). The quality of the resulting estimator depends more on whether the training pairs are sampled from a distribution close to the optimal policy π* than on which loss function is used to fit them.

This shifts the optimization target from "design a better loss" to "construct better training data," which is a different kind of research problem. It directs attention toward reward modeling, sampling strategies, and distribution-shift mitigation rather than loss function engineering. The unified framework (the 3×3 grid of loss function × data source in Table 1) makes this shift explicit: within every loss function column, moving from direct → sft-sample-rank → rso-sample-rank yields consistent, substantial improvements (e.g., +4.14 AutoSxS points on Reddit TL;DR with sigmoid-norm); across loss functions within the same data column, differences are smaller (1.02 points between sigmoid-norm and hinge-norm, both with rso-sample-rank). The implication is clear: the field's marginal returns to better data construction exceed its marginal returns to better loss functions, at least at the current frontier.

The work also reconciles a tension that had been latent in the DPO literature but never directly addressed. DPO's derivation shows that the reward function can be eliminated analytically from the preference optimization objective—the language model "is secretly a reward model." This created an implicit narrative that explicit reward models are unnecessary, vestigial components of the older RLHF pipeline. RSO challenges this by arguing that discrimination (pairwise preference judgment) is fundamentally easier to learn than generation (autoregressive sequence modeling), and therefore maintaining an explicit reward model—potentially at larger scale than the policy—enables better training data construction than what the policy could produce through implicit self-evaluation alone. The evidence for this is the gap between DPO-direct (no reward model, 67.72% AutoSxS) and RSO with rso-sample-rank (explicit 11B reward model guiding 770M policy, 71.86% AutoSxS), which represents a 6.1% relative improvement using the same sigmoid-norm loss. The reframing is not that DPO is wrong—its derivation is mathematically correct—but that the elimination of the reward model, while elegant, discards a capability (guided sampling toward π*) that has empirical value.

The paper also clarifies the relationship between best-of-N sampling and KL-regularized policy optimization by showing that best-of-N (called "rejection sampling" in the AnthropicHH and Llama2 literature) is the β → 0 limit of statistical rejection sampling from π_r. This connects a heuristic practice to a principled framework and reveals β as the crucial hyperparameter controlling the exploitation-regularization tradeoff. Prior work treated best-of-N as a separate technique; RSO shows it's a point on a continuum, and that intermediate β values (0.5 in the paper's experiments) yield better policies by avoiding the reward hacking that pure best-of-N suffers from. This makes the connection between sampling and the RLHF objective explicit in a way that had been obscured by terminological confusion.

In terms of which research directions become more attractive: reward model quality and robustness becomes the central bottleneck to improve, since RSO's performance depends on the reward model's ability to score and rank candidates. Improving the reward model—through better training data, architectural innovations, or ensemble methods—becomes a high-leverage investment because it improves both the rejection sampling distribution and the preference pair labeling. Conversely, research directions focused on increasingly sophisticated loss functions for offline preference optimization become less attractive, since the paper shows that the choice between logistic regression and SVM-style losses matters less than the data distribution those losses are applied to.

Follow-Up Research This Work Enables

1. Multi-round iterative RSO and the gap to online RLHF. The paper explicitly limits itself to one round of rejection sampling (SFT policy → rejection sample → train → stop). The most immediate extension is to close the loop: use the RSO-trained policy π_θ^(1) as the proposal distribution for a second round of rejection sampling, generating n_c candidates from π_θ^(1), scoring with the same frozen reward model, rejection-sampling n_d responses, constructing preference pairs, and training π_θ^(2). A systematic study would run this for k = 1, 2, 4, 8 rounds on both Reddit TL;DR and AnthropicHH, measuring whether performance saturates, continues to improve, or degrades (due to reward model over-optimization compounding across rounds). The key comparison is against online RLHF (PPO with the same reward model and reference policy) run for a matched number of policy updates. This would answer the central open question RSO raises: how much of the benefit of on-policy data can be recovered through repeated offline rejection sampling rounds, versus requiring true online interaction? The paper's T5-XXL results (Table 3) are suggestive—RSO's advantage over DPO shrinks to +1.1 AutoSxS points on Reddit TL;DR but remains large at +17.46 points on AnthropicHH—and multi-round iteration would reveal whether the shrinking gap on easier tasks means RSO is converging to the same ceiling as DPO, or whether multiple rounds can push past DPO's asymptote.

2. Reward model quality ablation and minimum-accuracy thresholds. The paper uses a single reward model with 73.23% validation accuracy on Reddit TL;DR and 69.75% on AnthropicHH, and sweeps β to find the optimal trust level for that specific accuracy. A systematic study would train a family of reward models at deliberately varied accuracy levels—by subsampling the human preference training data at rates from 1% to 100%, by varying the reward model size (T5-small through T5-XXL), or by training reward models with different amounts of label noise injected—and run the full RSO pipeline (rejection sampling + sigmoid-norm training) for each. The output would be a curve mapping reward model accuracy to final policy AutoSxS win rate, with DPO-direct as a baseline (which doesn't use the reward model). This would answer the practically crucial question: how accurate does a reward model need to be for RSO to outperform DPO? The paper's current results show RSO beats DPO with a 73%-accurate reward model on summarization and a 70%-accurate reward model on dialogue, but it's unknown whether the crossover point is at 60%, 65%, or 68%. This study would also reveal whether the optimal β shifts systematically with reward model accuracy—intuition suggests that less accurate reward models need larger β (more regularization, less trust in reward scores), but the paper provides no evidence.

3. RSO with decoder-only architectures and at modern LLM scale. All experiments use encoder-decoder T5 models at 770M–11B scale. The field has largely moved to decoder-only architectures (GPT, LLaMA, Mistral) at much larger scales (7B–70B+). A direct replication of the main Table 1 experiment using, for example, LLaMA-2-7B as the SFT policy, LLaMA-2-13B or a fine-tuned LLaMA-2-7B as the reward-ranking model, and the same Reddit TL;DR and AnthropicHH datasets would establish whether RSO's gains are architecture-dependent. The key open question is whether decoder-only models, which process prompts and responses causally rather than through a bidirectional encoder, learn implicit rewards more effectively—potentially narrowing the DPO-RSO gap. The AnthropicHH results at T5-large scale show a massive +16.97 AutoSxS point improvement from RSO over DPO, while Reddit TL;DR shows +4.14 points. If decoder-only models close this gap on dialogue tasks (because causal attention provides better implicit reward representations), RSO's value proposition would be domain-dependent in a way the current paper cannot assess. Conversely, if the gap remains large or widens, RSO becomes even more important for the dominant model architecture. This experiment is straightforward to design—it requires no new datasets or algorithms, only a model family swap—and would substantially clarify the generality of the paper's claims.

4. Decomposing RSO's gains: rejection sampling vs. preference pair labeling vs. reward model scale. RSO's improvement over DPO-direct combines three changes: (1) using a reward model at all (versus absorbing it into the policy), (2) rejection sampling to shift the candidate distribution toward π_rψ (versus sampling from π_sft), and (3) using a reward model that is larger than the policy (11B vs. 770M in the main experiments). A careful ablation would isolate these factors. The design: fix the loss function (sigmoid-norm) and test the following conditions: (a) DPO-direct (no reward model, off-policy human data), (b) sft-sample-rank with the same-sized reward model as the policy (T5-large reward, T5-large policy), (c) sft-sample-rank with an oversized reward model (T5-XXL reward, T5-large policy, as in the paper), (d) rso-sample-rank with same-sized reward model, (e) rso-sample-rank with oversized reward model (the paper's default RSO). Comparing (b) vs. (c) isolates the effect of reward model scale for SFT-policy sampling; comparing (c) vs. (e) isolates the effect of rejection sampling (π_rψ vs. π_sft) given a large reward model; comparing (a) vs. (b) isolates the value of having any explicit reward model at the same scale as the policy. The paper's current sft-sample-rank condition uses a T5-XXL reward model with a T5-large policy, confounding reward model scale with reward model presence. This ablation would clarify whether the paper's claim—"discrimination is easier than generation, so the reward model should be larger"—is causally supported or whether simply having any explicit reward model (even same-scale) captures most of the gain.

5. Stratified analysis by reward model confidence and failure modes. The paper reports aggregate metrics that average over all test examples, but the reward model's accuracy of ~70–73% means it is making systematic errors on ~27–30% of preference judgments. A critical follow-up would stratify test-set performance by the reward model's confidence or accuracy on that example type: bucket prompts by the reward model's validation-set accuracy on similar prompts (where similarity could be defined by prompt length, topic clustering, or reward model ensemble disagreement) and report RSO's AutoSxS win rate separately for each bucket, alongside DPO-direct's performance on the same buckets. The hypothesis: RSO substantially outperforms DPO on prompts where the reward model is highly accurate (the rejection sampling correctly pushes the distribution toward genuinely better responses), but may underperform DPO on prompts where the reward model is systematically wrong (the rejection sampling amplifies the reward model's blind spots, constructing preference pairs that teach the policy to prefer flawed responses). If such a crossover exists, it would provide a diagnostic for when to use RSO vs. DPO based on estimated reward model reliability per prompt, and would motivate research on detecting reward model uncertainty at inference time. The paper's current β = 0.5 is a global setting; a per-prompt β that scales with estimated reward model confidence could improve performance on the subset of prompts where the reward model is unreliable.

6. RSO for multi-objective alignment with separate reward models per objective. The paper treats alignment as a single-objective problem (overall human preference). In practice, alignment often involves balancing multiple competing objectives—helpfulness, harmlessness, honesty, conciseness—that may conflict. AnthropicHH, for instance, has separate helpfulness and harmlessness preference datasets. RSO's statistical rejection sampling framework naturally extends to multi-objective settings: train separate reward models for each objective, compute a vector of reward scores per candidate response, and design an acceptance criterion based on a weighted combination or a Pareto-dominance rule. For example, a candidate might be accepted with probability based on exp((α·r_helpful + (1-α)·r_harmless - r_max)/β), where α is a tunable tradeoff parameter. A concrete experiment: use the AnthropicHH helpfulness and harmlessness splits to train two separate T5-XXL reward models, run RSO with varying α from 0 (pure harmless) to 1 (pure helpful), and evaluate the resulting policies on both helpfulness and harmlessness metrics, mapping out the Pareto frontier. Compare this against multi-objective DPO (Zhou et al., 2023, which the paper cites) to test whether RSO's on-policy data construction provides the same benefits in the multi-objective setting as in the single-objective setting. This is a natural extension that the paper's framework makes straightforward—the rejection sampling algorithm (Algorithm 1) generalizes immediately to vector-valued rewards if the acceptance criterion is scalarized.

Practical Applications and Downstream Use Cases

1. Cost-efficient preference optimization for teams without RL infrastructure. RSO is an offline method that requires only supervised fine-tuning capabilities and reward model inference—no value model, no PPO training loop, no online interaction between policy and environment. For a team deploying a language model that already has access to human preference data (e.g., a company fine-tuning an open-source model like LLaMA-2 for a specific customer-support task with internally collected preference annotations), RSO provides a concrete recipe: train a pairwise reward model on the preference data (using the same model architecture or a slightly larger one), run rejection sampling once from the SFT policy using the paper's β = 0.5 and n_c = 64 settings, construct preference pairs via first-round ranking, and train with sigmoid-norm loss at γ = 0.05. The paper's results suggest this yields a policy that is preferred roughly 2× as often as a DPO-trained policy in human evaluation (48% vs. 21% on Reddit TL;DR, 31% vs. 15% on AnthropicHH, Table 4), while being substantially simpler to implement than online RLHF. The additional inference cost (64 SFT decodes per prompt for candidate generation) is parallelizable and, if the paper's "less than 10% of training time" claim holds approximately, represents a modest overhead for a significant alignment quality improvement. The primary practical requirement is serving infrastructure that can handle batched SFT decoding and reward model inference—a lower bar than the four-model PPO serving infrastructure.

2. Data-efficient alignment when human preference data is scarce or highly off-policy. The AnthropicHH results in Table 1 show an extreme case: DPO-direct (training on the original human preference data) achieves only 24.01% AutoSxS at T5-large scale and 52.80% at T5-XXL scale, while RSO (sigmoid-norm, rso-sample-rank) achieves 40.98% and 70.26% respectively. The gap—+16.97 points at T5-large, +17.46 points at T5-XXL—suggests that the human preference data for AnthropicHH is so far off-policy that training on it directly barely works, while the reward model (even at 69.75% accuracy) provides a substantially better training signal by enabling rejection sampling toward higher-quality responses. For practitioners working with preference data collected from diverse, uncontrolled sources—user feedback from production systems, crowdworker annotations on model outputs from multiple different policies, or historical data where the generating policy is unknown—RSO offers a way to extract useful signal that DPO cannot. The recipe: train a reward model on the available preference data (which may be noisy and off-policy, but the reward model only needs to learn pairwise discrimination, which is an easier problem), then use that reward model to construct on-policy-like preference pairs from the current SFT model. The paper's cross-task generalization result (Table 6, CNN/DailyMail) further suggests this works even when the preference data and the deployment domain differ—the reward model's discrimination ability transfers even when the policy's generation distribution shifts.

3. Asymmetric compute allocation for alignment: larger discriminators, smaller generators. The paper's finding that an 11B reward model guiding a 770M policy (RSO) outperforms a 770M policy training directly on preference data (DPO) has direct implications for hardware-constrained deployments. If a team has a fixed parameter budget for serving—say, they can afford to run inference with a 1B-parameter model at their required throughput—RSO suggests that alignment quality can be improved by spending additional offline compute on a larger reward model (which is only used during training, not at inference time) rather than on a larger policy model. The asymmetric scaling principle (spend more parameters on the discriminator during training because discrimination is easier to learn, then deploy a smaller generator) is validated by the paper's T5-large policy + T5-XXL reward model results. A concrete deployment scenario: a mobile keyboard application that needs to run a small language model (<1B parameters) for text suggestions on-device, with latency constraints preventing larger models. Using RSO, the offline training pipeline uses a much larger reward model (potentially cloud-hosted, not subject to the same latency constraints) to construct high-quality preference pairs for aligning the small on-device model, achieving better alignment than the small model could learn from off-policy data alone or from DPO. The paper provides no direct evidence at the <1B scale, but the principle is testable and the T5-large (770M) results are at the boundary of this regime.

4. Preference optimization when human annotation budget is the binding constraint. The paper does not study annotation efficiency directly, but its framework suggests a strategy: rather than spending a fixed human annotation budget on labeling arbitrary response pairs (which may be off-policy and provide low-value training signal for estimating π*), use a portion of the budget to train an initial reward model, use that reward model to rejection-sample high-quality candidate responses from the SFT policy, and then spend the remaining annotation budget on labeling pairs constructed from those high-quality candidates. This concentrates human effort on discriminating among responses in the region of output space where π* has high density, which should provide more information per annotation dollar about the shape of r* near the optimal policy. The paper's results provide indirect support: sft-sample-rank (which uses the reward model to construct pairs from SFT samples) outperforms direct (which uses the original human-labeled pairs) on all automated metrics (Table 1), suggesting that even existing human labels can be used more efficiently by first training a reward model and then generating new pairs, rather than training directly on the original data. A formal study of this annotation-efficiency hypothesis would be a natural follow-up. </response>

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023· training-methods← LLM-judged similar
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback2023· alignment← LLM-judged similar
- Fine-Tuning Language Models with Just Forward Passes2023· training-methods← in-domain predecessor
- DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization2025· training-methods← similar topic
Gated recurrent neural networks discover attention2023· rl-training← in-domain predecessor
- Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization2023· rl-training← in-domain predecessor
Stabilizing RLHF through Advantage Model and Selective Rehearsal2023· alignment← topic thread
- Read More2023· alignment← in-domain predecessor
WPO: Enhancing RLHF with Weighted Preference Optimization2024· rl-training← similar topic
- Offline Regularised Reinforcement Learning for Large Language Models Alignment2024· rl-training← in-domain predecessor
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline2024· evaluation← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

Contrastive Prefence Learning: Learning from Human Feedback without RL2023· rl-training→ in-domain predecessor
- Model-free Posterior Sampling via Learning Rate Randomization2023· rl-training→ in-domain predecessor
- Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 22023· training-methods→ topic thread
- Learning to summarize from human feedback2020· rl-training→ similar topic
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions2024· rl-training→ similar topic
- Reinforcement Learning from Human Feedback2025· rl-training→ similar topic
Efficient RLHF: Reducing the Memory Usage of PPO2023· training-methods→ topic thread
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models2023· training-methods→ in-domain predecessor
- A General Theoretical Paradigm to Understand Learning from Human Preferences2023· alignment→ topic thread
- RLHF Workflow: From Reward Modeling to Online RLHF2024· training-methods→ similar topic
WPO: Enhancing RLHF with Weighted Preference Optimization2024· rl-training→ similar topic
- Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code2024· rl-training→ in-domain predecessor
- Towards a Unified View of Preference Learning for Large Language Models: A Survey2024· alignment→ topic thread
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization2026· rl-training→ similar topic