ArXiv: 2405.19107
🎯 Pitch
You don't need costly pairwise preference data to align LLMs—simple thumbs-up/down feedback on single responses is enough, and it works better than prior single-trajectory methods like KTO. By jointly learning a value function that predicts per-token rewards alongside the policy, DRO significantly outperforms baselines, provided you keep policy and value parameters separate.
1. Executive Summary
This paper introduces Direct Reward Optimisation (DRO), a framework for aligning LLMs using single-trajectory data—triplets of prompt, response, and scalar reward (e.g., a thumbs-up/down)—rather than the pairwise preference data required by methods like DPO. The authors derive a theoretically principled mean-squared objective that jointly learns a policy and a value function, producing a practical instantiation called DRO-V, and evaluate it on the UltraFeedback dataset using T5 encoder-decoder models up to 3B parameters. DRO-V significantly outperforms Kahneman-Tversky Optimization (KTO), the primary existing single-trajectory baseline, achieving 63.4% and 57.5% win rates in direct side-by-side comparisons on T5-L and T5-XL models respectively, establishing that single-trajectory alignment is effective when the value function is learned jointly with the policy—but only when parameter sharing is avoided and per-token value predictions are used rather than a single scalar per batch.
2. Context and Motivation
The Core Problem: Pairwise Preference Data Is Expensive and Unnatural
The dominant paradigm for aligning large language models with human values—whether through RLHF (Christiano et al., 2017) or direct preference optimisation (DPO; Rafailov et al., 2023)—relies on pairwise preference data. Each training example is a quadruplet: a prompt, two independent responses, and a human judgment indicating which response is preferred. This data is typically collected by hiring trained raters to compare pairs of model outputs and select the better one.
The paper identifies two fundamental problems with this approach. First, it is expensive and unnatural (Section 1). Asking a rater to compare two completions and articulate a preference is cognitively demanding and requires paid annotators. In contrast, single-trajectory feedback—a user providing a thumbs-up or thumbs-down on a single model response—is far more natural and abundant. It is the kind of feedback that users already generate when interacting with deployed systems: upvoting a helpful answer, downvoting a hallucination, or clicking "regenerate" on an unsatisfactory completion. The paper argues that single-trajectory data exists "in the wild" at potentially orders of magnitude greater scale than curated pairwise preferences:
"annotating pairwise data is more expensive and less natural than simply indicating whether a single completion is satisfactory or not, e.g., by assigning a binary thumbs up or down rating to the model completion. The former is generally carried out by paid raters whereas the latter could be produced by users at a much larger scale. Consequently, single-trajectory data is much more abundant in the wild, hence, cheaper and more easily collected than scarce preference data."
This is not merely a cost argument. The paper draws a parallel to the history of deep learning, where reducing the cost of supervision has repeatedly unlocked step-changes in capability. Self-supervised learning in NLP (Devlin et al., 2019) and computer vision (Chen et al., 2020; Grill et al., 2020) succeeded precisely because it eliminated the bottleneck of expensive human labels, enabling training on internet-scale data. The authors frame single-trajectory alignment as an analogous shift: if algorithms can leverage naturally occurring single-trajectory feedback instead of requiring carefully curated pairwise comparisons, the scale of available training data could increase dramatically, with corresponding improvements in alignment quality.
Second, the increasing capability of LLMs makes pairwise annotation self-defeating (Section 1). As models improve, distinguishing between two strong responses becomes increasingly difficult for human raters. The authors cite Saunders et al. (2022) and Bowman et al. (2022) on this point:
"as LLMs improve in quality, the task of distinguishing between a pair of strong responses gets increasingly difficult, and would require additional efforts in improving both the quality and scale of the collected human feedback."
In other words, the better the model, the harder it is to collect meaningful pairwise preference data—a cruel feedback loop. Single-trajectory feedback (is this response good or bad?) remains tractable even when both possible responses are high-quality, because the rater only needs to evaluate one output against an absolute standard rather than make a fine-grained relative judgment.
Why This Problem Matters: Practical and Theoretical Significance
The practical significance is immediate. If alignment can be performed effectively on single-trajectory data, any deployed LLM system that logs user feedback (thumbs-up/down, regenerate requests, satisfaction ratings) becomes a source of alignment training data. This would dramatically lower the barrier to continuous improvement of deployed models and enable personalization at scale. Instead of running expensive, periodic data collection campaigns with professional raters, models could be fine-tuned continuously on organic user feedback.
The theoretical significance is equally important. The transition from pairwise to single-trajectory data is not a simple extension of existing algorithms. In pairwise methods like DPO, the partition function (a normalizing constant in the optimal policy expression that depends on the prompt) conveniently cancels out because the algorithm operates on the difference of rewards between two completions of the same prompt. In the single-trajectory setting, this cancellation does not occur. The partition function—or equivalently, a value function —remains in the objective and must be explicitly modeled. This means that designing a principled single-trajectory algorithm requires grappling with the estimation of this value function, a challenge that the dominant pairwise methods never had to address.
The paper also emphasizes that this shift connects alignment to the broader field of offline reinforcement learning (Levine et al., 2020). Single-trajectory data is exactly the form of data assumed in offline RL, where an agent learns from logged transitions generated by some unknown behavior policy. Despite this natural connection, the paper notes that "few offline alignment methods consider this setting" (Section 2).
Where Existing Approaches Fall Short
The RLHF pipeline is inherently pairwise and multi-stage. Standard RLHF (Section 2) involves three steps: (1) collect pairwise preference data, (2) train a reward model under the Bradley-Terry model of preferences, and (3) use reinforcement learning to optimize the LLM against this learned reward model, typically with a KL-divergence penalty to prevent the policy from drifting too far from a reference. This pipeline is computationally expensive (it requires online sampling from the LLM during training), architecturally complex (it requires training and maintaining a separate reward model), and fundamentally dependent on pairwise data. None of these components are compatible with single-trajectory feedback out of the box.
Direct preference optimisation (DPO) eliminates the reward model but not the pairwise data requirement. DPO (Rafailov et al., 2023) reformulates RLHF as a supervised learning objective that directly optimizes the policy from pairwise preferences, bypassing the need for an explicit reward model and online sampling. This was a significant simplification, and follow-up work has generalized the approach (Azar et al., 2023; Zhao et al., 2023a; Tang et al., 2024). However, all of these methods share the same fundamental limitation: they require pairwise data of the form where is preferred over . As the paper states, all such objectives can be subsumed under a unified loss (Section 2):
where different choices of produce DPO (), SLiC (), IPO (), and so on. The critical observation is that the inner term is a difference of log-ratios evaluated on two completions of the same prompt. This is what causes the partition function to cancel—and it is also what makes these methods fundamentally incompatible with single-trajectory data, where only one completion per prompt is available.
KTO is the only existing single-trajectory method, but it has significant limitations. The paper identifies Kahneman-Tversky Optimization (KTO; Ethayarajh et al., 2024) as essentially the sole existing algorithm designed for single-trajectory alignment. KTO is derived from prospect theory (Kahneman and Tversky, 1979), which models human decision-making under risk and uncertainty. The paper argues that KTO has two key weaknesses:
- Strong simplifying assumptions that bias the resulting policy. In particular, KTO assumes that the partition function (the normalizing constant in the optimal policy expression) is constant across all prompts in a batch. The paper explicitly contrasts this with its own approach:
"Unlike KTO, which assumes a constant partition function for each prompt of the batch, we do not make any assumptions on the form of or ."
This assumption is consequential: in reality, the partition function depends on the prompt —some prompts admit a wide range of good responses (large partition function), while others have only a narrow set of acceptable answers (small partition function). Assuming constancy across prompts introduces bias that can push the learned policy away from the true optimum.
- Reliance on prospect-theoretic principles rather than a direct derivation from the RLHF objective. KTO builds on notions of human risk aversion from prospect theory, which—while potentially descriptive of human behavior—does not naturally connect to the standard KL-regularized policy optimization framework that underpins most RLHF work. The paper positions DRO as a more "simple, general purpose and performant algorithm" (Section 2) that flows directly from the first principles of KL-regularized policy optimization, without "strong dependency on mathematical assumptions on risk preference or utility."
Online RL approaches are possible but introduce their own challenges. The paper briefly considers whether one could use online RL methods (Calandriello et al., 2024; Guo et al., 2024) with single-trajectory data. There are two options: (1) generate new completions online using the current policy, which is computationally expensive and technically challenging, or (2) use importance sampling to correct for the distribution mismatch between the offline data and the online policy, which introduces high variance. The paper chooses the offline setting to "circumvent the associated difficulties" while also noting that "offline RL brings orthogonal benefits of its own, such as simplicity and computational efficiency" (Section 2).
How This Paper Positions Itself Relative to Existing Work
The paper positions DRO as a direct, first-principles derivation of a single-trajectory alignment algorithm from the standard KL-regularized RLHF objective (Equation 1):
From this objective, the optimal policy has a known closed form (Equation 2):
where is the soft value function (or log partition function). The key insight is that rearranging this optimality condition yields an equation that must hold at the optimum (Equation 3):
This equation directly motivates a mean-squared error objective (Equation 4):
The paper's positioning is thus clear: rather than deriving an algorithm from prospect theory (KTO) or from pairwise preference modeling (DPO and its variants), DRO comes directly from the Bellman-like optimality condition of the KL-regularized policy optimization problem. This gives it three claimed advantages:
-
No assumptions on the partition function. Unlike pairwise methods (which rely on cancellation of ), DRO explicitly learns it. Unlike KTO (which assumes it is constant), DRO allows it to vary per prompt.
-
Joint learning of policy and value. The framework naturally produces an algorithm that learns both and , rather than optimizing the policy alone. This is significant because Proposition 1 (Section 3.2) shows that the quality of the learned policy depends on the accuracy of the value function approximation—a poorly learned biases the policy gradient.
-
Generality. The paper emphasizes that DRO is a framework that can be instantiated in multiple ways—DRO-V (with a learned value function), DRO-P (a policy-only variant in Appendix B)—and that these different instantiations correspond to different tradeoffs between computational cost and theoretical soundness.
The paper's core argument is thus that single-trajectory alignment is both practically important (because single-trajectory data is abundant) and theoretically non-trivial (because it requires explicitly modeling the value function that pairwise methods cancel out), and that DRO provides a simple, principled solution to this problem that outperforms the only existing competitor (KTO) by a significant margin.
3. Technical Approach
3.1 Reader Orientation
DRO is a training framework for aligning language models that learns from simple feedback signals—a prompt, a model response, and a scalar reward (like a thumbs-up or a numerical score)—rather than requiring the carefully constructed pairwise comparisons used in prior methods like DPO. The system solves the problem of expensive data collection by turning the standard RLHF optimization objective into a simple mean-squared regression problem that jointly learns a policy (the language model itself) and a value function (which estimates how good a prompt is in aggregate), enabling alignment on the kind of feedback that users naturally generate when interacting with deployed systems.
3.2 Big-Picture Architecture (Diagram in Words)
The DRO-V system has four major components:
-
Base Policy (
$\pi_{\text{ref}}$) — the pretrained and instruction-fine-tuned language model that serves as the starting point and regularization anchor. It is never updated directly; instead, it provides the reference distribution that the learned policy should not stray too far from. -
Learned Policy (
$\pi_\theta$) — a separate neural network (in practice, a second copy of the base model architecture) that is fine-tuned to produce responses that maximize reward while staying close to the reference policy. This is the output of the training process. -
Value Function (
$V_\varphi$) — a separate neural network that takes only a prompt$x$as input and outputs a single scalar estimating the expected reward-minus-regularization achievable from that prompt. This value function explicitly models the partition function that pairwise methods like DPO cancel out. -
DRO Loss — a mean-squared error objective that measures the discrepancy between the actual reward and the predicted reward computed from the policy log-ratio and value function. Minimizing this loss drives both the policy and value function toward the theoretically optimal solution.
Information flows as follows: a batch of (prompt, completion, reward) triplets enters the system → the value network processes each prompt and outputs $V_\varphi(x)$ → the policy network computes $\tau \log(\pi_\theta(y|x) / \pi_{\text{ref}}(y|x))$ for each completion → the loss $\frac{1}{2}(r - V_\varphi(x) - \tau \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)})^2$ is computed → gradients are back-propagated through both networks simultaneously, with the policy gradient rescaled by $1/\tau$ for numerical stability → parameters $\theta$ and $\varphi$ are updated → the process repeats for a fixed number of steps using the static offline dataset.
3.3 Roadmap for the Deep Dive
- First, the mathematical foundation: the KL-regularized policy optimization objective (Equation 1) and its closed-form optimal solution (Equation 2), which together define what "optimal" means and why the value function
$V^*$is a necessary object. - Second, the rearrangement of the optimality condition into Equation (3), which is the critical algebraic step that converts a theoretically optimal but practically intractable expression into a loss function that can be minimized with gradient descent.
- Third, the DRO loss function itself (Equation 4), including Theorem 1's guarantee that the true optimal policy and value function are the unique global minimum, and why this result justifies independently optimizing
$\pi$and$V$without worrying about their coupling. - Fourth, the approximation error analysis in Proposition 1, which explains what happens when the value function is imperfectly learned—a crucial practical concern since
$V$will always be approximate in real training. - Fifth, the gradient computation and practical implementation details in Algorithm 1, including the specific form of the policy and value gradients (Equations 7 and 8), the
$1/\tau$policy learning rate rescaling, and the design choices around separate networks and per-token value predictions. - Sixth, the alternative DRO-P formulation (Appendix B), which eliminates the value network by using batch-level variance estimation, and why this simpler variant underperforms DRO-V empirically.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a theoretical contribution paper whose core idea is that the KL-regularized RLHF optimality condition can be rearranged into a simple mean-squared error objective that jointly optimizes a policy and a value function on single-trajectory data, eliminating the need for pairwise preferences while remaining mathematically grounded in the same optimization framework that underpins DPO and IPO.
The KL-Regularized Policy Optimization Problem
The starting point is the canonical formulation of RLHF as a KL-regularized policy optimization problem. The goal is to find a policy $\pi^*$ that maximizes expected reward while penalizing deviation from a reference policy $\pi_{\text{ref}}$:
where $\rho$ is the distribution over prompts, $r(x,y)$ is the scalar reward for response $y$ given prompt $x$, $\tau > 0$ is the regularization coefficient controlling how strongly the policy is penalized for deviating from $\pi_{\text{ref}}$, and $\text{KL}(\pi || \pi_{\text{ref}})$ is the Kullback-Leibler divergence between the learned policy and the reference policy.
What it computes: this equation defines the objective that any alignment algorithm is trying to solve. For each prompt $x$, the policy $\pi$ samples a response $y$, receives a reward $r(x,y)$, and pays a penalty proportional to how much $\pi(y|x)$ differs from $\pi_{\text{ref}}(y|x)$. The optimization searches over all possible policies (all ways of mapping prompts to response distributions) for the one that maximizes this reward-minus-penalty quantity in expectation.
Why this form: the KL penalty is the standard choice in RLHF (Christiano et al., 2017; Ziegler et al., 2020) because it prevents the policy from collapsing to a degenerate distribution that only produces the single highest-reward response (which would be brittle and likely exploit reward model errors), while still allowing the policy to substantially improve over the reference when the reward signal is reliable. The coefficient $\tau$ controls this tradeoff: $\tau \to 0$ removes regularization entirely (policy can diverge arbitrarily), while $\tau \to \infty$ forces the policy to remain identical to $\pi_{\text{ref}}$ (no learning occurs). In practice, $\tau$ is treated as a hyperparameter tuned for each dataset and model.
The Closed-Form Optimal Policy
A critical property of the KL-regularized objective is that it has a known closed-form solution. The optimal policy $\pi^*$ can be written as:
where $V^*(x)$ is defined as:
What it computes: the optimal policy is the reference policy reweighted by an exponentiated reward term. Responses with higher rewards get higher probability, but the reweighting is modulated by $\tau$: when $\tau$ is small (weak regularization), the exponentiated reward dominates and the policy concentrates sharply on high-reward responses; when $\tau$ is large (strong regularization), the exponentiated reward has little effect and the policy stays close to $\pi_{\text{ref}}$. The denominator $e^{\frac{1}{\tau}V^*(x)}$ is a normalizing constant—also called the partition function—that ensures $\pi^*(\cdot|x)$ sums to 1 over all possible responses $y$. $V^*(x)$ itself is the "soft value function" or "log-sum-exp of rewards": it is large when the reference policy can produce high-reward responses (the prompt is "easy") and small when all responses under $\pi_{\text{ref}}$ have low reward (the prompt is "hard").
Why this form: this is the standard exponential-family solution to entropy-regularized optimization problems (Ziebart et al., 2008; Haarnoja et al., 2017; Schulman et al., 2018). The exponentiated reward $e^{r(x,y)/\tau}$ acts as an importance weight that multiplies the reference probability, and the denominator normalizes the result into a valid distribution. Unlike the Bradley-Terry model used in pairwise RLHF (where the partition function cancels when comparing two responses to the same prompt), the single-trajectory setting requires explicit modeling of $V^*(x)$ because there is no second response to difference against.
Rearranging Into a Regression Target
The key algebraic insight is that the optimality condition can be rearranged to express the reward $r(x,y)$ in terms of the optimal policy and value function. Starting from the closed-form expression for $\pi^*$, multiply both sides by the denominator, take the logarithm, and rearrange:
This equation must hold for the optimal $\pi^*$ and $V^*$ at every pair $(x,y)$.
What it computes: this equation states that the reward $r(x,y)$ decomposes into two terms: $V^*(x)$, which depends only on the prompt (it is the "baseline" value of receiving prompt $x$), and $\tau \log(\pi^*(y|x) / \pi_{\text{ref}}(y|x))$, which depends on the response and measures how much the optimal policy upweights (positive log-ratio) or downweights (negative log-ratio) response $y$ relative to the reference. If the optimal policy assigns higher probability to a response than the reference does, that response must have above-average reward (relative to $V^*(x)$); if it assigns lower probability, the reward must be below average.
Why this form: this rearrangement is the bridge between the theoretical optimum and a practical loss function. It transforms the condition "$\pi$ and $V$ are optimal" into an equation that must be satisfied at the optimum. This equation involves only quantities that are either observed ($r$), computable from the reference policy ($\pi_{\text{ref}}$), or parameterized by the learning algorithm ($\pi_\theta$, $V_\varphi$). It provides a direct regression target: when $\pi$ and $V$ are not optimal, the left and right sides of Equation (3) will differ, and minimizing the squared difference pushes them toward the optimum. Crucially, unlike the pairwise setting where subtracting this equation for two responses cancels $V^*(x)$, the single-trajectory setting retains $V^*(x)$ as an explicit quantity that must be learned.
The DRO Loss Function
From Equation (3), the paper defines the DRO loss as the mean squared error between the left and right sides, evaluated over the data distribution $\rho(x)\mu(y|x)$ where $\mu$ is the behavior policy that generated the offline data:
What it computes: for each prompt-completion pair $(x,y)$ in the offline dataset, the loss computes the squared difference between the observed reward $r(x,y)$ and the predicted reward $V(x) + \tau \log(\pi(y|x)/\pi_{\text{ref}}(y|x))$. The predicted reward is the sum of the value function (prompt-level baseline) and the policy improvement term (response-level adjustment). Minimizing this loss forces $\pi$ and $V$ to satisfy the optimality condition (Equation 3) in expectation under the data distribution. The factor $1/2$ is a conventional scaling for squared-error losses that simplifies gradient expressions.
Why this form: this is the most natural loss function for the regression problem defined by Equation (3). Mean squared error is the maximum-likelihood objective under a Gaussian error model, it is everywhere differentiable, and it has the property that its unique global minimum (when the model can exactly satisfy Equation 3) is zero. Alternative loss functions—such as absolute error or Huber loss—would also drive the residual to zero but would produce different gradient dynamics. The quadratic form is chosen for simplicity and because Theorem 1 establishes that the global minimum exactly recovers the optimal policy and value function, making it a statistically consistent objective.
Theorem 1: Existence and Uniqueness of the Global Optimum
Theorem 1 establishes two properties of the DRO loss:
"
$(\pi^*, V^*)$is a global optimum of the loss$\mathcal{L}(\pi, V)$. In addition, assuming the supports of$\mu$and$\pi_{\text{ref}}$coincide, then$(\pi^*, V^*)$is the unique global optimum of the loss$\mathcal{L}(\pi, V)$."
Proof sketch. The first part is immediate: plugging $\pi^*$ and $V^*$ into the loss gives zero because they satisfy Equation (3) by definition, and the loss is a sum of non-negative squared terms, so zero is the minimum possible value. For uniqueness: assume another pair $(\tilde{\pi}, \tilde{V})$ achieves zero loss. Then for all $(x,y)$ in the support of the data distribution, the residual must be zero, meaning $r(x,y) - \tilde{V}(x) = \tau \log(\tilde{\pi}(y|x) / \pi_{\text{ref}}(y|x))$. Solving for $\tilde{\pi}$ yields $\tilde{\pi}(y|x) \propto \pi_{\text{ref}}(y|x) e^{r(x,y)/\tau}$, and since $\tilde{\pi}$ must sum to 1, the normalizing constant must be $e^{\tilde{V}(x)/\tau}$ with $\tilde{V}(x) = \tau \log \mathbb{E}_{y\sim\pi_{\text{ref}}}[e^{r(x,y)/\tau}] = V^*(x)$. The support condition (that $\mu$ has the same support as $\pi_{\text{ref}}$) ensures that the zero-loss condition is enforced everywhere that $\pi_{\text{ref}}$ assigns probability, which is necessary to pin down $\tilde{\pi}$ uniquely.
What this means in practice: Theorem 1 guarantees that gradient descent on $\mathcal{L}_{\text{DRO}}$ has no spurious local minima and no "good but suboptimal" solutions—the only way to achieve zero loss is to recover the exact optimal policy and value function. More importantly for algorithm design, the proof shows that $\pi$ and $V$ can be optimized independently. The condition $\pi = \pi^*$ and $V = V^*$ is simultaneously the unique minimizer of the joint loss, but the optimization does not require enforcing any explicit coupling between them (such as the consistency condition $V(x) = \tau \log \mathbb{E}_{\pi_{\text{ref}}}[e^{(r - \tau\log(\pi/\pi_{\text{ref}}))/\tau}]$). This independence simplifies implementation: the policy and value networks can be separate neural networks updated with their own gradient computations, without needing to enforce that $V$ is the log-normalizer of $\pi$. The paper notes:
"The significance of Theorem 1 lies in the fact that, though
$\pi^*$and$V^*$are clearly related via$\log\pi^*(y|x) + V^*(x) = \log\pi_{\text{ref}}(y|x) + \frac{1}{\tau}r(x,y)$, there is no need to account for that connection during optimisation. We can optimize$\pi$and$V$independently and end up finding the optimum."
The Value Function as an Explicitly Learned Object
A central design choice in DRO-V—and a key differentiator from KTO—is the decision to parameterize and learn $V$ explicitly. The paper explains why this is necessary rather than optional.
Why the value function cannot be omitted. In the single-trajectory setting, there is no second response to difference against, so the partition function does not cancel. If one attempted to optimize a policy-only loss by dropping $V$ entirely (setting $V(x) \equiv 0$ or to some constant), the objective would become $\mathbb{E}[(r(x,y) - \tau \log(\pi/\pi_{\text{ref}}))^2]$. This is biased because the optimal policy satisfies $\tau \log(\pi^*/\pi_{\text{ref}}) = r - V^*$, not $\tau \log(\pi^*/\pi_{\text{ref}}) = r$. The missing $V^*$ term would push the learned policy toward incorrectly matching raw rewards rather than reward differences from the prompt baseline. The value function is not merely a variance-reduction baseline (as it would be in on-policy policy gradient methods); it is a structural component of the optimality condition.
What $V(x)$ captures. The value function $V^*(x) = \tau \log \mathbb{E}_{\pi_{\text{ref}}}[e^{r(x,y)/\tau}]$ is the soft-maximum (log-sum-exp) of rewards achievable from prompt $x$ under the reference policy. It is large when there exist responses that $\pi_{\text{ref}}$ assigns reasonable probability to and that also receive high rewards. It is small when all high-probability responses under $\pi_{\text{ref}}$ receive low rewards. In the optimal policy expression $\pi^*(y|x) = \pi_{\text{ref}}(y|x) e^{(r(x,y) - V^*(x))/\tau}$, the term $r(x,y) - V^*(x)$ is the "advantage" of response $y$ relative to the soft-optimal value of the prompt. Responses with positive advantage are upweighted; responses with negative advantage are downweighted. This interpretation clarifies why $V^*$ matters: it provides the zero point for the advantage computation, and getting that zero point wrong biases the entire policy.
Contrast with KTO's assumption. KTO assumes that the partition function is constant across all prompts in a batch. This is equivalent to setting $V^*(x) \equiv c$ where $c$ is the same for every $x$. The paper argues this is a strong and potentially harmful simplification, since in reality different prompts have very different value functions—a prompt asking for a simple factual answer has a narrow range of acceptable responses (low $V^*$), while an open-ended creative writing prompt has a wide range (high $V^*$). By learning $V_\varphi(x)$ as a function of $x$, DRO-V captures this prompt-dependent variation and avoids the bias introduced by the constant-$Z$ assumption.
Approximation Error Analysis (Proposition 1)
In practice, the learned value function $V_\varphi$ will not perfectly match the true $V_{\pi_\theta}$ (the value function that would be optimal for the current policy). Proposition 1 quantifies how this approximation error propagates to the learned policy.
The setup: consider fixing $V$ to some approximation (not necessarily optimal for the current $\pi$) and then optimizing $\pi \mapsto \mathcal{L}(\pi, V)$ with respect to the policy only. Let $\pi_V = \arg\min_\pi \mathcal{L}(\pi, V)$ be the resulting policy, and let $V_{\pi_V}(x) = \mathbb{E}_{y\sim\mu(\cdot|x)}[r(x,y) - \tau \log(\pi_V(y|x)/\pi_{\text{ref}}(y|x))]$ be the value function that would be optimal for $\pi_V$ (the "true" value function for that policy). Proposition 1 states:
and the error bound:
What the first equation means. When $V$ is not the true value function for $\pi_V$, the optimal policy under that fixed $V$ is not $\pi^*$ but a distorted version. The distortion term $(\pi_V/\mu)(V_{\pi_V} - V)$ effectively adds (or subtracts) a correction to the reward based on how wrong $V$ is, weighted by the ratio of the learned policy probability to the data-generating policy probability. When $V(x)$ overestimates $V_{\pi_V}(x)$ (thinks the prompt is better than it is), the effective reward for all responses is reduced, particularly for responses where $\pi_V(y|x) / \mu(y|x)$ is large. When $V(x)$ underestimates, the effective reward is increased.
What the second equation means. The log-ratio between the learned policy $\pi_V$ and the true optimal policy $\pi^*$ is bounded by the product of two terms: the value function error $|V_{\pi_V} - V|$ and the distribution mismatch term $|1 - \pi_V/\mu|$. The bound says that $\pi_V$ is close to $\pi^*$ if either (a) the value function is accurate ($V \approx V_{\pi_V}$) or (b) the learned policy is close to the data-generating policy ($\pi_V \approx \mu$, making the mismatch term near zero). Case (b) is intuitive: if the policy hasn't moved far from the behavior policy, then even a poor value function doesn't hurt much because the correction term $(\pi_V/\mu)(V_{\pi_V} - V) \approx 1 \cdot (V_{\pi_V} - V)$ is approximately constant across responses and gets absorbed into the normalizing constant.
Special case: state-independent baseline. The paper notes that if one uses a single scalar $V_c$ for all prompts (estimated as the batch average of $V_\pi(x)$), then $\pi_V = \pi^*$ when $V_{\pi^*}(x)$ happens to be constant across prompts. This provides a theoretical condition under which KTO's constant-partition-function assumption would be correct: when the soft value is genuinely prompt-independent. However, this is unlikely to hold in practice, which explains why DRO-V's per-prompt value function outperforms methods that assume constancy.
Why this analysis matters. Proposition 1 is the theoretical justification for why DRO-V learns a value function at all. It shows that the policy gradient (Equation 8), which uses $V_\varphi(x)$ as a subtracted baseline, is not merely a variance-reduction trick—it is structurally necessary to avoid bias when the data is off-policy ($\mu \neq \pi_\theta$). As the paper states:
"removing this term (or replacing it by any function
$V(x_i)$) would bias our policy gradient estimate. The reason for that being that the samples$y_i$are off-policy, i.e., drawn from$\mu$and not from$\pi_\theta$, thus in general,$\mathbb{E}_{y\sim\mu(\cdot|x_i)}[\nabla_\theta \log \pi_\theta(y|x_i) V(x_i)] \neq 0$."
In on-policy RL, the baseline $V(x)$ has expectation zero under the policy gradient because $\mathbb{E}_{\pi_\theta}[\nabla_\theta \log \pi_\theta] = 0$. But in the offline setting, the data comes from $\mu$, not $\pi_\theta$, so $\mathbb{E}_\mu[\nabla_\theta \log \pi_\theta] \neq 0$ in general, and subtracting an incorrect baseline introduces bias. The correctly learned $V_\varphi$ cancels this bias because the DRO loss explicitly trains it to satisfy $V(x) \approx \mathbb{E}_\mu[r - \tau \log(\pi_\theta/\pi_{\text{ref}})]$.
Practical Implementation: The Empirical Loss and Gradients
DRO-V is implemented by minimizing the empirical version of the DRO loss over a dataset of $n$ triplets $(x_i, y_i, r_i)$:
What it computes: this is the sample-average version of Equation (4), where the expectation over $x\sim\rho, y\sim\mu(\cdot|x)$ is replaced by the empirical average over the offline dataset. Each term in the sum is the squared residual for one training example.
Value function gradient (Equation 7):
What it computes: this is the standard gradient of a squared-error loss with respect to the value function parameters. The term in parentheses is the negative of the residual: $V_\varphi(x_i)$ (predicted value) minus $r(x_i, y_i) - \tau \log(\pi_\theta/\pi_{\text{ref}})$ (the target, which is the reward adjusted by the policy's current log-ratio). When $V_\varphi(x_i)$ is too high relative to the target, the residual is positive and the gradient pushes $V_\varphi$ downward; when too low, the gradient pushes upward. This gradient is multiplied by $\nabla_\varphi V_\varphi(x_i)$, the Jacobian of the value network output with respect to its parameters, which determines how each parameter should change to adjust the prediction.
Policy gradient (Equation 8):
What it computes: the policy gradient decomposes into two terms. The first term, $\nabla_\theta \log \pi_\theta(y_i|x_i) (r(x_i, y_i) - V_\varphi(x_i))$, is a policy gradient that increases the log-probability of response $y_i$ when its reward exceeds the value baseline $V_\varphi(x_i)$ (positive advantage) and decreases it when the reward falls below the baseline (negative advantage). This is structurally identical to the REINFORCE policy gradient, but with $V_\varphi(x_i)$ serving as the baseline. The second term, $-\frac{\tau}{2} \nabla_\theta (\log(\pi_\theta/\pi_{\text{ref}}))^2$, is an $\ell_2$ regularization that penalizes large deviations of $\log \pi_\theta$ from $\log \pi_{\text{ref}}$. The gradient of this term pushes $\pi_\theta$ toward $\pi_{\text{ref}}$ with strength proportional to $\tau$ times the current log-ratio.
Why this form—the $\ell_2$ regularization is not equivalent to a KL penalty in the off-policy setting. This is a subtle but important point. In on-policy RL, the gradient of the KL divergence $\text{KL}(\pi_\theta || \pi_{\text{ref}})$ with respect to $\theta$ is $\mathbb{E}_{\pi_\theta}[\nabla_\theta \log \pi_\theta \cdot \log(\pi_\theta/\pi_{\text{ref}})]$. In the off-policy setting, the gradient of the squared log-ratio under the data distribution $\mu$ is $\mathbb{E}_\mu[\nabla_\theta (\log(\pi_\theta/\pi_{\text{ref}}))^2] = 2 \mathbb{E}_\mu[\nabla_\theta \log \pi_\theta \cdot \log(\pi_\theta/\pi_{\text{ref}})]$. These two are generally different because the expectations are taken under different distributions ($\pi_\theta$ vs. $\mu$). The paper notes:
"these two regularisation losses do not lead to equivalent gradients for the reason that the samples
$y_i$are off-policy: indeed we have$\frac{1}{2}\mathbb{E}_{y\sim\mu(\cdot|x_i)}[\nabla_\theta (\log \frac{\pi_\theta(y|x_i)}{\pi_{\text{ref}}(y|x_i)})^2] \neq \nabla_\theta \text{KL}(\pi_\theta(\cdot|x_i), \pi_{\text{ref}}(\cdot|x_i))$unless the sampling is on-policy$\mu = \pi_\theta$."
The $\ell_2$ regularization on log-ratios is therefore the "natural extension of a usual on-policy regularised PG algorithm to the off-policy case" (Section 3.3). Using a standard KL penalty in the loss would produce biased gradients when the data comes from $\mu$.
Policy learning rate rescaling by $1/\tau$. In practice, the paper rescales the policy gradient by multiplying it by $1/\tau$. This means the effective policy update is:
without the leading factor of $\tau$. The paper states this "works better empirically" and hypothesizes that the global loss may be ill-conditioned because "its sensitivities w.r.t. the dimensions $\theta$ and $\varphi$ are different, leading to a high condition number." The rescaling makes the policy and value updates have comparable magnitudes, improving optimization stability.
Algorithm 1: DRO-V Training Loop
Algorithm 1 presents the full training procedure:
-
Inputs: a single-trajectory dataset
$\{(x_i, y_i, r_i)\}_{i=1}^N$, parameterized policy$\pi_\theta$and value function$V_\varphi$, reference policy$\pi_{\text{ref}}$, regularization scalar$\tau$, number of steps$K$, batch size$B$, and an optimizer. -
For each step
$k = 1, \dots, K$:- Sample a batch of
$B$triplets uniformly from the dataset. - Compute
$\nabla_\theta \hat{\mathcal{L}}$using Equation (8) and$\nabla_\varphi \hat{\mathcal{L}}$using Equation (7). - Update
$\theta$using the optimizer and$\nabla_\theta \hat{\mathcal{L}}$. - Update
$\varphi$using the optimizer and$\nabla_\varphi \hat{\mathcal{L}}$.
- Sample a batch of
-
Output: the trained policy
$\pi_\theta$.
Key operational detail: the dataset is static. The paper emphasizes that DRO-V is an offline algorithm: "The dataset of (prompt, completion, reward) triplets remains static during optimisation. This is because we do not use the current 'online' policy, parameterised by the most recent $\theta$ parameters, to regenerate completions for a given prompt." This distinguishes DRO-V from online RLHF methods like PPO, which require sampling new responses from the current policy at each training step. The offline nature simplifies implementation (no generation during training) and avoids the computational cost and engineering complexity of online sampling, but it also means that DRO-V cannot correct for distribution shift—if the learned policy $\pi_\theta$ diverges significantly from the behavior policy $\mu$ that generated the data, the gradients become increasingly biased because they are computed under $\mu$ rather than $\pi_\theta$. The $\ell_2$ regularization mitigates this by keeping $\pi_\theta$ close to $\pi_{\text{ref}}$, which is typically close to $\mu$.
Neural Network Implementation: The Criticality of Separate Networks
The paper investigates several architectural choices for implementing $\pi_\theta$ and $V_\varphi$ and finds that seemingly minor decisions have substantial impact on performance.
Two networks, not one. The natural question is whether $\theta$ and $\varphi$ can share parameters—for example, by using the same transformer backbone with two output heads (one for next-token prediction, one for value prediction). The paper finds the opposite: "using two separate networks, one for $\pi_\theta$ and one for $V_\varphi$, was beneficial empirically, compared to using policy logits as value outputs." Parameter sharing degrades performance. This is likely because the policy and value function have different learning dynamics: the policy learns to adjust token-level probabilities to match rewards, while the value function learns a prompt-level scalar that summarizes expected reward. Forcing them to share a representation creates a conflict where improvements for one task may harm the other. The full parameter sharing ablation (Appendix C, Figure 6) shows that using a single network for both $\pi$ and $V$ reduces winrate vs. SFT from 76.6% to as low as 55.5% (when also using a single scalar value per batch).
Multiple value predictions per batch, not one. The value function $V_\varphi$ is implemented to produce separate scalar outputs for each prompt in the batch, rather than a single scalar shared across all prompts. The paper states: "we found that using a single value per batch hurts performance." The ablation in Figure 5 confirms this: with two separate networks, using a single value per batch achieves 72.1% winrate vs. SFT compared to 76.6% with multiple values—a 4.5 percentage point gap. The multi-value version also wins head-to-head with 54.9% against the single-value version (Figure 6). This result is consistent with Proposition 1: if $V$ is forced to be constant across prompts, it cannot capture the prompt-dependent variation in the partition function, introducing bias into the policy gradient. The multiple-value implementation allows $V_\varphi(x_i)$ to vary with $x_i$, capturing the fact that different prompts have different baseline values.
Default configuration. The paper uses two separate T5 encoder-decoder networks for $\pi_\theta$ and $V_\varphi$, with $V_\varphi$ producing per-prompt (not per-batch) scalar outputs. The reference policy $\pi_{\text{ref}}$ is the initial instruction-tuned checkpoint (FLAN recipe; Chung et al., 2022), frozen throughout training.
The DRO-P Alternative: Policy-Only Variant
Appendix B explores whether the value function can be eliminated by analytically optimizing $V$ out of the loss. For a fixed policy $\pi$, the optimal value function is $V_\pi(x) = \mathbb{E}_{y\sim\mu}[r(x,y) - \tau \log(\pi(y|x)/\pi_{\text{ref}}(y|x))]$. Substituting this back into the loss yields:
What it computes: this is the variance (under the behavior policy $\mu$) of the residual $r - \tau \log(\pi/\pi_{\text{ref}})$ across different completions of the same prompt. If the policy and value function are optimal, this residual is exactly $V^*(x)$ for all $y$, so the variance is zero. Minimizing this variance pushes the policy toward optimality without requiring an explicit value network.
Why it is impractical in the pure single-trajectory setting. Computing the variance $\text{Var}_{y\sim\mu(\cdot|x)}$ requires multiple completions per prompt. In the strict single-trajectory setting, there is only one $(x, y)$ pair per datapoint, so the per-prompt variance cannot be estimated. To make this practical, DRO-P approximates the variance by computing it across the batch instead: $\text{Var}_{y\sim\mathcal{B}}$ replaces $\text{Var}_{y\sim\mu(\cdot|x)}$, where the batch contains examples with different prompts. This is a heuristic approximation because it conflates variance across prompts (which is legitimate, since prompts have different values of $r - \tau \log(\pi/\pi_{\text{ref}})$) with variance across completions of the same prompt (which is the quantity the theory requires to be zero at optimality).
Empirical performance. DRO-P "does not collapse and produces non-trivial resulting policies" that can beat the SFT baseline, but it is "less competitive" than DRO-V. The paper presents this as evidence that the explicit value function in DRO-V is not merely a convenience but a structurally important component: "This speaks to the necessity to use both a policy and a value function—in a theoretically sound and principled way—in order to achieve the best empirical results."
Connection to IPO. The paper notes that when the reward difference $r(x, y_2) - r(x, y_1)$ is interpreted as a preference signal, the DRO-P loss (with two samples per prompt) reduces to the IPO loss (Azar et al., 2023). This establishes DRO as a generalization: DRO-P with two samples per prompt is IPO; DRO-P with batch-level variance approximation is a single-trajectory heuristic; DRO-V with explicit value learning is the fully principled single-trajectory method.
Summary of Design Choices and Their Justifications
- Mean-squared error over preference-based loss: Enables single-trajectory learning by avoiding the need for pairwise comparisons. MSE directly regresses the optimality condition rather than modeling relative preferences.
- Explicit value function
$V_\varphi$: Captures the prompt-dependent partition function that pairwise methods cancel out. Theoretically necessary for unbiased off-policy gradients (Proposition 1). - Separate networks for policy and value: Avoids destructive interference between policy learning (token-level) and value learning (prompt-level). Empirically, parameter sharing degrades performance by ~4–20 percentage points depending on configuration.
- Per-prompt value predictions, not per-batch: Allows
$V(x)$to vary with prompt difficulty. A constant-per-batch value function degrades winrate by 4.5 percentage points. $\ell_2$regularization on log-ratios rather than KL penalty: The$\ell_2$form produces correct gradients in the off-policy setting; a standard KL penalty gradient would be biased because the expectation is under$\mu$not$\pi_\theta$.- Policy gradient rescaling by
$1/\tau$: Improves optimization conditioning by balancing the magnitudes of policy and value updates, which have different natural scales. - Offline, static dataset: Avoids the computational cost and engineering complexity of online sampling. The
$\ell_2$regularization prevents excessive divergence from the data-generating distribution. - Two-fold parameterization of DRO (V and P variants): DRO-V for best performance; DRO-P for computational efficiency when the value network is too expensive, at the cost of reduced alignment quality.
4. Key Insights and Innovations
Innovation 1: Treating Single-Trajectory Alignment as a Regression Problem Rather Than a Preference Modeling Problem
The dominant conceptual shift in this paper is not algorithmic but framing-based: the authors recast alignment from a preference discrimination task into a regression task on the RLHF optimality condition. All prior alignment methods—RLHF with Bradley-Terry models, DPO, IPO, SLiC—operate by learning to discriminate between better and worse responses, whether through an explicit reward model or implicitly through a contrastive loss. This framing is so entrenched that the field's language revolves around "preferences" even when the underlying data is not inherently comparative.
DRO breaks this framing entirely. Instead of asking "which of these two responses is better?", it asks "does this policy-value pair satisfy the equation that must hold at optimality?" The key conceptual move is the rearrangement in Equation (3): r(x,y) - V(x) = τ log(π(y|x) / π_ref(y|x)). This equation is not a modeling assumption—it is an algebraic consequence of the KL-regularized RLHF objective that the entire field already accepts as the target. By converting this condition into a mean-squared error objective (Equation 4), the paper transforms alignment into a supervised regression problem where the target is the observed reward and the predictor is a function of the learned policy and value function.
Why this is intellectually distinctive. The preference modeling paradigm forces a particular data format (pairs of responses with relative judgments) and a particular loss structure (contrastive, typically involving a sigmoid of log-ratio differences). DRO's regression framing decouples the optimization objective from the data format: because the loss only requires a single reward per (prompt, response) pair, it naturally accommodates single-trajectory data without any architectural contortions. This is not an incremental extension of DPO to single-trajectory data—it is a fundamentally different conceptualization of what alignment optimization means. DPO and its variants ask "does the policy correctly predict which response humans prefer?" DRO asks "does the policy satisfy the Bellman equation of KL-regularized RL?"
This reframing has downstream consequences that the paper explicitly develops but that originate from this single conceptual move: the value function V(x) is no longer a nuisance term to cancel out (as in DPO) or a constant to assume away (as in KTO), but a first-class object to be learned. The regression framing makes this natural—V(x) is just one of the predictors in the regression, alongside the policy log-ratio. This is a conceptual rehabilitation of the value function from an algebraic inconvenience to a structurally necessary component, and it is arguably the paper's deepest contribution to how the field should think about alignment.
Comparison to prior work. DPO (Rafailov et al., 2023) explicitly leverages the cancellation of the partition function by working with differences of log-ratios. IPO (Azar et al., 2023) generalizes this within the same contrastive framework. KTO (Ethayarajh et al., 2024) attempts single-trajectory alignment but remains within a preference-adjacent paradigm—it derives from prospect theory, models "desirability" rather than directly optimizing the RLHF objective, and assumes a constant partition function. DRO is the first method to derive a single-trajectory algorithm directly from the optimality condition of the RLHF objective without introducing auxiliary modeling assumptions (prospect theory, Bradley-Terry, constant partition function) or requiring data transformations (pairing, differencing).
The evidence that this reframing matters is in Figures 1 and 2: DRO-V achieves 63.4% win rate over KTO on T5-L and 57.5% on T5-XL, both substantially above 50%. These are not incremental gains—they represent a qualitative improvement in single-trajectory alignment quality achieved purely through a more principled theoretical formulation.
Innovation 2: The Value Function as a Necessary Object, Not a Variance-Reduction Baseline
In standard policy gradient RL, the value function V(x) serves as a baseline—a term subtracted from the reward to reduce the variance of the gradient estimator without introducing bias, because E_π[∇ log π · V(x)] = 0 when the expectation is under the current policy. The value function is optional; omitting it (or using a crude constant) yields unbiased but high-variance gradients.
DRO's second major conceptual contribution is demonstrating that in the offline single-trajectory setting, the value function is not optional—it is structurally necessary to avoid bias. This is a category shift in how the field should think about value functions in alignment. The paper proves this in two ways. First, directly: because the data comes from a behavior policy μ rather than the current policy π_θ, the expectation E_μ[∇ log π_θ · V(x)] is not generally zero, so omitting V or using a wrong V biases the policy gradient, not merely increases its variance. Second, via Proposition 1: the error in the learned policy |log(π_V / π*)| is bounded by a term proportional to the value function approximation error |V_π_V - V|, meaning that an inaccurate value function directly translates to a suboptimal policy, regardless of how much data is available.
Why this is intellectually distinctive. The RLHF literature has largely treated the value function (or partition function) as an algebraic artifact. DPO's central insight was that this artifact cancels out in the pairwise setting, eliminating the need to model it. KTO's approach was to assume it away by treating it as constant across prompts. DRO takes the opposite stance: the value function is real, important, and must be explicitly learned. This is not obvious. The natural instinct when moving from pairwise to single-trajectory data would be to find a clever way to make the partition function cancel (as DPO did) or to assume it away (as KTO did). DRO instead argues that the correct response is to model it properly, and shows both theoretically (Theorem 1, Proposition 1) and empirically (Figure 5) that this choice matters.
The evidence for this claim's practical significance is in the parameter sharing ablation (Figure 5, Appendix C Figure 6). When V is constrained to a single scalar per batch (forcing it to be constant across prompts, akin to KTO's assumption), win rate vs. SFT drops from 76.6% to 72.1%—a 4.5 percentage point degradation. When a single network is used for both policy and value (forcing shared representations), performance collapses to 55.5% win rate vs. SFT. These results demonstrate that the value function is not just a theoretical nicety—getting it wrong measurably harms alignment quality, and the degradation scales with how severely the value function is constrained.
Distinguishing incremental from fundamental. This is a fundamental conceptual shift, not an incremental improvement. Prior work asked "how can we avoid learning the value function?" DRO asks "what happens if we learn it properly?" and shows the answer is "substantially better alignment." This repositions the value function from a problem to be solved to a source of useful signal, which has implications beyond this paper—it suggests that future alignment methods should invest in better value function learning rather than in more clever ways to cancel it out.
Innovation 3: The Architecture-Awareness Finding—Parameter Separation as a First-Order Design Constraint
A subtle but practically crucial contribution is the empirical finding that architectural choices about parameter sharing are not implementation details but first-order determinants of alignment quality. The paper shows that using separate networks for policy and value, and using per-prompt (rather than per-batch) value predictions, are not minor optimizations—they are necessary conditions for DRO-V to work well.
What makes this distinctive. Most alignment papers treat architecture as an implementation detail orthogonal to algorithmic innovation. DPO works with any policy network; KTO works with any policy network; the question of whether value and policy should share parameters doesn't arise because those methods don't learn a value function. DRO introduces a joint optimization over (π, V) that creates an architectural design space where previously none existed, and the paper systematically maps which choices in that space matter.
The results are stark. Figure 5 and Appendix C Figure 6 show that every step toward tighter coupling between policy and value degrades performance:
- Separate networks + multiple value predictions per batch (the full DRO-V): 76.6% vs. SFT (baseline).
- Separate networks + single value per batch: 72.1% vs. SFT (−4.5 points).
- Single network + multiple values: 55.5% vs. SFT (−21.1 points).
- Single network + single value: 57.6% vs. SFT (−19.0 points).
The head-to-head comparisons in Figure 6 confirm the pattern: the double-network-multiple-values variant beats the double-network-single-value variant 54.9% to 45.1%, and beats the single-network variants by 25+ point margins.
Why this is not obvious. The natural inclination in deep learning is to share parameters whenever possible—shared representations are more parameter-efficient, can enable transfer learning between related tasks, and are the default in multi-task and multi-head architectures. The paper's finding that parameter sharing is actively harmful for joint policy-value optimization is counterintuitive and practically important. The authors hypothesize that the policy and value function have different learning dynamics (token-level vs. prompt-level) that create destructive interference when forced into a shared representation. Whatever the mechanism, the implication is clear: if you're going to learn a value function, give it its own network.
Significance beyond raw performance. This finding changes how practitioners should think about implementing alignment methods that involve value functions. It establishes that the architecture of the value network is not a hyperparameter to tune lazily but a design constraint that must be gotten right for the method to work at all. This is analogous to the finding in GAN literature that discriminator and generator architectures must be carefully balanced—not a theoretical insight, but a practical one with substantial impact on reproducibility and real-world deployment. It also raises a deeper question that the paper does not fully answer: why does parameter sharing fail so dramatically? Is it an optimization issue (different gradient scales, conflicting update directions) or a representational one (the features needed for next-token prediction are incompatible with those needed for value prediction)? Answering this could lead to better architectures for joint policy-value learning.
Innovation 4: The DRO Framework as a Unification of Pairwise and Single-Trajectory Alignment
DRO is presented as a single-trajectory method, but a deeper contribution is that it unifies pairwise and single-trajectory alignment within a common mathematical framework. This unification is not center-stage in the paper but emerges naturally from the derivation.
The unification argument. The DRO loss (Equation 4) is derived from the optimality condition of KL-regularized RLHF without assuming any particular data format. When applied to single-trajectory data, it yields DRO-V (with an explicit value function) or DRO-P (with batch-level variance approximation). When applied to data with two completions per prompt, the DRO-P loss reduces to the IPO loss (Appendix B):
"when replacing the difference by a preference indicator, the above loss simply reduces to the IPO loss"
This means DRO is not merely an alternative to DPO/IPO—it is a generalization of them. The pairwise preference methods emerge as a special case of the DRO framework when data includes paired responses and the reward difference is interpreted as a preference signal. The single-trajectory methods (DRO-V, DRO-P) are the more general case that handles the data format naturally available in deployment.
Why this is intellectually significant. The field currently treats pairwise and single-trajectory alignment as fundamentally different problems requiring fundamentally different algorithms. DPO/KTO is the standard framing: DPO solves the pairwise problem, KTO solves the single-trajectory problem, and they have different derivations, different assumptions, and different theoretical justifications (Bradley-Terry vs. prospect theory). DRO shows that both can be derived from the same underlying optimality condition with different choices about how to handle the value function.
This unification has practical implications. It suggests that one could design hybrid methods that train on a mixture of pairwise and single-trajectory data within the same framework—using Equation (4) directly on single-trajectory data and the IPO-reduced form on pairwise data, with a shared policy network. This would allow leveraging both expensive curated pairwise data (when available) and abundant organic single-trajectory feedback, without needing to choose between DPO and KTO. The paper does not explore this hybrid setting, but the framework makes it natural.
Distinguishing from prior work. Prior work on generalized preference optimization (Tang et al., 2024) unified various pairwise methods (DPO, IPO, SLiC) under a single loss function but remained within the pairwise paradigm. DRO's unification crosses the pairwise/single-trajectory boundary, which is a more fundamental divide. This is not an incremental extension of the Tang et al. unification—it operates at a different level of abstraction, showing that the pairwise/single-trajectory distinction is an artifact of how the partition function is handled rather than a fundamental difference in the optimization problem.
The evidence for the viability of this unification is indirect but suggestive: DRO-V outperforms KTO (the specialized single-trajectory method) while being mathematically equivalent to IPO (a pairwise method) in the appropriate limit. This suggests that a single coherent framework can outperform specialized methods designed for specific data formats, which is the hallmark of a successful unification.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. All experiments use the UltraFeedback dataset (Cui et al., 2023). The paper preprocesses this dataset to create an offline dataset of triplets
(prompt, completion, reward), where the same prompt may appear across multiple triplets with different completions. The rewards are normalized to have mean 0 and variance 1 across the entire dataset. UltraFeedback is a standard benchmark in the alignment literature, consisting of prompts with model-generated completions and associated scalar feedback, making it suitable for evaluating single-trajectory methods. -
Base models. Experiments use T5 encoder-decoder language models (Raffel et al., 2020), specifically two sizes: T5-Large (770M parameters) and T5-XL (3B parameters). Checkpoints are initialized from instruction fine-tuning according to the FLAN recipe (Chung et al., 2022), which the paper denotes as the SFT (supervised fine-tuning) policy and uses as the reference policy
π_ref. The paper does not justify why T5—an encoder-decoder architecture rather than the decoder-only transformers more common in contemporary LLM alignment work—was chosen beyond citing its availability in the authors' infrastructure (Roberts et al., 2022). This architectural choice means the results may not directly transfer to decoder-only models like LLaMA or PaLM, since encoder-decoder models have different in-context learning and generation dynamics. -
Metrics. The primary metric is side-by-side win rate evaluated by an LLM judge. Specifically, the paper uses PaLM 2 (Anil et al., 2023) as an automated judge. For each pair of trained policies being compared, completions are sampled for a set of test prompts, and PaLM 2 judges which response is better based on helpfulness and fulfillment of the instruction. The win rate is the percentage of comparisons where the first model's response is preferred. This follows the now-standard practice established by Zheng et al. (2023) and used in chatbot arena evaluations. For the main results in Figures 1 and 2, the paper reports standard deviations computed over 5 evaluation folds of 1,000 test prompts each. For ablation studies, win rates are typically reported as a single number against the SFT baseline without confidence intervals.
-
Baselines. The primary baseline is Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024), which the paper identifies as essentially the only existing algorithm specifically designed for single-trajectory alignment. For KTO, the paper implements a per-batch criterion to determine which samples are "desirable" (thumbs-up) based on whether each scalar reward exceeds the average reward over the minibatch. The same KL divergence estimator from the original KTO paper is used, without loss weighting. The secondary baseline is the SFT policy (the initial instruction-tuned checkpoint before any alignment training), against which both DRO-V and KTO are compared.
-
Generation budget / compute accounting. All comparisons are performed at equal training compute, not equal inference compute. Both DRO-V and KTO are offline methods that train on a static dataset without generating new completions during training. The paper reports training step counts (40,000 steps for T5-L, 20,000 steps for T5-XL) with batch size 32, and uses identical training infrastructure (TPU v5e) for both methods. However, DRO-V requires training two separate networks (policy and value) while KTO trains only one (policy), meaning DRO-V uses approximately twice the parameters and compute per training step. This asymmetry is not accounted for in the comparisons—the paper compares best checkpoints after equal numbers of training steps, not equal FLOPs or wall-clock time.
-
Cross-validation / statistical protocol. The paper follows a checkpoint selection protocol designed to avoid overfitting: models are evaluated every 2,000 training steps via side-by-side comparison against the SFT policy, and the best checkpoint on this metric is selected as representative of the algorithm. For the main KTO comparison, the paper optimizes the KTO baseline as much as possible by searching over three values of the regularization parameter
τ ∈ {0.1, 1.0, 5.0}(withτ = 5.0being optimal for KTO), while presenting DRO-V results at a single defaultτ = 1.0without hyperparameter search. For the main results in Figures 1 and 2, standard deviations are computed over 5 evaluation folds. For the ablation studies in Figures 3–5, win rates are reported as point estimates without error bars.
Main Quantitative Results
Head-to-Head Comparison: DRO-V vs. KTO and SFT
The central empirical claim of the paper is that DRO-V significantly outperforms both the SFT baseline and the KTO single-trajectory baseline. The results are presented in Figures 1 and 2 for T5-L and T5-XL respectively.
T5-Large results (Figure 1). DRO-V achieves a 78.9% ± 0.3% win rate against the SFT baseline, substantially higher than KTO's 67.5% ± 0.7%. In direct side-by-side comparison, DRO-V beats KTO with a 63.4% ± 1.0% win rate. All three numbers are well above 50% with tight confidence intervals, indicating statistically robust outperformance. The KTO baseline is reported at its best-performing τ value of 5.0, while DRO-V uses the default τ = 1.0. The paper notes: "In the interest of fair comparison, we optimise our baseline as much as possible and present the best KTO results we obtained after search over three values of the τ regularisation parameter."
T5-XL results (Figure 2). At the 3B parameter scale, DRO-V achieves 81.5% ± 1.0% win rate vs. SFT, compared to KTO's 78.2% ± 0.7%. The direct DRO-V vs. KTO comparison yields 57.5% ± 0.8% in favor of DRO-V. While still a statistically significant win (well above 50% given the tight error bars), the margin narrows from 13.4 percentage points on T5-L to 7.5 points on T5-XL. This narrowing could indicate that KTO's constant-partition-function assumption becomes less harmful at larger model scales (where the model may learn to compensate), or that DRO-V's advantage in learning a separate value function diminishes when the policy network is more capable. The paper does not discuss this trend.
Interpretation of win rates. The paper does not report absolute performance metrics (e.g., reward achieved, task accuracy, or human preference scores), only relative win rates against baselines. This means the results quantify how often one model is preferred over another but not how good either model is in absolute terms. For context, the SFT policy against which both are compared is an instruction-tuned T5 model that has not undergone any RLHF-style alignment. Achieving ~80% win rate against SFT indicates substantial improvement but does not reveal whether the aligned model approaches the performance of stronger baselines like DPO-trained models or RLHF-trained models.
Impact of Regularization Parameter τ
Figure 4 reports the sensitivity of both DRO-V and KTO to the regularization parameter τ, evaluated on T5-L models.
DRO-V sensitivity (Figure 4, left). At τ = 1.0, DRO-V achieves its highest win rate of 78.9% vs. SFT. At τ = 0.1 (weak regularization), performance drops to 70.5%, and at τ = 5.0 (strong regularization), it drops to 76.6%. The paper notes that τ = 1.0 is "an intuitive value" and presents it as the default, without searching over this parameter for the main DRO-V results. This is an unusual methodological choice—presenting results at the optimal hyperparameter for the baseline while using an untuned default for the proposed method could be seen as either conservative (not cherry-picking DRO-V's hyperparameters) or unfair (KTO is given the benefit of hyperparameter optimization).
KTO sensitivity (Figure 4, right). KTO's performance varies substantially with τ. At τ = 0.1, KTO achieves only 63.5% vs. SFT; at τ = 1.0, 61.9%; at τ = 5.0, the best value, 67.5%. This 5.6 percentage point range indicates that KTO is quite sensitive to regularization strength. More importantly, KTO's best performance (67.5%) remains well below DRO-V's performance even at DRO-V's suboptimal τ settings (70.5% at τ = 0.1, 76.6% at τ = 5.0). This means the performance gap between the methods is not an artifact of hyperparameter tuning—DRO-V outperforms the best-tuned KTO across all tested τ values.
What these results show about regularization. The fact that KTO requires a relatively large τ (5.0) to perform well, while DRO-V works best at τ = 1.0, is consistent with the paper's theoretical argument. KTO's constant-partition-function assumption may cause it to overfit to reward signal when regularization is weak, requiring stronger KL penalties to stay close to the reference policy. DRO-V's explicit value function provides a more accurate baseline, allowing it to make effective use of reward signal even at moderate regularization strengths.
Impact of Learning Rate
Figure 3 reports win rates vs. SFT when varying the learning rate for T5-L models.
Joint learning rate sweep (Figure 3, top line). When varying the learning rate for both policy and value networks simultaneously, DRO-V achieves 73.7% at lr = 1e-5, 78.7% at 5e-5, 78.9% at 1e-4 (default), and 78.4% at 5e-4. The performance is remarkably stable within the range 5e-5 to 5e-4 (all within 0.5 percentage points), with only the lowest learning rate showing a meaningful degradation. This stability is practically important because it means DRO-V does not require precise learning rate tuning.
Value-only learning rate sweep (Figure 3, bottom line). When fixing the policy learning rate at 1e-4 and varying only the value function learning rate, performance ranges from 76.8% at 1e-5 to 79.1% at 5e-4, with a slight monotonic improvement as the value learning rate increases. The paper notes this as "a small yet monotonic improvement in learning the value V faster than the policy" and concludes that "the performance of DRO-V remains very stable within an order of magnitude change for learning rates."
Implication for the policy gradient rescaling. The relative insensitivity to value learning rate supports the paper's claim that the 1/τ policy gradient rescaling is important for conditioning. Without this rescaling, the policy and value gradients would have different natural scales, and changing the value learning rate would likely have a larger impact. The stability observed in Figure 3 suggests the rescaling is effective at decoupling the optimization dynamics.
Parameter Sharing Ablation
Figure 5 (and the full pairwise comparison matrix in Appendix C, Figure 6) reports the performance of four architectural variants of DRO-V, evaluated at 10,000 training steps on T5-L models. The paper notes these results are from "slightly undertrained" models compared to the full 40,000-step main results.
Four variants (Figure 5). The variants are:
- Double Network, Multiple Values (the full DRO-V configuration): 76.6% win rate vs. SFT.
- Double Network, Single Value (policy and value use separate networks, but
Vproduces one scalar shared across the batch): 72.1% vs. SFT. - Single Network, Single Value (shared parameters for policy and value, one scalar per batch): 57.6% vs. SFT.
- Single Network, Multiple Values (shared parameters, per-prompt value predictions): 55.5% vs. SFT.
Head-to-head comparisons (Appendix C, Figure 6). The pairwise win rate matrix confirms the hierarchy:
- Double Network, Multiple Values beats Double Network, Single Value 54.9% to 45.1%.
- Double Network, Multiple Values beats Single Network, Single Value 70.4% to 29.6%.
- Double Network, Multiple Values beats Single Network, Multiple Values 74.7% to 25.3%.
- Double Network, Single Value beats Single Network, Single Value 62.9% to 37.1%.
Interpretation. The degradation from the full DRO-V to the constrained variants is not additive but approximately hierarchical. Moving from separate networks to shared networks (comparing Double/Multiple at 76.6% to Single/Multiple at 55.5%) causes a ~21 percentage point drop. Moving from multiple values to a single value (comparing Double/Multiple at 76.6% to Double/Single at 72.1%) causes a ~4.5 percentage point drop. The combination of both constraints (Single/Single at 57.6%) is marginally worse than shared networks alone (55.5%), suggesting that parameter sharing is the dominant source of degradation and that once parameters are shared, the per-batch vs. per-prompt distinction matters less.
The paper interprets this as evidence that "most of the hit [comes] from the single or double network choice." This supports the theoretical claim that the policy and value function have incompatible learning dynamics that create destructive interference when forced into a shared representation.
Qualitative Results
Appendix E presents six side-by-side examples of completions from DRO-V and KTO on T5-XL models. These examples are illustrative rather than systematically analyzed, but they suggest a pattern: DRO-V responses tend to be more focused, concise, and directly responsive to the prompt, while KTO responses are sometimes more verbose or less directly on-topic. The paper notes: "Overall, DRO-V responses appear slightly more helpful and focused, and are generally preferred by the Palm2 LLM judge. We found that it's often a combination of helpfulness and concision, rather than just one of these dimensions, that tilts the decision one way or another."
For example, when asked "List the components of an Ethernet cable," DRO-V provides a structured, detailed response with specific components (conductors, wires, connectors) and types (Cat 5, Cat 7, Category 5), while KTO's response is less organized and mentions less relevant items (switches, hubs). When asked to determine entailment, DRO-V provides a reasoned explanation while KTO gives a one-word answer. These qualitative differences are consistent with the quantitative win rates but should be treated as anecdotal given the small sample size and lack of systematic qualitative analysis methodology.
Ablation Studies and Robustness Checks
-
DRO-P (policy-only variant) vs. DRO-V: Appendix B describes DRO-P, a variant that eliminates the value network by analytically optimizing
Vout of the loss and approximating the resulting per-prompt variance with batch-level variance. The paper reports that DRO-P "does not collapse and produces non-trivial resulting policies, that can in fact beat the SFT policy," but is "less competitive" than DRO-V. No quantitative comparison between DRO-P and DRO-V is provided in terms of win rates. This is a notable omission—the paper claims that the value function is necessary for best performance, but provides only a qualitative statement about DRO-P being "less competitive" without quantifying the gap. Including this comparison would have directly tested the paper's central claim about the importance of the value function. -
Regularization parameter τ sweep on DRO-V and KTO (Figure 4): As discussed above, both methods are evaluated at
τ ∈ {0.1, 1.0, 5.0}on T5-L. DRO-V shows robustness across this range (70.5%–78.9%), with optimal performance atτ = 1.0. KTO shows greater sensitivity (61.9%–67.5%), with optimal performance atτ = 5.0. The key robustness finding is that DRO-V at its worstτ(70.5%) still outperforms KTO at its bestτ(67.5%), meaning the DRO-V advantage is not an artifact of suboptimal KTO tuning. -
Learning rate robustness (Figure 3): DRO-V performance is stable across an order of magnitude in learning rate (
5e-5to5e-4yields 78.7%, 78.9%, 78.4% for joint sweep; 76.8% to 79.1% for value-only sweep). This is a practically important finding—it means DRO-V does not require precise learning rate tuning. The only notable degradation is at1e-5(73.7% for joint, 76.8% for value-only), which is an order of magnitude below the default. -
Parameter sharing configurations (Figures 5 and 6): Four architectural variants are compared. The subset of ablations and their key findings: (1) Using a single value per batch instead of per-prompt values: −4.5 percentage points win rate vs. SFT (76.6% → 72.1%). (2) Using a single network for both policy and value: −21.1 percentage points (76.6% → 55.5%). (3) Combining both constraints: −19.0 percentage points (76.6% → 57.6%). The non-obvious finding is that parameter sharing is far more damaging than simplifying the value function—the value function being poorly parameterized (single per-batch scalar) still allows reasonable performance as long as it has its own network, while forcing policy and value into shared parameters causes near-total collapse regardless of value function expressiveness.
-
Model scale comparison (T5-L vs. T5-XL, Figures 1 and 2): Both DRO-V and KTO improve with scale: DRO-V goes from 78.9% to 81.5% vs. SFT; KTO goes from 67.5% to 78.2% vs. SFT. The DRO-V vs. KTO margin narrows from 63.4% to 57.5%. This is a robustness check that DRO-V's advantage persists at larger scale, but the narrowing trend is not analyzed—it could indicate that KTO catches up at even larger scales, though testing this would require experiments beyond the 3B parameter range used in this paper.
-
Batch-level vs. prompt-level value prediction (Figures 5 and 6, Appendix C): Within the double-network configuration, using per-prompt values beats using a single per-batch value 54.9% to 45.1% in head-to-head comparison. Within the single-network configuration, the direction is less clear (Single/Multiple at 55.5% vs. SFT, Single/Single at 57.6% vs. SFT), but head-to-head comparison shows Single/Multiple beats Single/Single 54.8% to 45.2%. The interaction is complex, but the overall pattern supports per-prompt value predictions as beneficial.
-
Missing ablation: value network capacity and architecture. The paper uses a separate T5 network of the same size for the value function, which doubles the parameter count. No experiments vary the value network's capacity (e.g., using a smaller network, or a non-transformer architecture) to determine whether a full transformer is necessary or whether a lighter-weight value head would suffice. This is a practical concern—doubling parameters for the value function may be prohibitive at larger scales.
-
Missing ablation: effect of dataset size and composition. All experiments use the full UltraFeedback dataset. No experiments vary the amount of training data or the distribution of reward values to assess DRO-V's data efficiency or robustness to label noise. This is relevant because the paper's motivation hinges on leveraging "potentially orders of magnitude more data" from user feedback, which would likely be noisier and have different reward distributions than the curated UltraFeedback dataset.
-
Missing ablation: number of training steps. The paper trains T5-L for 40,000 steps and T5-XL for 20,000 steps, selecting the best checkpoint every 2,000 steps. No learning curves are shown, making it impossible to assess whether DRO-V converges faster than KTO, whether it overfits, or whether the performance ordering changes over the course of training. This is a significant omission because offline RL methods are known to be sensitive to the number of training steps—performance can degrade with too much training due to distribution shift.
-
Missing ablation: τ sweep on T5-XL. The
τsweep in Figure 4 is performed only on T5-L. The paper states: "we re-used thoseτchoices for T5-XL experiments as well," meaning the optimalτidentified on T5-L was applied to T5-XL without verification. If the optimal regularization strength changes with model scale (as it might, since larger models may need different KL penalties), this could disadvantage one method relative to the other at the larger scale.
Critical Assessment
Claim 1: DRO-V significantly outperforms KTO on single-trajectory alignment.
Supported, with qualifications about scope and measurement. The win rate data in Figures 1 and 2 clearly show DRO-V beating KTO at both tested model scales (63.4% on T5-L, 57.5% on T5-XL), with the KTO baseline tuned to its best hyperparameter while DRO-V uses a default. The margin is statistically significant given the reported standard deviations (±1.0% or less).
However, several qualifications are in order:
-
The comparison is not compute-matched. DRO-V trains two separate T5 networks (policy + value), doubling the parameter count and per-step compute relative to KTO, which trains only a policy network. Both are trained for the same number of steps. A fairer comparison would match FLOPs or wall-clock time, which might close some of the performance gap. The paper does not address this asymmetry.
-
The evaluation metric is an LLM judge, not ground-truth quality. Win rates measured by PaLM 2 reflect that model's preferences, which may not align with human preferences or with objective response quality. There is no human evaluation or task-accuracy metric to corroborate the automated judgments. While LLM-as-judge is standard practice, it introduces potential confounding: if PaLM 2 has biases (e.g., preferring longer responses, or responses in a particular style), these biases affect both methods equally only if they produce responses with similar characteristics. The qualitative examples suggest DRO-V responses are more concise, which could be either genuinely better or simply aligned with PaLM 2's preferences.
-
Single model family, single dataset. All results use T5 encoder-decoder models on UltraFeedback. Whether DRO-V's advantage over KTO transfers to decoder-only architectures (LLaMA, GPT), larger scales (7B, 70B), or other datasets is untested. The paper's claim that DRO-V is a "simple and empirically compelling method for single-trajectory policy optimisation" is supported only in the specific context tested.
-
The gap narrows at larger scale. The DRO-V over KTO win rate drops from 63.4% (T5-L) to 57.5% (T5-XL). The paper does not discuss whether this trend would continue at even larger scales. It is possible that KTO's constant-partition-function assumption becomes less harmful as model capacity increases (the model can learn to compensate), or that the value function's benefit diminishes. Without experiments at 7B+ scale, the claim of "significant" outperformance at production-relevant scales is uncertain.
Claim 2: The value function is structurally necessary, not merely helpful.
Partially supported, with inferential gaps. Proposition 1 provides a theoretical argument that value function errors bias the policy. The parameter sharing ablation (Figures 5 and 6) shows that constraining the value function (single per-batch value) degrades performance by 4.5 percentage points, and that eliminating the separate value network (shared parameters) degrades it by ~21 points. These results are consistent with the claim that the value function matters.
However, these experiments do not distinguish between two hypotheses: (a) the value function is necessary because it corrects for off-policy bias (the paper's theoretical claim), or (b) the value function is helpful because it provides a useful inductive bias or regularization that improves optimization, but is not strictly necessary. The fact that DRO-P (no value network at all) "does not collapse and produces non-trivial resulting policies" that beat SFT suggests that the value function is not absolutely necessary for improvement, just important for achieving the best results. The quantitative gap between DRO-P and DRO-V is not reported, making it impossible to calibrate just how much the value function contributes.
Additionally, the parameter sharing ablation confounds two factors: whether a value function is learned, and how it is parameterized. The shared-network variants still have a value function—they just learn it poorly due to parameter conflict. The DRO-P variant, which truly eliminates the value function, would isolate the value function's necessity, but its performance is described only qualitatively. The paper's strongest evidence for the value function's necessity would be a three-way comparison: DRO-V (separate value network), DRO-V with shared parameters (value function exists but is poorly learned), and DRO-P (no value function). Such a comparison is partially present (Figures 5 and 6, Appendix B description) but not presented as a coherent ablation.
Claim 3: DRO is a general framework that unifies single-trajectory and pairwise alignment.
Supported theoretically but not empirically. The derivation in Appendix B shows that DRO-P with two samples per prompt reduces to the IPO loss, which is a pairwise method. This establishes a formal connection between the DRO framework and pairwise alignment. However, no experiments test this unification claim. Specifically:
- No experiments train DRO on a mixture of single-trajectory and pairwise data to show that the framework handles both seamlessly.
- No experiments compare DRO-V against pairwise methods like DPO or IPO on pairwise data to verify that DRO-V is competitive in the regime it theoretically generalizes.
- No experiments show that DRO-V trained on single-trajectory data can match or approach the performance of pairwise methods trained on pairwise data from the same underlying reward distribution.
The unification claim is thus a theoretical contribution without empirical validation. It is intellectually significant (as argued in the Key Insights section) but the experiments do not directly support it.
What would strengthen the paper
-
Compute-matched comparisons. Run KTO with twice the training steps (or DRO-V with half) to control for the fact that DRO-V uses two networks. Alternatively, implement a version of DRO-V where the value function is a lightweight head on top of the policy network's encoder (not a separate transformer) to reduce the parameter count asymmetry.
-
Decoder-only model experiments. At minimum, replicate the main DRO-V vs. KTO comparison on a decoder-only architecture (e.g., LLaMA-7B) to establish that the findings are not T5-specific.
-
Quantitative DRO-P comparison. Report the win rate of DRO-P vs. SFT and vs. DRO-V to calibrate the value function's contribution on an absolute scale.
-
Learning curves. Show win rate vs. training steps for DRO-V and KTO to assess convergence speed, overfitting behavior, and whether the performance ordering is consistent throughout training.
-
Human evaluation or task-accuracy metrics. Supplement the LLM-judge win rates with either human preference judgments or task-specific accuracy metrics (if UltraFeedback supports them) to validate that the LLM judge's preferences correspond to genuine quality improvements.
-
Scale to 7B+ parameters. The paper's motivation emphasizes leveraging abundant single-trajectory data at scale, but the experiments stop at 3B parameters. Demonstrating the approach at scales where alignment is most impactful (and most expensive) would substantially strengthen the practical relevance.
Overall assessment
The experiments genuinely support the narrow claim that DRO-V outperforms KTO on T5 models evaluated on UltraFeedback with an LLM judge. They provide suggestive but not conclusive evidence for the broader claims about the value function's necessity and the framework's generality. The paper's theoretical contributions are stronger than its empirical validation—the DRO framework is elegantly derived and well-motivated, but the experiments leave unaddressed several practical questions about scalability, architectural generality, and the gap between LLM-judge preferences and genuine alignment quality. The methodological choices (not controlling for parameter count asymmetry, not reporting DRO-P results quantitatively, testing only on T5) limit the strength of conclusions that can be drawn about DRO-V's practical advantages over KTO in real-world deployment scenarios.
6. Limitations and Trade-offs
The Theoretical Guarantees Require a Support Condition That May Not Hold in Practice
The assumption or constraint. Theorem 1's uniqueness proof for the global optimum of the DRO loss requires a support condition: "assuming the supports of μ and π_ref coincide, (i.e., for all x ∈ supp ρ, supp(μ(⋅|x)) = supp(π_ref(⋅|x))), then (π*, V*) is the unique global optimum." The data-generating behavior policy μ and the reference policy π_ref must assign non-zero probability to exactly the same set of responses for every prompt. If there exists some response y that π_ref can generate but μ never generated in the offline dataset, or vice versa, the uniqueness guarantee does not hold.
The consequence. In practice, the offline dataset is collected by some unknown behavior policy μ—possibly a different model, or an earlier checkpoint, or a model with different decoding parameters. This behavior policy almost certainly has a different support than the reference policy π_ref used during DRO training. When the supports differ, Theorem 1 only guarantees that the DRO loss has a global minimum at (π*, V*), not that this minimum is unique. There could be multiple distinct (π, V) pairs that all achieve zero (or near-zero) loss, and gradient descent could converge to any of them—including ones that produce policies different from π* on responses outside μ's support. This is the standard extrapolation problem in offline RL: the algorithm cannot learn about actions it never sees in the data. For alignment, this means DRO-V's learned policy may be unreliable for responses that are plausible under π_ref but absent from the training data—exactly the kind of distribution-shifted responses that tend to cause safety issues in deployed systems.
What evidence exists in the paper. The paper provides no empirical investigation of this support mismatch. All experiments use UltraFeedback data where μ and π_ref likely have substantial overlap (both are derived from instruction-tuned models), but no analysis of support coverage is presented. There is no experiment where μ and π_ref are deliberately mismatched (e.g., μ from a smaller or differently-trained model) to test whether DRO-V degrades gracefully or catastrophically when the uniqueness guarantee is violated. The paper cites this condition in Theorem 1 but does not return to it in the experimental section.
Mitigation status. Not addressed. The paper does not discuss how to verify or ensure the support condition in practice, does not propose diagnostics for when it is violated, and does not suggest algorithmic modifications (e.g., adding an exploration bonus or entropy regularization) to mitigate the problem. This is a fundamental limitation shared with virtually all offline RL methods, but the paper's framing—"we do not make any assumptions on the form of Z or V"—overstates the assumption-free nature of DRO by not foregrounding the support condition that its strongest theoretical result depends on.
DRO-V Requires Training a Separate Full-Size Transformer for the Value Function, Doubling Compute and Memory
The assumption or constraint. DRO-V parameterizes the value function V_φ as a separate neural network, and the paper finds that "using two separate networks, one for π_θ and one for V_φ, was beneficial empirically, compared to using policy logits as value outputs." The ablation in Figure 5 (and Appendix C, Figure 6) shows that sharing parameters between policy and value networks causes win rate vs. SFT to collapse from 76.6% to as low as 55.5%—a ~21 percentage point degradation. The paper's default configuration therefore uses two complete T5 encoder-decoder models: one for the policy and one for the value function. This doubles the parameter count, memory footprint, and per-training-step FLOPs relative to methods like KTO that train only a policy network.
The consequence. The headline comparison between DRO-V and KTO (Figures 1 and 2) is not compute-matched. Both are trained for 40,000 steps (T5-L) or 20,000 steps (T5-XL) with batch size 32, but DRO-V processes approximately twice as many parameters and FLOPs per step. A practitioner choosing between methods would need to know whether DRO-V's 63.4% win rate advantage (T5-L) persists when KTO is given equivalent compute—either by training KTO for twice as many steps, using a larger batch size, or using a larger model. The paper provides no such comparison. At deployment time, the value network is not needed for inference (only the policy π_θ generates responses), so inference cost is identical. But the training cost asymmetry matters for organizations with fixed training budgets deciding how to allocate compute between different alignment methods.
Moreover, the need for a full separate network limits DRO-V's scalability to very large models. For a 70B-parameter policy, DRO-V would require an additional 70B-parameter value network plus the memory to store both during training. The paper's experiments stop at 3B parameters (T5-XL) and do not discuss whether the separate-network requirement holds at larger scales or whether the value network's capacity could be reduced.
What evidence exists in the paper. The evidence comes from the parameter sharing ablation in Figures 5 and 6. The paper explicitly notes the cost: separate networks double the parameter count. However, the paper frames this as an empirical finding ("we found that using two separate networks... was beneficial empirically") rather than as a limitation, and does not quantify the wall-clock time or FLOPs difference between DRO-V and KTO training runs. The training speeds reported (0.5 steps/second for T5-L, 0.1 steps/second for T5-XL) are for DRO-V only, with no comparable numbers for KTO.
Mitigation status. Partially addressed through DRO-P (Appendix B), which eliminates the value network entirely by using batch-level variance estimation. However, the paper describes DRO-P only qualitatively as "less competitive" than DRO-V without reporting win rates, so practitioners cannot assess the performance/cost tradeoff. The paper does not explore intermediate options: a smaller value network (e.g., a T5-Small paired with T5-L policy), a shared encoder with separate policy and value heads, or distillation of the value network after training. These unexplored options could recover most of DRO-V's performance at substantially lower training cost.
All Experiments Use a Single Model Architecture (T5 Encoder-Decoder) and a Single Dataset (UltraFeedback), with No Evidence of Transfer to Decoder-Only LLMs
The assumption or constraint. Every experiment in the paper uses T5 encoder-decoder language models—specifically T5-L (770M) and T5-XL (3B)—evaluated exclusively on the UltraFeedback dataset. The paper characterizes this as a limitation in Section 6: "our empirical study is limited, both in terms of number of tasks and scale. Further work is required to more broadly establish the performance gains that our approach provides when considering the largest language models." The choice of T5 is architectural: these are encoder-decoder transformers with bidirectional attention over the input (prompt), unlike the decoder-only architectures (GPT, LLaMA, PaLM) that dominate contemporary LLM alignment work. The value function V_φ(x) is computed from the prompt x alone—in T5, this means encoding the prompt through the full encoder stack to produce a representation, then mapping that representation to a scalar.
The consequence. It is unclear whether DRO-V's performance advantage over KTO transfers to decoder-only architectures. In a decoder-only model, there is no separate encoder—the prompt and response are concatenated and processed by the same transformer stack. Implementing a value function V_φ(x) that takes only the prompt would require either (a) running the transformer on the prompt alone (potentially wasting causal attention capacity designed for full sequences), (b) using a separate encoder network (incurring the same doubling of parameters as in T5, but now requiring a different architecture for the value network than for the policy), or (c) extracting the value from the policy network's representation of the last prompt token (which the parameter sharing ablation suggests would severely degrade performance). The optimal implementation strategy for decoder-only models is non-obvious and untested.
Additionally, UltraFeedback is a specific dataset with its own reward distribution and prompt characteristics. Whether DRO-V's advantage generalizes to other alignment datasets, tasks beyond helpfulness (e.g., safety, honesty, instruction-following), or reward scales different from the mean-0-variance-1 normalization used here is unknown. The paper's motivation—leveraging "potentially orders of magnitude more data coming from user feedback"—implies deployment on diverse, noisy, real-world feedback data, but all experiments use a curated research dataset.
What evidence exists in the paper. The paper explicitly acknowledges this limitation in Section 6. No experiments with decoder-only models, other datasets, or other alignment tasks are presented. No ablation studies vary reward normalization schemes to test sensitivity. The qualitative examples (Appendix E) are from T5-XL on UltraFeedback prompts, confirming that the evaluation is entirely within this narrow scope.
Mitigation status. Acknowledged but not addressed. The paper calls for "further work" to establish performance at larger scales and with the largest language models. Given the architectural differences between T5 and decoder-only LLMs, this is not a trivial extension—it may require redesigning how the value function is implemented, and the separate-network requirement may have different implications for models with different inductive biases.
The Value Function Gradients Are Biased When the Data Is Off-Policy, and the Error Bound Depends on a Density Ratio That Can Be Unbounded
The assumption or constraint. Proposition 1 provides an error bound for the policy learned when optimizing against an approximate value function: the log-ratio between the learned policy π_V and the true optimal policy π* is bounded by a term involving |1 - π_V(y|x) / μ(y|x)|. The quantity π_V(y|x) / μ(y|x) is the density ratio between the learned policy and the data-generating behavior policy at a particular response y. The error bound is tight when this ratio is close to 1 (the learned policy stays near the behavior policy), but can be arbitrarily large when the ratio diverges—that is, when π_V assigns high probability to responses that μ rarely generates, or low probability to responses that μ frequently generates.
The consequence. DRO-V has no mechanism to prevent the learned policy from diverging into regions where π_V / μ is large. The ℓ_2 regularization in the policy gradient (Equation 8) penalizes deviation from π_ref, not from μ, and π_ref may itself differ from μ. If during training, π_θ begins to upweight a response that μ rarely generates but π_ref assigns moderate probability to, the density ratio π_θ/μ becomes large. In this regime, Proposition 1's error bound becomes vacuous—the learned policy could be arbitrarily far from π*—and the gradient updates based on data from μ provide essentially no signal about whether this upweighting is correct. This is the standard distribution shift problem in offline RL, and DRO-V provides no solution beyond hoping that the ℓ_2 regularization toward π_ref keeps π_θ close enough to μ that the density ratio stays bounded.
The practical manifestation would be that DRO-V overfits to the training data: it learns to assign high probability to responses that happen to have high reward in the dataset, without learning whether those responses are genuinely good or are artifacts of the specific behavior policy that collected the data. The policy might then produce these responses at test time even when they are inappropriate, because the value function and policy gradient provide no corrective signal for responses outside μ's support.
What evidence exists in the paper. There is no direct evidence. The paper does not measure the density ratio π_θ/μ during training, does not evaluate overfitting (e.g., by comparing train vs. validation loss or by checking whether policy entropy collapses), and does not present learning curves that would reveal degradation with extended training. The offline nature of the algorithm is discussed in Section 3.3 ("The dataset of (prompt, completion, reward) triplets remains static during optimisation") but the associated risks are not empirically assessed. There is no comparison with an online variant that regenerates completions from the current policy, which would directly test whether distribution shift is harming DRO-V's performance.
Mitigation status. Partially addressed through the ℓ_2 regularization toward π_ref, which provides an indirect constraint on how far π_θ can move. However, π_ref is not μ—it is the initial SFT checkpoint before alignment—so the regularization does not directly bound π_θ/μ. The paper does not discuss techniques from the offline RL literature that explicitly address distribution shift, such as policy constraints (e.g., KL toward μ rather than π_ref), conservative value estimates, or importance-weighting corrections. The offline setting is presented as a benefit ("simplicity and computational efficiency") without a corresponding discussion of its fundamental limitations.
Evaluation Relies Exclusively on an LLM Judge with No Human Validation or Task-Accuracy Metrics
The assumption or constraint. All quantitative results in the paper (Figures 1–5) are measured using side-by-side comparisons judged by PaLM 2, an LLM-as-judge following the protocol of Zheng et al. (2023). The paper states: "following now standard practice, our evaluation pipeline consists in automated evaluation; specifically, side-by-side comparison. We use the PaLM2 LLM as a judge." No human evaluation, no task-specific accuracy metrics, and no reward model scoring are reported. The only supplementary evidence is six qualitative examples in Appendix E, which are illustrative rather than systematically evaluated.
The consequence. LLM judges are known to exhibit systematic biases—preferring longer responses, preferring responses in a particular style, preferring responses from models similar to the judge's own training distribution, and exhibiting position bias (favoring the first or second response depending on presentation order). The paper does not control for any of these confounds. The qualitative examples suggest DRO-V responses tend to be more concise than KTO responses (e.g., the entailment example where DRO-V provides a reasoned explanation while KTO gives a one-word answer; the Ethernet cable example where DRO-V's response is more structured). If PaLM 2 has a length bias or a verbosity preference, this could systematically advantage one method over the other in ways unrelated to genuine response quality.
More critically, the paper's central claim—that DRO-V provides better alignment from single-trajectory data—is evaluated only through relative preferences, not absolute alignment quality. A method could achieve high win rates against KTO and SFT while still producing responses that are factually incorrect, unhelpful, or unsafe by human standards. Without task-accuracy metrics (e.g., correctness on factual QA subsets of UltraFeedback) or human preference judgments, the relationship between LLM-judge win rates and genuine alignment improvement is uncalibrated.
What evidence exists in the paper. The main results report standard deviations over 5 evaluation folds, suggesting the win rates are statistically robust under resampling of the test prompts. However, this only quantifies variance due to prompt sampling, not bias due to judge preferences. There is no correlation analysis between PaLM 2's judgments and any ground-truth quality signal. The qualitative examples (Appendix E) are presented as confirming the quantitative results but are selected by the authors and may not be representative. The paper acknowledges the automated evaluation approach without discussing its limitations or potential biases.
Mitigation status. Not addressed. The paper does not report inter-judge agreement (e.g., using multiple different LLM judges), does not ablate the judge prompt format, does not include human evaluation, and does not report task-accuracy metrics even for subsets of UltraFeedback where correctness could be verified. The reliance on a single LLM judge without validation is a significant gap between the paper's claims about "alignment" and the evidence provided. Alignment to human preferences is the stated goal; human preferences are never directly measured.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper initiates a shift in how the alignment community thinks about data requirements for RLHF, but it is best understood as a methodological reframing with practical implications rather than a paradigm shift. The core conceptual move—treating single-trajectory alignment as regression on the KL-regularized optimality condition rather than as a preference discrimination problem—is genuinely novel and opens up a design space that was previously invisible to the field. However, the empirical evidence (single model family, single dataset, moderate scale) limits how confidently we can project this reframing onto production-scale alignment pipelines.
What changes immediately. The paper establishes that single-trajectory data can be used for alignment without the constant-partition-function assumption that KTO makes, and that explicitly learning a value function yields stronger results than assuming it away. This is not merely an incremental improvement over KTO—it is a different conceptual path to the same destination. The 63.4% win rate over KTO on T5-L (Figure 1) and the 4.5 percentage point degradation when constraining the value function to a per-batch scalar (Figure 5) together argue that the value function is not an implementation detail but a first-order modeling choice. The practical consequence is that teams working with single-trajectory data now have reason to invest in learning a separate value network rather than adopting KTO's simpler approach.
More broadly, the paper reconciles a tension between the theoretical formulation of RLHF and the practical algorithms used to implement it. The canonical RLHF objective (Equation 1) has a known closed-form solution involving a value function V*(x). Pairwise methods (DPO, IPO) exploit the cancellation of this term when differencing two completions of the same prompt. KTO assumes it is constant. DRO is the first method to say: this term is real, it varies per prompt, and learning it properly improves alignment. This closes a gap between theory and practice that had been papered over by the pairwise data format. The implication is that future alignment methods should not treat the partition function as a nuisance to eliminate but as signal to exploit—it encodes information about prompt difficulty and the range of acceptable responses that a purely contrastive loss discards.
Which research directions become more attractive. The paper makes value function learning for alignment a newly legitimate research direction. Prior to this work, learning a value function was seen as a cost of online RLHF methods (PPO requires a value network for variance reduction) that offline methods like DPO elegantly avoided. DRO reframes the value function as a source of useful inductive bias, not merely a computational overhead. This suggests several lines of work: better value function architectures (do we need a full transformer, or can a lighter head suffice?), value function pre-training (can we learn a general-purpose value function that transfers across reward models?), and value-guided decoding (using V_φ(x) at inference time to adjust sampling parameters per prompt).
The paper also makes hybrid pairwise/single-trajectory training newly tractable. Because DRO's framework unifies pairwise and single-trajectory alignment under the same optimality condition (Appendix B shows IPO is a special case of DRO-P), it becomes natural to train on mixed datasets—curated pairwise preferences for high-stakes safety-critical behavior and abundant single-trajectory feedback for broad helpfulness. This was not conceptually coherent under the previous paradigm where DPO and KTO had entirely different derivations and theoretical justifications.
Which directions become less attractive. The paper's findings about parameter sharing (Figures 5 and 6) cast doubt on the strategy of adding alignment heads to existing policy networks. The catastrophic ~21 percentage point degradation from sharing parameters between policy and value networks suggests that joint policy-value optimization with shared representations is fundamentally difficult, not just a matter of tuning. This makes approaches that attempt to extract value estimates from the policy network's internal representations (e.g., using the final token embedding as V(x)) seem unpromising without novel architectural innovations to prevent destructive interference. Simplifying assumptions about constant partition functions—KTO's approach—also lose credibility given DRO-V's consistent outperformance across τ values (Figure 4).
Follow-Up Research This Work Enables
1. Value function capacity scaling: does it need to be a full transformer? The paper uses a separate T5 network of equal size for the value function, doubling training parameters. The parameter sharing ablation (Figure 5) shows that a full separate network (76.6% vs. SFT) dramatically outperforms shared parameters (55.5%), but does not test intermediate configurations. A strong follow-up would sweep value network capacity while keeping the policy network fixed: attach a lightweight value head (a small MLP or a single transformer layer) to a frozen copy of the policy network's encoder, and measure whether win rate vs. SFT saturates well before reaching the full network's parameter count. The key question is whether DRO-V's performance comes from having any separate parameters for the value function (avoiding destructive interference) or from having a high-capacity separate network. The practical stakes are high: if a 10% overhead value head achieves 95% of the full-network performance, DRO-V becomes far more practical at scale.
2. Online-offline hybrids: does regenerating completions during training close the density ratio gap? Proposition 1's error bound depends on π_V/μ, the density ratio between the learned policy and the data-generating behavior policy. When this ratio diverges, the bound becomes vacuous and the policy gradient is biased in unknown ways. The paper explicitly chooses the offline setting to "circumvent the associated difficulties" of online sampling (Section 2), but a natural follow-up would periodically regenerate completions from the current policy (e.g., every 2,000 steps), add them to the dataset with fresh reward labels, and continue training. This is a mild online correction that keeps μ tracking π_θ, bounding the density ratio. The experiment would compare: (a) pure offline DRO-V, (b) DRO-V with periodic resampling and relabeling, and (c) KTO with the same resampling. The prediction from Proposition 1 is that the gap between (a) and (b) should grow with training steps as distribution shift accumulates, and that (b) should show less performance degradation with extended training. Learning curves beyond 40,000 steps would be essential.
3. Stress-testing the uniqueness condition: what happens when μ and π_ref have deliberately mismatched supports? Theorem 1 requires supp(μ) = supp(π_ref) for uniqueness of the global optimum. In practice, the offline dataset is collected by some model μ that may differ substantially from the reference policy π_ref. A direct stress test: create training data where μ is a small or differently-trained model (e.g., T5-Small or a non-instruction-tuned T5), use the standard FLAN-tuned T5-L as π_ref, and measure DRO-V's win rate vs. SFT as a function of the KL divergence between μ and π_ref. The prediction is that DRO-V's performance should degrade as the mismatch grows, and that the degradation should be larger than for methods that make weaker assumptions about the partition function (KTO). If DRO-V remains robust even under substantial mismatch, the support condition may be sufficient but not necessary; if it degrades sharply, practitioners need a diagnostic for when their data is "close enough" to π_ref.
4. Decoder-only implementation strategies: how should V_φ(x) be computed when there is no separate encoder? The paper uses T5 encoder-decoder models where the value function naturally operates on the encoder output. Decoder-only models (GPT, LLaMA) lack a separate encoder. A systematic comparison of implementation strategies on a decoder-only model (e.g., LLaMA-7B) would test: (a) running the transformer on the prompt alone and extracting the last-token representation as input to a value head, (b) adding a separate small bidirectional encoder for the value function (breaking the "same architecture" pattern but potentially cheaper than a full second decoder), (c) using a separate decoder-only model of the same architecture (the direct analog of the T5 setup), and (d) using the same network with a value head but with a stop-gradient on the shared encoder when computing value gradients (attempting to prevent destructive interference). The paper's parameter sharing ablation predicts that (d) will underperform (a)-(c), but the relative ordering of (a)-(c) is unknown and practically consequential. This experiment would also test whether DRO-V's advantage over KTO transfers to decoder-only architectures at all.
5. Mixed-data training: pairing DRO-V with DPO on combined pairwise and single-trajectory data. The paper's Appendix B shows that DRO-P with two samples per prompt reduces to the IPO loss. This suggests a natural experiment: create a dataset that is half pairwise preferences (from UltraFeedback's original paired annotations) and half single-trajectory triplets (the format used in this paper), and train a single policy using a combined loss—DRO-V on the single-trajectory portion, IPO on the pairwise portion, with a shared policy network and separate value network (needed only for the DRO-V term). Compare against (a) DPO trained on the full pairwise data alone, (b) DRO-V trained on the single-trajectory data alone, and (c) a naive concatenation where DPO and KTO are applied separately to their respective data subsets. The prediction is that the combined DRO/IPO training should outperform either data source alone when total data volume is held constant, because the pairwise data provides high-quality relative signal for difficult discriminations while the single-trajectory data provides scale. This would directly test the paper's unification claim and establish whether single-trajectory data is a complement to or a substitute for pairwise preferences.
6. Verifier ensemble diagnostics: can we detect over-optimization of the value function? The paper's offline training means there is no natural check on whether V_φ is overfitting to spurious reward patterns in the training data. A diagnostic experiment: train multiple DRO-V instances with different value function initializations (same policy initialization, different random seeds for V_φ). Measure the variance of V_φ(x) predictions on held-out prompts as training progresses. If variance grows with training steps while policy win rate plateaus or declines, this would indicate value function over-optimization—different value functions learn different spurious patterns that all fit the training data equally well but produce diverging policy gradients. This diagnostic could serve as an early stopping criterion, analogous to using a validation set in supervised learning. A positive result (value ensembles detect overfitting before performance degrades) would give practitioners a practical tool for deploying DRO-V safely; a negative result (value function variance is uncorrelated with downstream performance) would suggest that DRO-V is robust to value function approximation error in ways not captured by Proposition 1's worst-case bound.
Practical Applications and Downstream Use Cases
1. Continuous alignment from production user feedback. Any deployed LLM system that collects user feedback—thumbs-up/down on chatbot responses, "was this helpful?" prompts after code generation, satisfaction ratings after summarization—generates exactly the (x, y, r) triplets that DRO-V consumes. The paper's motivation section explicitly argues that "single-trajectory data is much more abundant in the wild" (Section 1), and DRO-V provides the algorithmic machinery to use it. The practical workflow: every night, aggregate the day's user feedback into a dataset of (prompt, model response, binary or scalar rating) triplets, normalize rewards to mean 0 variance 1 (as done in this paper), and run DRO-V fine-tuning starting from the current deployed policy as π_ref, with a separate value network trained from scratch or warm-started from the previous day's checkpoint. The 63.4% win rate improvement over KTO (T5-L, Figure 1) suggests that the choice of alignment algorithm for this pipeline matters substantially—using KTO would leave performance on the table relative to DRO-V. The compute cost is training a second model of equal size to the policy, which is significant but may be justified for high-value deployments where even small alignment improvements translate to user retention or satisfaction.
2. Bootstrapping alignment for new domains or languages without pairwise annotation. Collecting pairwise preference data requires trained annotators, annotation guidelines, quality control, and typically weeks of calendar time. Single-trajectory feedback is often available from the start—user satisfaction ratings in production logs, upvotes on community forums, or even automatic metrics (e.g., code execution success, factuality checks against knowledge bases) that can serve as scalar rewards. DRO-V enables bootstrapping an aligned policy from this weaker signal before investing in pairwise annotation. The practical workflow: launch with a base instruction-tuned model, collect 10,000–100,000 single-trajectory feedback points from early users, train DRO-V to produce an initial aligned policy, and then—only if further improvement is needed—commission pairwise annotation on the hardest cases where the single-trajectory signal is ambiguous. The 81.5% win rate over SFT on T5-XL (Figure 2) suggests that even with moderate-scale data, DRO-V produces substantial improvements over the unaligned baseline, potentially making pairwise annotation unnecessary for many applications.
3. Lightweight alignment for on-device or specialized models. The paper's finding that the value network must be separate from the policy network (Figure 5: shared parameters cause a 21-point degradation) has an important corollary: the value network is only needed during training. At inference time, only the policy network π_θ is used. This means DRO-V can train a small on-device policy using a larger value network (providing better training signal) that is then discarded. The practical configuration: train a T5-L policy with a T5-XL value network (or larger), then deploy only the T5-L policy. The value network's additional capacity may improve the quality of the learned policy by providing more accurate baseline estimates (Proposition 1's bound depends on value function accuracy), while the inference footprint is unchanged. This is a form of teacher-student asymmetry specific to alignment: the value function is a training-time teacher that doesn't ship. The paper provides no experiments varying value network capacity independently of policy capacity, but the theoretical motivation is direct and the practical upside is substantial for resource-constrained deployments.
4. Data filtering and prompt difficulty estimation using the learned value function. The value function V_φ(x) estimates the expected reward-minus-regularization achievable from prompt x under the reference policy. This is effectively a prompt difficulty score for alignment: prompts where V_φ(x) is low are those where π_ref cannot produce high-reward responses—they are "hard" from an alignment perspective. This could be used operationally to route difficult prompts to a more capable model, to flag them for human review, or to prioritize data collection. After training DRO-V, compute V_φ(x) on a large pool of unlabeled prompts, sort by value, and sample the lowest-V prompts for targeted pairwise annotation (since these are the cases where single-trajectory feedback is least informative). The paper does not evaluate this use case, but it follows directly from the learned value function that DRO-V produces and requires no additional training.
When to Prefer This Method
The paper explicitly positions DRO-V against KTO as the primary single-trajectory baseline, and implicitly against DPO/IPO as the dominant pairwise methods. The decision rules below are grounded in the paper's empirical comparisons and theoretical analysis.
-
Prefer DRO-V over KTO when you have single-trajectory data and training compute budget allows for a second network of comparable size to the policy network. Evidence: DRO-V achieves 63.4% and 57.5% win rates over KTO on T5-L and T5-XL respectively (Figures 1 and 2), and the 4.5 percentage point degradation from using a per-batch constant value (Figure 5) suggests KTO's constant-partition-function assumption directly limits performance. The training cost is roughly 2× KTO's due to the separate value network, but inference cost is identical. The oracle set of assumptions that would make KTO preferable—that the partition function is genuinely constant across prompts—is tested implicitly by the single-value ablation and rejected.
-
Prefer DRO-V as an alternative to DPO/IPO when pairwise preference data is unavailable or prohibitively expensive, and you have access to single-trajectory feedback at scale. Evidence: the paper demonstrates that single-trajectory training can produce policies that substantially outperform the SFT baseline (78.9% win rate on T5-L, 81.5% on T5-XL; Figures 1 and 2) and that DRO's framework is theoretically unified with IPO (Appendix B). However, the paper provides no direct comparison between DRO-V trained on single-trajectory data and DPO trained on pairwise data from the same underlying reward distribution, so the performance gap between single-trajectory and pairwise alignment remains unquantified. The conditional is: DRO-V is preferable when the data constraint (no pairwise preferences) is binding, not necessarily when both data types are available.
-
Prefer DRO-P (the policy-only variant, Appendix B) over DRO-V when training compute or memory constraints make a second full-size transformer prohibitive, and you are willing to accept a reduction in alignment quality. Evidence: the paper describes DRO-P as beating SFT but being "less competitive" than DRO-V, without reporting quantitative win rates. This is a thin evidence base for a practical decision, but the direction is clear from the parameter sharing ablation—any constraint on the value function reduces performance, and eliminating it entirely is the extreme case of that constraint. The specific condition is: when 2× training parameters is infeasible and some alignment improvement over SFT is better than none.
-
Prefer KTO over DRO-V when you need the absolute simplest implementation (single network, no value function, standard loss) and the 4–21 percentage point performance gap documented in this paper is acceptable for your use case. This might apply in rapid prototyping, educational contexts, or extremely resource-constrained settings where a second network is genuinely impossible. The paper optimizes KTO's hyperparameters (τ = 5.0 being optimal, Figure 4) and reports its best performance as 67.5% win rate vs. SFT on T5-L, which is substantially above chance (50%) and may be sufficient for some applications.
