RL Training2024~112 minarXiv:2406.11827

WPO: Enhancing RLHF with Weighted Preference Optimization

score 9Wenxuan Zhou, Ravi Agrawal, Shujian Zhang +5Zoom AI↑ 17★ 41Jun 17, 2024cs.CLhttps://github.com/wzhouad/wpo

🎯 Pitch

Shockingly, simply reweighting off-policy preference pairs by how likely the current model would have generated them closes much of the gap to expensive on-policy training. WPO achieves this by multiplying DPO’s loss per sample by the model’s own probability—boosting Alpaca Eval 2 win rates by up to 5.6 points—yet costs nothing extra.

1. Executive Summary

This paper proposes Weighted Preference Optimization (WPO), a method that simulates on-policy preference optimization using off-policy data by reweighting preference pairs according to their probability under the current policy—addressing the distributional gap that arises when the data-generating policy differs from the target policy. Evaluated on instruction-following benchmarks (Alpaca Eval 2 and MT-bench) using Mistral-7B and Llama-3-8B-Instruct models, WPO improves over Direct Preference Optimization (DPO) by up to 5.6% in length-controlled win rate on Alpaca Eval 2 in the off-policy setting and achieves a length-controlled winning rate of 76.7% against GPT-4-turbo when combined with a hybrid on-policy/off-policy data mixture using Gemma-2-9b-it. The method operates by multiplying DPO's per-pair loss by the length-normalized joint probability of the preferred and dispreferred outputs under the current policy—effectively upweighting preference pairs that are more representative of what the current model would generate—establishing that off-policy data can be made to approximate on-policy behavior without the computational cost of online sampling, though the performance gap between true on-policy and off-policy optimization persists.

2. Context and Motivation

The Core Problem: Off-Policy Preference Optimization Suffers from a Distributional Gap

The fundamental problem this paper addresses is a subtle but consequential mismatch in how we train language models to align with human preferences. When we use Direct Preference Optimization (DPO) or similar methods to fine-tune an LLM on preference data—pairs of responses where one is labeled "better" than the other for a given prompt—we're typically training on data that was not generated by the model we're currently training. This is the off-policy setting: the preference pairs were produced by other models (e.g., GPT-4, Llama-2, or a mixture of systems) and then curated into datasets like Ultrafeedback (Cui et al., 2023). Compared to on-policy training, where the model generates its own outputs and receives feedback on them in real time, off-policy training is dramatically cheaper and easier to scale. You don't need to run expensive online sampling loops or repeatedly query reward models during training—you just load a static dataset and optimize.

But this convenience comes at a cost, and the cost is what the paper identifies as the distributional gap. The preference pairs in an off-policy dataset were drawn from some other distribution over outputs—call it the data-generating distribution—that differs from the distribution of outputs the current policy model would produce. During DPO training, every preference pair in the dataset receives equal weight in the loss function, regardless of how probable or improbable those outputs are under the policy being trained. This means the model spends as much optimization effort trying to rank a pair of outputs that it would never generate in practice as it does on pairs that are highly representative of its own behavior.

The paper expresses this clearly in Section 3.2:

"DPO treats both instances equally in its loss calculation, ignoring the fact that the first tuple, representing a more probable output of the current policy, should ideally exert a greater influence on the optimization process. This oversight can lead to suboptimal performance, as DPO does not prioritize learning from the most representative or probable output of the policy model."

This is not merely a theoretical concern—it has practical consequences. Off-policy DPO can waste compute on preference pairs that are essentially irrelevant to the policy's actual output distribution, while underweighting pairs that would provide the most useful training signal. The result is suboptimal alignment and, as the paper documents in Appendix A (Figure 5), greater training instability: DPO performance collapses after 2–3 epochs due to reward model overoptimization, while WPO remains stable across 5+ epochs.

Why This Problem Matters: Economics Meets Performance

Understanding the distributional gap is important for three interlocking reasons—one economic, one empirical, and one conceptual.

Economic: On-policy training is expensive. True on-policy RLHF requires generating outputs from the current policy during training, scoring them with a reward model, and updating the policy—all in a tight loop. This is what PPO (Schulman et al., 2017) does, and while PPO remains a gold standard for alignment quality, its computational cost is prohibitive for many practitioners. Off-policy methods like DPO eliminate the online sampling step entirely, making them attractive for cost-sensitive or resource-constrained settings. If we can close the performance gap between off-policy and on-policy training without adding back the computational cost of online sampling, we get the best of both worlds: on-policy-quality alignment at off-policy cost. This is exactly what WPO aims to do.

Empirical: Off-policy DPO consistently underperforms on-policy alternatives. The paper does not just assert that the distributional gap exists—prior work (Tang et al., 2024a; Xu et al., 2024; Tajwar et al., 2024) has established empirically that on-policy preference optimization outperforms off-policy methods, and the paper's own experiments (Figure 3) confirm this across both Mistral-7B and Llama-3-8B-Instruct. The gap is real, measurable, and reproducible. Yet the field lacks a principled method for mitigating it without reverting to the expense of online sampling. This is the gap that WPO fills.

Conceptual: This is a classic off-policy RL problem, now manifesting in LLM alignment. The distributional gap that WPO addresses is not new to machine learning—it's the central challenge of off-policy reinforcement learning, well-documented in the RL literature (Fujimoto et al., 2019; Kumar et al., 2019, 2020). In standard RL, off-policy methods suffer from distributional shift between the behavior policy (which collected the data) and the target policy (which is being optimized), leading to instability, overestimation bias, and inefficient learning. The paper explicitly draws this connection (Section 1), citing these RL references and positioning WPO as a domain-specific solution for the LLM alignment context, where the "policy" is the language model's output distribution and the "data" is preference pairs. What makes the LLM setting distinct—and what justifies a specialized solution rather than off-the-shelf RL methods—is that the "actions" are sequences of tokens (high-dimensional and structured), the "reward" is implicit in the preference pairs (no scalar reward function is learned), and the optimization is done through a direct preference objective rather than through Q-functions or policy gradients.

Where Prior Approaches Fall Short

The paper situates its contribution against several bodies of prior work, each of which addresses parts of the off-policy problem but leaves gaps that WPO fills.

Standard DPO assumes uniform data relevance. DPO (Rafailov et al., 2023) revolutionized LLM alignment by eliminating the need to train a separate reward model. It reparameterizes the Bradley-Terry preference model so that the policy itself encodes the reward, producing a simple maximum-likelihood objective on preference pairs. But DPO's loss function is an expectation over a uniform sampling of the preference dataset—every pair in the batch contributes equally to the loss. Section 3.2 makes this critique explicit. The problem is not with DPO's mathematical derivation from the Bradley-Terry model; that derivation is correct assuming the preference data represents the true preference distribution. The problem is that in off-policy settings, the preference data was generated by a different policy, so uniform weighting over that dataset does not correspond to the true preference distribution under the current policy. DPO has no mechanism to account for this mismatch.

DPO variants that improve the loss function don't address the data distribution. The paper compares WPO against several DPO variants: ORPO (Hong et al., 2024), which combines SFT and preference optimization into a single objective; KTO (Ethayarajh et al., 2024), which works with unpaired preference data using prospect theory; and SimPO (Meng et al., 2024), which uses a reference-free reward with length normalization. These methods improve how the preference signal is converted into a loss, but they still treat all preference pairs uniformly. The distributional gap—the mismatch between the data-generating policy and the target policy—remains unaddressed. WPO is orthogonal to these innovations; in fact, the paper shows (Table 3) that WPO's reweighting can be applied on top of IPO, SimPO, and KTO, yielding consistent improvements. This is a strong signal that the distributional gap is a distinct problem from loss function design.

On-policy methods are expensive. The paper acknowledges that on-policy approaches exist and work well. Self-Play Fine-Tuning (SPIN; Chen et al., 2024) iteratively generates outputs from the current policy and pairs them with human-labeled winners. Iterative DPO (Xu et al., 2023) retrains the policy on freshly sampled data at each iteration. Direct Nash Optimization (DNO; Rosset et al., 2024) uses on-policy sampling to estimate a preference gap. Adversarial Preference Optimization (Cheng et al., 2023) incorporates contrastive losses that compare the policy's outputs against a reference. SAMI (Fränken et al., 2024) optimizes a mutual information bound using contrastive estimation. All of these methods produce high-quality alignment because the training data faithfully reflects the current policy's output distribution. But they all require generating outputs from the policy during training—either once per iteration or continuously. This online sampling cost is what WPO aims to avoid. The paper's conceptual innovation is to ask: can we get the benefits of on-policy data relevance without actually generating on-policy data?

The hybrid approach is promising but incomplete. Rosset et al. (2024) showed that mixing high-quality off-policy outputs (e.g., from GPT-4) with on-policy outputs can outperform either approach alone. The paper's own experiments (Section 4.3, Figure 3) confirm this: the hybrid setting (off-policy Ultrafeedback + on-policy sampled outputs) yields the best results for both Mistral and Llama-3. This makes intuitive sense—off-policy data can introduce high-quality examples the policy hasn't "discovered" on its own, while on-policy data keeps the training grounded in the policy's actual output distribution. But hybrid methods don't eliminate the distributional gap within the off-policy portion of the data. WPO can be applied on top of hybrid data mixtures (as Table 1 shows) to further improve the utilization of the off-policy component.

No prior method reweights off-policy data to simulate on-policy behavior at zero cost. This is the paper's key claim to novelty. Before WPO, the only way to get preference data that reflected the current policy's distribution was to sample from the current policy—which costs compute. The idea of reweighting existing off-policy data according to policy probabilities such that the weighted dataset approximates the on-policy distribution is, to the authors' knowledge, previously unexplored in the LLM alignment context. This is a form of importance sampling—a well-known technique in statistics and RL—but adapted to the specifics of the DPO loss and the structure of language model outputs (token sequences with length normalization to prevent vanishing weights).

How the Paper Positions Itself

The paper frames WPO as a bridge between the efficiency of off-policy training and the effectiveness of on-policy training. The core conceptual move is described in Section 3.2 through a thought experiment:

Take the existing off-policy preference dataset and convert it into a preference labeling function—a rule that, given any pair of outputs, can tell you which is preferred (if either), based on whether that pair appears in the original dataset.
Conceptually resample a new preference dataset by drawing prompts from the original dataset, generating fresh output pairs from the current policy, and keeping only those pairs that the labeling function can label.
By the law of large numbers, if you did this infinitely many times, the relative frequency of each preference pair in the new dataset would be proportional to the joint probability of its outputs under the current policy—i.e., $\pi_\theta(y_w|x) \cdot \pi_\theta(y_l|x)$ .
In practice, you never actually do this resampling—instead, you achieve the same effect by multiplying each pair's contribution to the DPO loss by $w(x, y_w) \cdot w(x, y_l)$ , where $w(x, y) = \pi_\theta(y|x)$ (length-normalized).

This is the WPO objective (Equation 1):

$L_{\text{WPO}} = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ w(x, y_w) w(x, y_l) \log p(y_w \succ y_l \mid x) \right]$

The paper explicitly positions this as a zero-cost improvement over DPO: the only additional computation is evaluating the current policy's probabilities on the existing preference pairs (which is already done during DPO's forward pass), and those probabilities are detached from the gradient so they don't influence the policy through any channel other than the weighting.

The paper also positions itself relative to a broader trend in alignment research: the tension between cost and quality. It acknowledges that WPO does not fully close the gap between off-policy and on-policy performance (Section 5, Limitations), and that on-policy data—especially on-policy dispreferred outputs (Section 4.3, Figure 4)—remains valuable. But it argues that WPO significantly narrows the gap and does so without any computational overhead beyond standard DPO training. This makes it particularly suitable for resource-constrained settings, for rapid experimentation, and as a drop-in enhancement to existing DPO pipelines.

A subtle but important aspect of the positioning: the paper does not claim that WPO makes off-policy training better than on-policy training in any absolute sense. The hybrid results (Table 1) show that adding on-policy data on top of WPO further improves performance, and the on-policy vs. off-policy comparison (Figure 3) shows that the choice depends on the base model's quality—with Llama-3-Instruct, on-policy wins; with Mistral-7B, off-policy actually slightly edges out on-policy, likely because the Mistral-7B SFT model's sampled outputs are lower quality and would steer training toward suboptimal behavior. This nuance is important: WPO is not a replacement for on-policy data but a way to get more value out of the off-policy data you already have.

The Two Gaps: Distributional Mismatch and Confidence Variability

The paper identifies not just one but two related problems that WPO addresses. The first is the distributional gap proper: off-policy data doesn't reflect the current policy's output distribution. The second is more subtle: weight variability due to input-dependent model confidence. Even if you correctly weight preference pairs by their probability under the current policy, language models exhibit varying confidence levels across different inputs (Si et al., 2023; Xiong et al., 2024). Some prompts naturally elicit high-confidence, high-probability outputs, while others produce diffuse output distributions where even "good" responses have low probability. If you naively weight by raw sequence probability, preference pairs on low-confidence inputs will be systematically downweighted compared to pairs on high-confidence inputs—even if both pairs are equally "on-policy" in terms of being generated by the current model.

This is the motivation for the weight alignment mechanisms described in Section 3.3. The paper shows (Figure 2) that without alignment, the weight distribution of outputs sampled from the policy model itself is highly variable—some "on-policy" outputs receive weights far below 1.0. The goal of weight alignment is to calibrate the weights so that on-policy outputs (where the model's behavior is "as expected" regardless of absolute confidence) receive weights clustered around 1.0, while outputs that genuinely deviate from on-policy behavior receive lower weights. The two proposed alignment methods—greedy alignment and sampled alignment—normalize token-level probabilities by reference values (the max token probability for greedy alignment, the expected probability of a randomly sampled token for sampled alignment) so that the weight reflects relative likelihood rather than absolute probability. This is a domain-specific refinement that goes beyond standard importance sampling and reflects the particular challenges of working with sequence-level probabilities in language models.

3. Technical Approach

3.1 Reader Orientation

This paper is a methodological contribution that introduces a weighting scheme for preference optimization—it does not build a new system from scratch but rather modifies the training objective of Direct Preference Optimization (DPO) to make off-policy preference data behave more like on-policy data. The core idea is elegantly simple: during DPO training, multiply each preference pair's contribution to the loss by the probability that the current policy model would generate both the preferred and dispreferred outputs, thereby upweighting pairs that are "representative" of the current model's behavior and downweighting pairs that are irrelevant or improbable under the current policy.

3.2 Big-Picture Architecture (Diagram in Words)

The WPO framework has five conceptual components that operate within a standard DPO training loop:

Preference Dataset (off-policy): A static collection of $(x, y_w, y_l)$ triples—prompts with paired preferred and dispreferred responses—sampled from models other than the one being trained (e.g., GPT-4, Llama-2). This is the input data that would normally feed DPO directly.
Current Policy Model ( $\pi_\theta$ ): The language model being fine-tuned, initialized from an SFT checkpoint. During training, it serves two distinct roles: (a) it is the model being optimized (its parameters $\theta$ are updated via gradient descent), and (b) it is the source of the probability estimates used to compute the weights for each preference pair (these probabilities are detached from the gradient, so they act as fixed scaling factors during the backward pass).
Reference Model ( $\pi_{\text{ref}}$ ): A frozen copy of the SFT model, used in the DPO loss's implicit reward formulation to prevent the policy from drifting too far from its starting point. This is standard DPO infrastructure, unchanged by WPO.
Weight Computation Module: For each preference pair $(x, y_w, y_l)$ in a training batch, this module computes $w(x, y_w)$ and $w(x, y_l)$ —the length-normalized sequence probabilities of $y_w$ and $y_l$ under the current policy $\pi_\theta$ , with an optional calibration step (weight alignment) that normalizes token-level probabilities by reference values to correct for input-dependent confidence variability. These weights are computed during the forward pass and detached before backpropagation.
WPO Loss Function: The weighted version of the DPO loss, where each pair's log-probability of the preference is multiplied by $w(x, y_w) \cdot w(x, y_l)$ . This single scalar loss is backpropagated through $\pi_\theta$ to update the policy.

Information flows as follows: a batch of $(x, y_w, y_l)$ triples is sampled uniformly from the off-policy preference dataset → the current policy model computes $\log \pi_\theta(y_w|x)$ and $\log \pi_\theta(y_l|x)$ token-by-token → these log-probabilities are length-normalized and exponentiated to produce scalar weights $w(x, y_w)$ and $w(x, y_l)$ → the weights are multiplied together to form the pair weight → this pair weight multiplies the standard DPO per-pair loss → the weighted loss is summed over the batch and backpropagated to update $\theta$ .

3.3 Roadmap for the Deep Dive

First, the standard DPO objective and its relationship to the Bradley-Terry preference model, since WPO is a direct modification of DPO and understanding what DPO computes is prerequisite to understanding what WPO changes.
Second, the formal definition of the distributional gap—why uniform weighting over off-policy data is suboptimal—and the thought experiment of bootstrapping an on-policy dataset from off-policy data, since this thought experiment is the conceptual justification for the entire method.
Third, the WPO objective itself: the mathematical form, the definitions of the weights, the length normalization that prevents vanishing weights on long sequences, and the gradient detachment that prevents the weights from influencing the policy through any channel other than the pair weighting.
Fourth, the weight alignment mechanisms (greedy alignment and sampled alignment), which calibrate the raw sequence probabilities to account for input-dependent model confidence—a subtle but important refinement that ensures on-policy outputs receive weights near 1.0 regardless of the absolute confidence level.
Fifth, how WPO integrates with non-DPO loss functions (IPO, SimPO, KTO), since the weighting scheme is loss-agnostic and the paper demonstrates universal improvement across loss families.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological paper whose core idea is that multiplying each preference pair's contribution to the DPO loss by the joint probability of its outputs under the current policy simulates on-policy data relevance in an off-policy training loop.

Standard DPO and the Bradley-Terry Preference Model

DPO operates within the Bradley-Terry (BT) framework for modeling pairwise preferences. The BT model assumes that the probability of preferring output $y_w$ over $y_l$ given prompt $x$ is determined by an underlying scalar reward function $r^*(x, y)$ that captures how "good" the output is:

$p^*(y_w \succ y_l \mid x) = \frac{\exp(r^*(x, y_w))}{\exp(r^*(x, y_w)) + \exp(r^*(x, y_l))} = \sigma(r^*(x, y_w) - r^*(x, y_l))$

where $\sigma(\cdot)$ is the sigmoid function, $r^*(x, y)$ is the latent (unknown) reward that determines human preferences, $y_w$ is the preferred output, and $y_l$ is the dispreferred output.

What it computes: the probability that a human annotator would prefer $y_w$ over $y_l$ , expressed as the sigmoid of the reward difference between the two outputs. When the rewards are equal ( $r^*(x, y_w) = r^*(x, y_l)$ ), the preference probability is 0.5 (no systematic preference). When $y_w$ has a much higher reward, the probability approaches 1.0.

Why this form: the Bradley-Terry model is the standard maximum-entropy model for pairwise comparisons—it makes the least additional assumptions about the preference-generating process beyond the existence of a scalar reward. The exponential transformation ensures that rewards are always positive when converted to preference strengths, and the ratio formulation means that only relative (not absolute) reward magnitudes matter.

DPO's key insight is to reparameterize the reward function in terms of the policy itself. Given the optimal policy $\pi^*$ for a KL-constrained RLHF objective, the corresponding optimal reward has the closed form:

$r^*(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$

where $\beta$ is the KL penalty coefficient controlling how far $\pi^*$ can deviate from the reference model $\pi_{\text{ref}}$ , and $Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp(r^*(x,y)/\beta)$ is the partition function that normalizes the distribution.

What it computes: the reward $r^*(x, y)$ is expressed as a scaled log-ratio of the policy probability to the reference probability, plus a prompt-dependent constant. The log-ratio term captures how much more (or less) likely $\pi^*$ is to produce $y$ compared to $\pi_{\text{ref}}$ ; a positive log-ratio means $\pi^*$ upweights $y$ relative to $\pi_{\text{ref}}$ , implying higher reward.

Why this form: this reparameterization is exact for the optimal policy under the KL-constrained RLHF objective—it is not an approximation. The partition function $Z(x)$ is the same for all outputs $y$ given the same prompt $x$ , so it cancels when computing reward differences between two outputs for the same prompt. This cancellation is what allows DPO to avoid learning a separate reward model: the preference probability can be expressed purely in terms of the policy and reference log-probabilities.

Substituting the reward reparameterization into the BT model and canceling $Z(x)$ (since it appears identically in both numerator and denominator of the sigmoid argument), the preference probability becomes:

$p^*(y_w \succ y_l \mid x) = \sigma\left(\beta \log \frac{\pi^*(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi^*(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$

DPO then trains the policy $\pi_\theta$ (parameterized by $\theta$ ) to maximize the likelihood of the observed preferences under this model. The DPO loss is the negative log-likelihood, averaged uniformly over the preference dataset $\mathcal{D}$ :

$L_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right) \right]$

where the expectation is over a uniform draw from the dataset—every preference pair in $\mathcal{D}$ contributes equally to the expected loss.

What it computes: the average negative log-likelihood of the observed preferences under the policy-parameterized BT model. For each pair $(x, y_w, y_l)$ , the model computes the implicit reward difference $\beta[\log(\pi_\theta(y_w)/\pi_{\text{ref}}(y_w)) - \log(\pi_\theta(y_l)/\pi_{\text{ref}}(y_l))]$ , passes it through the sigmoid to get a preference probability, and penalizes the model if this probability is far from 1.0 (since we know $y_w$ is preferred).

Why this form: maximum likelihood estimation is the standard approach for fitting probabilistic models to data. The uniform expectation encodes the assumption that all preference pairs in $\mathcal{D}$ are equally informative about the true preference distribution—an assumption that WPO will explicitly challenge.

The gradient of the DPO loss reveals its operational behavior:

$\nabla L_{\text{DPO}} = -\beta \cdot \mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ \sigma(\hat{r}(x, y_l) - \hat{r}(x, y_w)) \cdot \left(\nabla \log \pi_\theta(y_w \mid x) - \nabla \log \pi_\theta(y_l \mid x)\right) \right]$

where $\hat{r}(x,y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ is the implicit reward estimate.

What it computes: the gradient pushes the policy to increase the log-probability of $y_w$ (the preferred output) and decrease the log-probability of $y_l$ (the dispreferred output). The sigmoid factor $\sigma(\hat{r}(y_l) - \hat{r}(y_w))$ acts as an adaptive learning rate: when the model already strongly prefers $y_w$ over $y_l$ (i.e., $\hat{r}(y_w) \gg \hat{r}(y_l)$ , so $\hat{r}(y_l) - \hat{r}(y_w) \ll 0$ , so $\sigma(\cdot) \approx 0$ ), the gradient is near zero because the model already "gets it right." When the model is confused ( $\hat{r}(y_w) \approx \hat{r}(y_l)$ , so $\sigma(\cdot) \approx 0.5$ ), the gradient is largest, driving a strong correction. This same gradient structure will be preserved in WPO, but multiplied by the pair weight.

Why this form: the sigmoid weighting provides an automatic curriculum: easy pairs (where the reward gap is already large) contribute little gradient, allowing the model to focus on harder pairs. This is a desirable property of the cross-entropy loss on the Bradley-Terry model—it naturally emphasizes examples near the decision boundary.

The critical limitation of this objective—the one WPO addresses—is that the uniform expectation $\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}}$ treats all preference pairs as equally relevant to the current policy, regardless of how probable those outputs are under $\pi_\theta$ .

The Distributional Gap: Why Uniform Weighting Fails

The distributional gap arises because the preference dataset $\mathcal{D}$ was generated by a different policy (or mixture of policies) than the one being trained. Formally, the data-generating distribution for outputs is some $p_{\text{data}}(y \mid x)$ , while the current policy is $\pi_\theta(y \mid x)$ . When $p_{\text{data}} \neq \pi_\theta$ , the pairs in $\mathcal{D}$ are not representative of what $\pi_\theta$ would produce if it were generating outputs on-policy.

Consider two extreme preference pairs, both present in the off-policy dataset:

Pair A: $(x, y_w, y_l)$ where both $y_w$ and $y_l$ have high probability under $\pi_\theta$ . These are outputs the current model would plausibly generate. Training on this pair directly improves the model's behavior on its own likely outputs.
Pair B: $(x, y_w, y_l)$ where both $y_w$ and $y_l$ have extremely low probability under $\pi_\theta$ (e.g., they use vocabulary, formatting, or reasoning styles that the current model never employs). Training on this pair teaches the model to rank outputs it would never produce—it's essentially wasted optimization effort from the perspective of improving the model's actual output quality.

Standard DPO gives Pair A and Pair B equal weight in the loss. If Pair B is common in the off-policy dataset (which it often is, since the data comes from different models), a significant fraction of the optimization budget is spent on irrelevant comparisons.

The paper illustrates this with a concrete hypothetical in Section 3.2:

"Consider two instances of preference data: (x^{(1)}, y^{(1)}_w, y^{(1)}_l) and (x^{(2)}, y^{(2)}_w, y^{(2)}_l), where the first tuple is sampled directly from the current policy model, while the second tuple is sampled from a different distribution from the current policy model. Despite this difference in sampling probability, DPO treats both instances equally in its loss calculation."

The consequence is inefficient learning: the model's gradient steps are diluted by low-relevance pairs, and the effective sample size of "useful" training examples is smaller than the dataset size suggests.

The Bootstrapping Thought Experiment: Conceptual Justification for WPO

WPO's conceptual foundation is a thought experiment that describes how one would construct a truly on-policy preference dataset from off-policy data if computational cost were no object. The thought experiment proceeds in four steps:

Step 1: Convert the off-policy dataset into a labeling function. Define a function $f(x, y_1, y_2)$ that takes any triple of (prompt, output1, output2) and returns a preference label if that exact triple (or its reverse) exists in the original dataset:

$f(x, y_1, y_2) = \begin{cases} y_1 \succ y_2, & (x, y_1, y_2) \in \mathcal{D} \\ y_2 \succ y_1, & (x, y_2, y_1) \in \mathcal{D} \\ \text{NA}, & \text{otherwise} \end{cases}$

The paper assumes $\mathcal{D}$ contains no conflicting preferences—if $(x, y_1, y_2)$ is in the dataset, then $(x, y_2, y_1)$ is not. This is a reasonable assumption for curated preference datasets.

Step 2: Bootstrap a new dataset by on-policy sampling with rejection. For an infinite number of iterations: uniformly sample a prompt $x$ from the original dataset, then sample two outputs $y_1$ and $y_2$ from the current policy $\pi_\theta$ , and check whether $f(x, y_1, y_2)$ returns a label (i.e., whether the pair is in the original dataset or its reverse). If it does, keep the labeled pair; if it returns NA, reject and discard the sample.

Step 3: Apply the law of large numbers. After infinitely many iterations, the relative frequency with which any particular pair $(x, y_w, y_l)$ appears in the resampled dataset is proportional to:

$\pi_\theta(y_w \mid x) \cdot \pi_\theta(y_l \mid x) \cdot p(x)$

where $p(x)$ is the prompt distribution (uniform over the original dataset's prompts). The first two factors are the probabilities of sampling $y_w$ and $y_l$ independently from $\pi_\theta$ ; the rejection step simply discards pairs not in the labeling function's domain, which doesn't change the relative frequencies among pairs that are retained.

Step 4: Recognize that this is equivalent to reweighting the original dataset. Rather than actually performing this resampling (which would be computationally prohibitive—it requires generating outputs from $\pi_\theta$ at every training step, defeating the purpose of off-policy training), we can achieve the same expected loss by keeping the original dataset but reweighting each pair by the joint probability of its outputs under $\pi_\theta$ . The expectation transformation is:

$\mathbb{E}_{(x, y_w, y_l) \sim \text{resampled}}[\cdot] \propto \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[ \pi_\theta(y_w \mid x) \cdot \pi_\theta(y_l \mid x) \cdot (\cdot) \right]$

This is the core insight: multiplying each pair's loss contribution by $w(x, y_w) \cdot w(x, y_l)$ (where $w(x, y) = \pi_\theta(y \mid x)$ or a normalized variant) makes the off-policy dataset behave, in expectation, like it was resampled from the current policy.

Why this form: the bootstrapping thought experiment is an application of importance sampling, a standard technique in statistics and reinforcement learning where samples from one distribution are reweighted to estimate expectations under another distribution. The importance weight $\frac{\pi_\theta(y_w|x) \cdot \pi_\theta(y_l|x)}{p_{\text{data}}(y_w, y_l|x)}$ would be the standard importance sampling ratio, but since the data-generating distribution $p_{\text{data}}$ is unknown and the pairs were not drawn i.i.d. from a single behavior policy, a precise importance weight is unavailable. The WPO approach effectively uses an unnormalized weight proportional to $\pi_\theta(y_w|x) \cdot \pi_\theta(y_l|x)$ , which captures the relative "on-policy-ness" of different pairs even without knowing the absolute data-generating probabilities.

The WPO Objective: Mathematical Form and Operational Mechanics

The WPO loss function is a direct modification of the DPO loss where each pair's log-likelihood term is multiplied by the product of the (length-normalized) policy probabilities of its outputs:

$L_{\text{WPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ w(x, y_w) \cdot w(x, y_l) \cdot \log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right) \right]$

where $w(x, y)$ is the weight for a single output $y$ given prompt $x$ , and both $w(x, y_w)$ and $w(x, y_l)$ are detached from the computation graph—they are treated as constants during backpropagation, so the gradient does not flow through them to update $\theta$ . The only path for gradient flow to $\theta$ is through the log-probabilities $\log \pi_\theta(y_w|x)$ and $\log \pi_\theta(y_l|x)$ that appear inside the sigmoid argument (the same as in standard DPO).

What it computes: for each preference pair, the loss computes the standard DPO per-pair loss (negative log-likelihood of the preference under the policy-parameterized BT model) and then scales it by a factor $w(x, y_w) \cdot w(x, y_l)$ that reflects how probable both outputs are under the current policy. Pairs with highly probable outputs receive large weights and dominate the loss; pairs with improbable outputs receive small weights and barely influence the gradient.

Why this form: the multiplicative scaling is a direct consequence of the bootstrapping thought experiment. Each factor $w(x, y)$ represents the probability of sampling that output from $\pi_\theta$ , and since the two outputs are sampled independently in the thought experiment, their joint probability is the product. The gradient detachment is critical: if the weights received gradients, the optimization would have an incentive to artificially inflate the probabilities of all outputs to increase the weights (a degenerate solution), rather than to learn the correct preference ordering. By detaching, the weights serve purely as data-selection factors, not as trainable quantities.

The weight definition and length normalization. For a sequence output $y = (y_1, y_2, \ldots, y_{|y|})$ , the raw sequence probability under $\pi_\theta$ is the product of conditional token probabilities:

$\pi_\theta(y \mid x) = \prod_{t=1}^{|y|} \pi_\theta(y_t \mid x, y_{<t})$

In practice, this product becomes extremely small for sequences of typical length (each token probability is <1, so multiplying hundreds of them yields values like $10^{-50}$ or smaller). Additionally, the product decays exponentially with sequence length, so longer sequences have systematically lower probabilities than shorter ones, regardless of content quality.

To address both issues, WPO uses length-normalized sequence probability as the weight:

$w(x, y) = \exp\left( \frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t}) \right)$

where $|y|$ is the number of tokens in output $y$ .

What it computes: the exponential of the average per-token log-probability. Equivalently, it is the geometric mean of the per-token probabilities: $w(x, y) = \left(\prod_{t=1}^{|y|} \pi_\theta(y_t|x, y_{<t})\right)^{1/|y|}$ . This normalization makes the weight independent of sequence length (in expectation, sequences of different lengths but similar per-token quality receive similar weights) and keeps the numerical values in a reasonable range (typically between 0.01 and 1.0 for well-trained models).

Why this form: length normalization is necessary because the raw sequence probability would otherwise make weights vanish for all but the shortest outputs, destroying the signal. The geometric mean preserves the per-token average quality signal while removing the length artifact. This is a standard technique in language modeling (e.g., perplexity is computed as the exponential of the average negative log-likelihood per token), adapted here for importance weighting. An arithmetic mean of probabilities would also work but would be less natural probabilistically—the geometric mean corresponds to the average information-theoretic "surprise" per token.

Gradient structure of WPO. The gradient of WPO with respect to the policy parameters $\theta$ is:

$\nabla L_{\text{WPO}} = -\beta \cdot \mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ w(x, y_w) \cdot w(x, y_l) \cdot \sigma(\hat{r}(x, y_l) - \hat{r}(x, y_w)) \cdot \left(\nabla \log \pi_\theta(y_w \mid x) - \nabla \log \pi_\theta(y_l \mid x)\right) \right]$

where the weights $w(x, y_w)$ and $w(x, y_l)$ are constants (detached) and $\hat{r}(x,y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ remains the implicit reward.

What it computes: the same gradient as DPO—increase $\log \pi_\theta(y_w|x)$ , decrease $\log \pi_\theta(y_l|x)$ , with the sigmoid factor as an adaptive learning rate—but now each pair's contribution is scaled by the product of the output weights. Pairs with high $w(x, y_w) \cdot w(x, y_l)$ receive the full gradient magnitude; pairs with low weights are effectively ignored.

Why this form: the gradient structure preserves all the desirable properties of DPO (the adaptive sigmoid scaling, the direct preference signal without a separate reward model) while adding the pair-weighting mechanism. The weights act as per-example gradient multipliers that implement a form of soft data selection: rather than hard-filtering the dataset to keep only "on-policy" pairs (which would require a threshold and might discard useful information), WPO smoothly varies the contribution of each pair, giving maximum influence to the most representative pairs while still allowing low-weight pairs to contribute a small amount of gradient.

Practical implementation in the training loop. Algorithm 1 in the paper specifies the procedure:

Sample a mini-batch of tuples $(x, y_w, y_l)$ from the off-policy dataset $\mathcal{D}$ .
For each tuple, compute the length-normalized weights $w(x, y_w)$ and $w(x, y_l)$ using the current policy $\pi_\theta$ (forward pass, probabilities not used for gradient).
Optionally apply weight alignment (Section 3.3) to calibrate the weights.
Compute the WPO loss $L_{\text{WPO}}$ using Equation (1), where the weights are detached constants.
Update $\theta$ using gradient descent: $\theta \leftarrow \theta - \alpha_t \nabla_\theta L_{\text{WPO}}$ .

The computational overhead compared to DPO is minimal: step 2 already requires computing $\log \pi_\theta(y_w|x)$ and $\log \pi_\theta(y_l|x)$ , which is exactly what DPO computes for the loss anyway. The only additional operations are the length normalization (dividing by $|y|$ , exponentiating) and the multiplication of the weights—all negligible compared to the cost of the forward and backward passes through the language model.

Weight Alignment: Calibrating Weights for Input-Dependent Confidence

The raw length-normalized weights $w(x, y) = \exp\left(\frac{1}{|y|}\sum_t \log \pi_\theta(y_t|x, y_{<t})\right)$ have a subtle problem: language models exhibit different levels of confidence across different inputs. For some prompts, the model's output distribution is sharply peaked (high probability on a single token at each position), producing high weights even for mediocre outputs. For other prompts, the distribution is diffuse (probability spread across many plausible tokens), producing low weights even for excellent outputs.

This matters for WPO's goal of simulating on-policy RL. In true on-policy RL, all outputs generated by the current policy should receive equal weight in the preference optimization—they are all "on-policy" by definition. But with raw weights, an on-policy output on a difficult/ambiguous prompt might receive a weight of 0.1, while an on-policy output on an easy prompt receives a weight of 0.9. This introduces an unintended bias where the model's confidence (which is input-dependent) confounds the "on-policy-ness" signal.

The paper demonstrates this problem empirically in Figure 2: when sampling outputs from the policy model itself (Mistral-sft-beta on Ultrafeedback prompts) and plotting their raw weight distribution, the weights span a wide range (roughly 0.2 to 1.2) rather than clustering near 1.0 as desired.

The solution is weight alignment: calibrate the weights so that outputs generated by the current policy receive weights near 1.0 on average, regardless of the input-dependent confidence level. The paper proposes two alignment methods, both operating at the token level.

Greedy alignment. This method normalizes each token's probability by the maximum token probability at that position—the probability of the token the model considers most likely:

$w(x, y) = \exp\left( \frac{1}{|y|} \sum_{t=1}^{|y|} \log \frac{\pi_\theta(y_t \mid x, y_{<t})}{\max_{v \in \mathcal{V}} \pi_\theta(v \mid x, y_{<t})} \right)$

where $\mathcal{V}$ is the vocabulary (the set of all possible tokens), and $\max_{v \in \mathcal{V}} \pi_\theta(v \mid x, y_{<t})$ is the probability of the single most likely token at position $t$ given the prefix.

What it computes: for each token, divide its actual probability by the probability of the greedy-first token (the token the model would pick in greedy decoding). Then take the geometric mean across the sequence. If the output was generated by greedy decoding, every token's probability equals the maximum, so every ratio is 1.0, and the weight is exactly $\exp(0) = 1.0$ . If the output deviates from greedy choices, the ratios are <1.0, and the weight drops below 1.0—but crucially, the normalization by the per-position maximum removes the input-dependent overall confidence level, since both numerator and denominator are drawn from the same distribution at the same position.

Why this form: the max-token normalization calibrates against the "best possible" token at each position. This is a natural reference point because the greedy token represents the model's most confident prediction—any deviation from it represents a deliberate choice to sample a lower-probability token, which should reduce the weight. The per-position normalization ensures that prompts where the model is uniformly confident (high max probabilities everywhere) and prompts where the model is uniformly uncertain (low max probabilities everywhere) produce comparable weights, since in both cases an output that exactly follows greedy decoding gets weight 1.0.

Sampled alignment (default method). This method normalizes each token's probability by the expected probability of a randomly sampled token from the model's distribution at that position:

$w(x, y) = \exp\left( \frac{1}{|y|} \sum_{t=1}^{|y|} \log \frac{\pi_\theta(y_t \mid x, y_{<t})}{\sum_{v \in \mathcal{V}} \pi_\theta(v \mid x, y_{<t})^2} \right)$

where $\sum_{v \in \mathcal{V}} \pi_\theta(v \mid x, y_{<t})^2$ is the expected probability of a token sampled from the model's distribution (i.e., if you sample a token $v$ from $\pi_\theta(\cdot|x, y_{<t})$ , its own probability $\pi_\theta(v|x, y_{<t})$ is a random variable whose expectation is $\sum_v \pi_\theta(v)^2$ , by the definition of expectation of a function of a random variable).

What it computes: for each token, divide its actual probability by the expected probability of a token drawn from the model's own distribution at that position. If the output token was sampled from $\pi_\theta$ (as it would be in on-policy generation with temperature 1.0), the expected value of the ratio numerator/denominator is 1.0—the weight is calibrated so that randomly sampled outputs have anticipated weight near 1.0. Outputs that are more probable than a typical random sample get weights >1.0; outputs that are less probable get weights <1.0.

Why this form: the denominator $\sum_v \pi_\theta(v)^2$ is a measure of distribution concentration known as the Simpson diversity index or the collision probability—it is high (near 1.0) when the distribution is sharply peaked and low (near 0) when the distribution is uniform over many tokens. Normalizing by this value adjusts for the fact that a probability of 0.1 means something very different on a peaked distribution (where it's relatively low) versus a flat distribution (where it might be the highest any token gets). This is the paper's default alignment method because it outperforms greedy alignment empirically (Table 2) and produces a more concentrated weight distribution for on-policy outputs (Figure 2, green curve).

Why weight alignment is necessary for the simulation to work. Without alignment, WPO's weights confound two distinct sources of variation: (a) whether the output is genuinely representative of the current policy's behavior (the signal we want), and (b) the model's baseline confidence level on the given prompt (a nuisance variable). Alignment removes (b) by normalizing against a position-specific reference, so the remaining weight variation reflects only (a). This makes the weighted dataset a better approximation of true on-policy resampling, where all on-policy outputs would receive weight 1.0 regardless of prompt difficulty.

Empirical validation of alignment methods. Table 2 shows the results of an ablation study on Mistral-7B in the off-policy setting:

WPO with sampled alignment: 24.4% length-controlled win rate on Alpaca Eval 2, 23.7% win rate vs GPT-4, 60.1% pairwise win rate vs DPO on MT-bench.
WPO with greedy alignment: 23.0% LC, 21.4% win rate, 57.9% pairwise.
WPO without alignment: 22.0% LC, 20.3% win rate, 54.4% pairwise.
DPO baseline: 20.6% LC, 18.6% win rate, 50% pairwise (reference).

The ranking (sampled > greedy > no alignment > DPO) matches the concentration of the weight distributions in Figure 2, supporting the claim that better alignment of weights to the ideal on-policy distribution (all weights = 1.0 for on-policy outputs) leads to better downstream performance.

WPO with Non-DPO Loss Functions

A key design choice in WPO is that the weighting scheme is independent of the loss function. The paper demonstrates this by applying WPO-style reweighting to three preference optimization objectives beyond DPO: IPO, SimPO, and KTO.

IPO (Identity Preference Optimization; Azar et al., 2024). IPO uses a squared-loss formulation rather than DPO's sigmoid cross-entropy. The standard IPO loss for a preference pair is:

$L_{\text{IPO}} = \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} - \frac{1}{2} \right)^2$

What it computes: the squared difference between the implicit reward gap (policy-vs-reference log-ratio difference between preferred and dispreferred outputs) and a target value of $\frac{1}{2}$ . IPO aims to push the reward gap to exactly 0.5 rather than to infinity, which provides a bounded optimization target that can be more stable than DPO's sigmoid (which asymptotically flattens for large gaps).

WPO-IPO: the paper multiplies each pair's squared loss by $w(x, y_w) \cdot w(x, y_l)$ (with weight alignment and gradient detachment), exactly as in WPO-DPO:

$L_{\text{WPO-IPO}} = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ w(x, y_w) \cdot w(x, y_l) \cdot \left( \beta \log \frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} - \beta \log \frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)} - \frac{1}{2} \right)^2 \right]$

SimPO (Simple Preference Optimization; Meng et al., 2024). SimPO uses a reference-free reward formulation with length normalization built into the reward itself:

$L_{\text{SimPO}} = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w \mid x) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l \mid x) - \gamma \right) \right]$

where $\gamma$ is a margin hyperparameter enforcing a minimum reward gap. The key innovation is that the reward is simply the length-normalized log-probability of the output under $\pi_\theta$ , without any reference model.

WPO-SimPO: the paper multiplies each pair's loss by $w(x, y_w) \cdot w(x, y_l)$ , using the same length-normalized weights (with alignment) as for DPO:

$L_{\text{WPO-SimPO}} = -\mathbb{E}_{(x,y_w,y_l) \sim \mathcal{D}} \left[ w(x, y_w) \cdot w(x, y_l) \cdot \log \sigma\left( \frac{\beta}{|y_w|} \log \pi_\theta(y_w) - \frac{\beta}{|y_l|} \log \pi_\theta(y_l) - \gamma \right) \right]$

KTO (Kahneman-Tversky Optimization; Ethayarajh et al., 2024). KTO is fundamentally different from DPO, IPO, and SimPO in that it uses unpaired preference data—each output is labeled as "desirable" or "undesirable" independently, rather than being part of a preference pair. The KTO loss processes favored and disfavored outputs separately:

$L_{\text{KTO}} = \mathbb{E}_{(x,y) \sim \mathcal{D}_{\text{desired}}} \left[ 1 - \sigma\left( \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - z_{\text{ref}} \right) \right] + \mathbb{E}_{(x,y) \sim \mathcal{D}_{\text{undesired}}} \left[ \sigma\left( \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - z_{\text{ref}} \right) \right]$

where $z_{\text{ref}}$ is a reference point (a hyperparameter) and the two expectations are over separate sets of desired and undesired outputs (not paired).

WPO-KTO: since KTO uses unpaired data, the pair-weighting approach (multiply by $w(x, y_w) \cdot w(x, y_l)$ ) does not directly apply. Instead, the paper weights each output individually by $w(x, y)$ and normalizes the total weight of desired and undesired outputs separately within each batch:

$L_{\text{WPO-KTO}} = \mathbb{E}_{(x,y) \sim \mathcal{D}} \left[ \frac{w(x, y)}{\sum_{(x',y') \in B_{\text{desired}}} w(x', y')} \cdot \text{KTO-desired-loss}(x, y) \cdot \mathbb{1}[y \text{ is desired}] + \frac{w(x, y)}{\sum_{(x',y') \in B_{\text{undesired}}} w(x', y')} \cdot \text{KTO-undesired-loss}(x, y) \cdot \mathbb{1}[y \text{ is undesired}] \right]$

What it computes: within a batch, the total weight of desired outputs is normalized to sum to 1, and the total weight of undesired outputs is normalized to sum to 1 (separately). This ensures that neither category dominates the loss due to systematic weight differences, while still allowing more "on-policy" outputs within each category to receive higher relative weight than less on-policy outputs.

Why this form for KTO: without normalization, if desired outputs systematically have higher weights than undesired outputs (or vice versa), the overall loss would be biased toward one category. The separate normalization preserves the intra-category weighting signal (preferring on-policy over off-policy outputs within each category) while maintaining the inter-category balance that KTO's original loss assumes.

Results of applying WPO to alternative losses (Table 3). On Mistral-7B in the off-policy setting:

WPO-IPO: 29.4% LC win rate (up from IPO's 25.0%), 25.7% win rate vs GPT-4 (up from 21.2%), 54.1% pairwise MT-bench vs baseline (up from 50%).
WPO-SimPO: 21.9% LC (up from SimPO's 21.5%), 24.6% win rate (up from 21.4%), 52.5% pairwise (up from 50%).
WPO-KTO: 21.1% LC (up from KTO's 14.9%), 20.3% win rate (up from 12.3%), 60.0% pairwise (up from 50%).

Every combination of WPO + alternative loss outperforms the corresponding baseline loss. This is strong evidence that the distributional gap is a distinct problem from the choice of loss function, and that WPO's reweighting provides a complementary improvement. The fact that improvements are observed across loss functions with fundamentally different structures (sigmoid cross-entropy, squared loss, reference-free reward, unpaired prospect-theoretic loss) suggests the weighting mechanism is broadly applicable.

Design choice: detached weights. Across all loss variants, the weights $w(x, y)$ are detached from the gradient graph. If they were not detached, the optimizer would have a direct incentive to increase $\pi_\theta(y|x)$ for all outputs (both preferred and dispreferred) to increase the weights, which would work against the preference optimization's goal of widening the gap between preferred and dispreferred outputs. Detachment ensures the weights serve purely as data-relevance indicators, not as trainable quantities.

4. Key Insights and Innovations

Innovation 1: Reframing Off-Policy Preference Optimization as a Data Relevance Problem (Not a Loss Function Problem)

The dominant assumption in preference optimization, from DPO onward through its many variants, has been that the loss function is the primary lever for improving alignment quality. The field's energy has gone into devising better objectives: sigmoid cross-entropy (DPO), squared-loss (IPO), reference-free rewards with length normalization (SimPO), prospect-theoretic formulations (KTO), and more. Each of these innovations asks: "Given a fixed preference dataset, what is the best mathematical objective to extract the preference signal?"

WPO makes a fundamentally different move. It argues—conceptually and empirically—that the relevance of the training data to the current policy matters as much as, and independently from, how you process that data. The paper's key reframing is that off-policy preference optimization is not just a loss-function-design problem; it is also a data-relevance problem. In standard DPO, every preference pair enters the loss with equal weight, regardless of whether the pair represents outputs the current policy would plausibly generate or outputs from a distribution so different that optimizing on them is wasted effort. This is a categorical shift in how the problem is understood: it moves the source of suboptimality from the computation applied to the data (the loss) to the data itself (which pairs matter and how much).

What makes this insight intellectually distinctive is that it is not obvious from the mathematics of DPO. DPO's derivation from the Bradley-Terry model is mathematically correct under the assumption that the preference data faithfully represents the true preference distribution. The derivation does not encode any notion of "data-generating policy" versus "target policy." It took stepping back and asking a meta-question—"What implicit assumption about data relevance is baked into the uniform expectation over the dataset, and what happens when that assumption is violated?"—to identify the distributional gap as a distinct failure mode.

The empirical evidence that this is a distinct problem from loss-function design comes from Table 3: applying WPO's reweighting on top of IPO, SimPO, and KTO—losses with fundamentally different mathematical forms and assumptions—yields consistent improvements across all of them. If the distributional gap were addressable through better loss design, one of these losses would already capture the benefit. The fact that WPO helps additively on top of all of them strongly supports that data relevance and loss design are orthogonal axes of improvement.

This is a fundamental reframing rather than an incremental refinement. It opens a new axis of research: not just "what loss should we use on preference data?" but "how should we weight, select, or generate preference data to maximize its relevance to the current policy?" The paper's bootstrapping thought experiment (Section 3.2)—conceptually resampling an on-policy dataset from off-policy data via rejection sampling—is not just a derivation trick; it's a new way to think about what the ideal training data should look like and how to approximate it without the cost of online generation.

Innovation 2: Achieving On-Policy-Like Data Relevance at Zero Sampling Cost via Importance Weighting

Before WPO, the only way to get preference data that reflected the current policy's output distribution was to sample from the current policy—which meant running the model in inference mode during training, scoring outputs with a reward model, and constructing new pairs on the fly. This is what PPO, iterative DPO, SPIN, DNO, and other on-policy methods do. The computational cost of online sampling was widely considered the unavoidable price of data relevance. Off-policy methods (DPO on static datasets) traded away data relevance for cost efficiency; on-policy methods traded away cost efficiency for data relevance. The two were seen as mutually exclusive.

WPO breaks this assumed trade-off. It shows that you can reweight existing off-policy data using nothing more than the probabilities the policy already computes during its forward pass—no additional generation, no reward model queries, no online sampling loop—and achieve a training signal that approximates what you would get from actually resampling the data on-policy. This is a fundamental insight because it reveals that the information needed to distinguish on-policy-relevant pairs from off-policy-irrelevant pairs is already latent in the policy's own probability estimates. You don't need to generate new data to know which existing data is representative; you just need to look at how probable the existing data is under the current policy.

The intellectual move here is an application of importance sampling to preference optimization, but the adaptation is non-trivial in ways that matter. Standard importance sampling in RL uses the ratio $\pi_\theta(a|s) / \pi_{\text{behavior}}(a|s)$ to reweight transitions, but in the LLM preference setting, there is no single behavior policy—the off-policy data comes from a heterogeneous mixture of models and curation steps, so the denominator is unknown. WPO sidesteps this by using unnormalized weights proportional to $\pi_\theta(y_w|x) \cdot \pi_\theta(y_l|x)$ , which captures the relative on-policy-ness of different pairs without needing absolute data-generating probabilities. This is a practically important simplification that makes the method work with arbitrary off-policy datasets.

The practical significance is captured in a single sentence from the paper: "This method not only addresses the distributional gap problem but also enhances the optimization process without incurring additional costs." The significance extends beyond performance gains (up to 5.6% over DPO on Alpaca Eval 2 length-controlled win rate in the off-policy setting, Table 1) to the economics of alignment research: WPO enables rapid experimentation with preference optimization at DPO-level cost but with substantially improved results, lowering the barrier to entry for alignment work and making on-policy-quality alignment accessible to groups without the compute budget for online sampling. This is an incremental mechanism (importance weighting is not new) that produces a fundamentally different cost-performance trade-off curve, making it a significant practical innovation.

Innovation 3: Isolating the Importance of On-Policy Dispreferred Data Through Targeted Ablation

The paper's most striking and potentially counterintuitive finding is not just that on-policy data matters, but that on-policy dispreferred outputs matter far more than on-policy preferred outputs. This insight emerges from the controlled ablation in Section 4.3, where the authors decompose WPO's joint weighting $w(x, y_w) \cdot w(x, y_l)$ into two variants: WPO-W (weight only the preferred output) and WPO-L (weight only the dispreferred output). The results (Figure 4) show that WPO-L performs nearly identically to full WPO, while WPO-W consistently underperforms and often underperforms even standard DPO.

This is a conceptual advance because it reveals an asymmetry in how preference optimization uses the two halves of a preference pair. The gradient of the DPO/WPO loss pushes the policy to increase $\log \pi_\theta(y_w|x)$ and decrease $\log \pi_\theta(y_l|x)$ . The paper's finding indicates that where the model learns what NOT to do (away from $y_l$ ) is far more sensitive to the on-policy-ness of the data than where it learns what TO do (toward $y_w$ ). In retrospect, this makes intuitive sense: teaching the model to avoid outputs it would actually produce requires those outputs to be representative of its own behavior—you can't effectively penalize outputs the model would never generate anyway, because they aren't competing with the correct answer in the model's probability space. Conversely, preferred outputs can serve as positive examples even if they're somewhat off-policy, because the model can learn "move toward this kind of output" without needing the exact output to be probable under its own distribution.

Before this work, the literature treated preferred and dispreferred outputs symmetrically in discussions of data quality. The finding that dispreferred data quality is the bottleneck flips the focus of data curation efforts: rather than investing in better positive examples, alignment practitioners should invest in ensuring that the negative examples (the outputs the model should avoid) are drawn from the model's own distribution. This has direct practical implications for how to construct hybrid on-policy/off-policy datasets and for understanding why self-play methods (which generate on-policy negatives naturally) work well.

The evidence for this claim is robust across two different base models (Mistral-7B and Llama-3-8B-Instruct) and three RL settings (off-policy, on-policy, hybrid) in Figure 4. This is a diagnostic finding—it does not propose a new method but rather reveals which part of an existing method is doing the work, which is both intellectually clarifying and practically actionable.

Innovation 4: Identifying Input-Dependent Confidence as a Confound in Importance Weighting and Correcting It via Token-Level Calibration

A subtle but critical obstacle to using policy probabilities as data-relevance weights is that language models exhibit input-dependent variability in confidence: the same model producing equally "on-policy" outputs on different prompts will assign systematically different absolute probabilities to those outputs because some prompts naturally elicit sharper output distributions than others (Si et al., 2023; Xiong et al., 2024). If you naively weight by raw sequence probability, preference pairs from high-confidence prompts are systematically upweighted relative to pairs from low-confidence prompts, introducing an unintended bias unrelated to actual data relevance.

This problem is specific to the language modeling context and is not addressed by standard importance sampling techniques from RL, where the "policy" typically outputs a small number of scalar action probabilities. In an LLM, the "action" is a sequence of hundreds or thousands of tokens, each with its own probability, and the overall sequence probability is affected by both the quality of the output and the prompt-dependent entropy of the model's predictions.

The paper's solution—weight alignment via token-level normalization—is conceptually elegant. Rather than calibrating at the sequence level (which would require knowing what a "typical" on-policy sequence probability looks like for each prompt, an ill-defined quantity), it calibrates at the individual token level by normalizing each token's probability against a position-specific reference. The two reference choices—greedy alignment (normalize by the max token probability) and sampled alignment (normalize by the expected token probability under random sampling)—both have clean operational interpretations: greedy alignment asks "how much does this token deviate from the model's top choice?," while sampled alignment asks "how probable is this token compared to a random draw from the model's distribution?"

The empirical demonstration that this matters comes from Figure 2: without alignment, the weight distribution of outputs sampled directly from the policy model spans a wide range (roughly 0.2 to 1.2) rather than concentrating near 1.0, which is what on-policy outputs should ideally receive if the weights are to simulate uniform on-policy sampling. Sampled alignment produces the tightest distribution, and Table 2 confirms that sampled alignment yields the best downstream performance (24.4% LC win rate vs. 23.0% for greedy alignment and 22.0% for no alignment on Mistral-7B off-policy).

This is an incremental refinement of the importance weighting mechanism, but it addresses a problem that would otherwise undermine the entire approach. Without alignment, the weights would confound "is this output on-policy?" with "is this prompt naturally low-entropy?," making the weighted dataset a poor approximation of true on-policy resampling. The token-level calibration is a domain-specific solution to a domain-specific problem, and its necessity reveals something deeper about the challenge of applying importance sampling to sequence-level language model outputs.

Innovation 5: Demonstrating That Peak WPO Performance Does Not Coincide with Peak DPO Performance—And That WPO Resists DPO's Overoptimization Collapse

A fascinating and underexplored finding in the paper comes from Appendix A (Figure 5), where training dynamics are compared across epochs. DPO's performance on Alpaca Eval 2 peaks around epoch 2 and then collapses sharply—a pattern the authors attribute to reward model overoptimization (Rafailov et al., 2024), where the policy learns to exploit the implicit reward signal without genuinely improving output quality. WPO not only achieves higher peak performance but maintains stable performance across 5+ epochs without collapsing.

This is significant beyond the raw performance numbers because it suggests that WPO's reweighting does more than just improve data efficiency—it fundamentally changes the optimization landscape in a way that resists overoptimization. The mechanism is likely that by downweighting off-policy pairs (which may contain spurious patterns the policy can exploit to artificially inflate the reward gap without improving real quality), WPO removes some of the "reward hacking" opportunities that DPO's uniform weighting exposes the model to. The policy can only overoptimize on pairs that are heavily weighted, and WPO ensures that those pairs are precisely the ones representative of the policy's own behavior—pairs where "cheating" the reward signal is harder because the outputs are already close to what the policy naturally produces.

This is a diagnostic finding with practical and theoretical implications. Practically, it means WPO requires less careful early stopping and hyperparameter tuning than DPO—a significant usability improvement. Theoretically, it connects the distributional gap problem to the reward overoptimization problem, suggesting they may share a common root: when off-policy data contains outputs far from the model's natural distribution, the model can learn to satisfy the preference signal on those outputs through superficial changes (e.g., shifts in formatting, length, or style) that don't generalize to on-policy behavior. By reweighting toward on-policy data, WPO closes this loophole.

The paper does not make this connection explicitly in theoretical terms—it notes the stability result empirically and attributes it to on-policy simulation mitigating overoptimization—but the implication is clear and important: the distributional gap is not just an efficiency problem; it is a robustness problem. Off-policy DPO doesn't just learn slower; it learns less stably, and eventually learns the wrong things. WPO addresses both the speed and the stability of learning, which is a stronger claim than "better data utilization" alone would suggest. This insight is supported by the epoch comparison in Figure 5, where WPO's peak (epoch 5) substantially exceeds DPO's peak (epoch 2), confirming that the two methods converge to different solutions rather than merely reaching the same solution at different speeds.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses the binarized Ultrafeedback dataset (Cui et al., 2023), which contains approximately 63k preference pairs. These pairs were sampled from models other than the SFT model being trained—including GPT-4 and Llama-2 (Touvron et al., 2023)—making this an off-policy dataset by construction. For the hybrid RL setting, the authors augment this with on-policy outputs: they sample 5 outputs from the SFT model (using top-p sampling with p=0.95 and temperature 0.7) and add one output from gpt-4-turbo, then use gpt-4-turbo with an additive scoring prompt to annotate preferences. Specifically, outputs scoring 5 or 6 are selected as $y_w$ , and a random output with a score at least one point lower is chosen as $y_l$ ; prompts where such a pair cannot be formed are discarded. In the "+ Ultrafeedback" variant, discarded prompts are backfilled with their original Ultrafeedback preference pairs.
Base models. The paper uses two base model families for the main experiments: Mistral-7B (specifically the SFT checkpoint mistral-7b-sft-beta from HuggingFaceH4, which serves as the starting point for all preference optimization runs) and Llama-3-8B-Instruct (meta-llama/Meta-Llama-3-8B-Instruct). The Gemma-2-9b-it model (Team et al., 2024) is used for the strongest reported result (76.7% length-controlled win rate) but not in the systematic comparisons. These models span different scales (7B–9B) and families, providing some evidence of cross-architecture transfer, though all are in the sub-10B parameter range, and no results are reported for models at substantially different scales (e.g., 1B or 70B+).
Metrics. The paper evaluates on two instruction-following benchmarks. Alpaca Eval 2.0 (Li et al., 2023) consists of 805 representative instructions; for each instruction, the evaluated model's response is compared head-to-head against gpt-4-turbo's response by an auto-evaluator (also gpt-4-turbo), and the win rate is the probability the auto-evaluator prefers the evaluated model. Alpaca Eval 2 also provides a length-controlled win rate (Dubois et al., 2024) that corrects for gpt-4-turbo's known bias toward longer responses. MT-bench (Zheng et al., 2023) comprises 80 challenging multi-turn questions; the paper uses two scoring methods: (1) single-answer grading where gpt-4-0613 assigns scores from 1–10 (reported as average score), and (2) pairwise comparison where gpt-4-0613 compares two responses and declares a winner or tie (recorded as 0.5), producing a pairwise win rate. The authors note that the average MT-bench score shows limited separation between methods, and prioritize the pairwise win rate as the more discriminative metric. For downstream capability evaluation, the paper also reports results on the OpenLLM leaderboard (ARC, TruthfulQA, WinoGrande, GSM8k, HellaSwag, MMLU) in Appendix A (Table 4).
Baselines. The paper compares against several preference optimization methods, all evaluated under the same off-policy Ultrafeedback training setup where possible. These include DPO (Rafailov et al., 2023)—the direct predecessor that uses uniform weighting over preference pairs; ORPO (Hong et al., 2024)—which combines SFT and preference optimization into a single objective without a reference model; KTO (Ethayarajh et al., 2024)—which uses unpaired preference data and a prospect-theoretic loss; and SimPO (Meng et al., 2024)—which uses a reference-free reward with length normalization. For ORPO, KTO, and SimPO on Mistral-base, the paper reports results from the official model checkpoints rather than reproducing training (due to the hyperparameter sensitivity of these methods). For Llama-3-Instruct in the off-policy setting, official checkpoints are unavailable and these baselines are not reported. The SFT model (the checkpoint before any preference optimization) is also reported as a lower bound.
Generation budget / compute accounting. The paper does not report generation budgets or FLOP counts in the conventional sense—WPO does not involve generation during training (it is an off-policy method). The computational cost of WPO relative to DPO is argued to be negligible: the only additional operations are computing length-normalized sequence probabilities for $y_w$ and $y_l$ (already computed during DPO's forward pass for the loss itself), applying the weight alignment formula, and multiplying the per-pair loss by the detached weights. Training time is reported in Appendix C as approximately 1.5 hours on 8× H100 GPUs for Mistral-base and around 4 hours for Llama-3-Instruct, but these are given as practical reference points rather than as normalized FLOP counts against which methods are compared. No per-pair or per-epoch FLOP accounting is provided, making it impossible to verify the "zero additional cost" claim quantitatively from the paper alone.
Cross-validation / statistical protocol. All training configurations are run for 5 independent trials with different random seeds, and the paper reports both the mean and standard deviation of the results (Table 1). Statistical significance is assessed at $p < 0.05$ , with statistically significant gains underlined in the results table. This is a relatively standard protocol for alignment benchmarking and provides some protection against seed-dependent variation, though the number of trials (5) is modest. There is no cross-validation across data splits (the Ultrafeedback dataset has a fixed train/test structure), and the difficulty estimation component present in the reference example is not relevant here (WPO does not condition on prompt difficulty). For hyperparameter selection, the paper reports using a search within the range recommended by Meng et al. (2024) for Llama-3-Instruct, and adopting the official Zephyr hyperparameters for Mistral-base, but does not describe the search procedure in detail or report which hyperparameter configurations were tested and rejected.

Main Quantitative Results

Off-Policy Setting: WPO vs. DPO and Other Preference Optimization Methods

The headline result in the off-policy setting is that WPO achieves a length-controlled win rate of 24.4% on Alpaca Eval 2 with Mistral-7B, compared to 20.6% for DPO—an improvement of approximately 3.8 percentage points (Table 1). On Llama-3-8B-Instruct, WPO achieves 33.8% LC win rate versus DPO's 28.2%, an improvement of 5.6 percentage points—the largest reported gap. These results establish WPO's core claim: reweighting off-policy data by policy probabilities consistently and substantially outperforms uniform-weighted DPO.

Breaking down by metric (Table 1):

Mistral-7B, off-policy setting:

WPO: 24.4% LC win rate, 23.7% raw win rate vs. GPT-4, 7.37 average MT-bench score, 60.1% pairwise win rate vs. DPO on MT-bench (meaning WPO wins 60.1% of head-to-head comparisons against DPO)
DPO: 20.6% LC, 18.6% raw win rate, 7.36 MT-bench score, 50% pairwise vs. itself (reference)
SFT baseline: 9.5% LC, 5.8% raw win rate, 6.64 MT-bench score
Best competing method (SimPO): 21.5% LC, 21.4% raw win rate, 7.32 MT-bench score

The standard deviations (from 5 runs) are modest: WPO shows ±1.4 on LC win rate and ±2.1 on raw win rate, while DPO shows ±0.7 and ±1.0 respectively. The gains are statistically significant (underlined in the table, $p < 0.05$ ). Notably, WPO's improvements on MT-bench average score are negligible (7.37 vs. 7.36 for DPO), which the authors attribute to the average score metric having poor discriminability. The pairwise win rate on MT-bench (60.1% for WPO vs. DPO) provides clearer separation.

Llama-3-8B-Instruct, off-policy setting:

WPO: 33.8% LC win rate, 31.0% raw win rate, 8.14 MT-bench score, 58.1% pairwise vs. DPO
DPO: 28.2% LC, 24.0% raw win rate, 8.10 MT-bench score, 50% pairwise (reference)
SFT baseline: 26.0% LC, 25.3% raw win rate, 7.97 MT-bench score

The 5.6 percentage point gain on Llama-3 is larger than on Mistral (3.8 points), which may relate to the finding in Section 4.3 that Llama-3 benefits more from on-policy-like data because of its higher initial quality. The standard deviations for WPO on Llama-3 are ±1.3 on LC and ±1.8 on raw win rate. All gains are statistically significant.

Comparison to non-DPO baselines: WPO (24.4% LC) substantially outperforms ORPO (14.7%), KTO (14.9%), and SimPO (21.5%) on Mistral-7B. This includes SimPO, which was the previous strongest off-policy method on Mistral-base and actually outperforms DPO (21.5% vs. 20.6% LC). The fact that WPO beats both DPO and SimPO—which use different loss formulations—supports the claim that the distributional gap is a distinct problem from loss function design.

Hybrid Setting: Combining Off-Policy and On-Policy Data

The hybrid setting (Table 1, lower rows) incorporates on-policy outputs sampled from the SFT model, mixed with off-policy outputs from gpt-4-turbo, and annotated by gpt-4-turbo. This represents a stronger baseline than pure off-policy because some training data is already on-policy. The key question is whether WPO's reweighting provides additional value beyond having partial on-policy data.

Mistral-7B, hybrid setting:

WPO: 42.0% LC win rate, 46.2% raw win rate, 7.38 MT-bench score, 56.4% pairwise vs. DPO
DPO: 37.9% LC, 40.3% raw win rate, 7.14 MT-bench score, 50% pairwise (reference)
WPO + Ultrafeedback: 43.1% LC, 49.6% raw win rate, 7.23 MT-bench score, 58.8% pairwise vs. DPO

WPO maintains a 4.1 percentage point LC advantage over DPO even in the hybrid setting, demonstrating that reweighting the off-policy portion of the data still matters. Adding back the discarded Ultrafeedback prompts (the "+ Ultrafeedback" condition) provides a further ~1 percentage point gain (43.1% LC), confirming that more data—even off-policy—can be beneficial when properly weighted. The +Ultrafeedback WPO achieves a 49.6% raw win rate against GPT-4-turbo on Alpaca Eval 2, meaning the 7B model is preferred nearly half the time against GPT-4-turbo.

Llama-3-8B-Instruct, hybrid setting:

WPO: 45.8% LC win rate, 50.0% raw win rate, 8.18 MT-bench score, 54.8% pairwise vs. DPO
DPO: 44.2% LC, 48.6% raw win rate, 8.16 MT-bench score, 50% pairwise (reference)
WPO + Ultrafeedback: 48.6% LC, 52.1% raw win rate, 8.14 MT-bench score, 55.1% pairwise vs. DPO

The gap narrows to 1.6 percentage points LC (45.8% vs. 44.2%) in the basic hybrid setting, but widens again to 4.4 points in the +Ultrafeedback setting (48.6% vs. 44.2% for basic hybrid DPO). The +Ultrafeedback WPO result on Llama-3-8B (48.6% LC) represents the strongest reported result for an 8B model in the paper and closely approaches GPT-4-turbo parity. The raw win rate of 52.1% means Llama-3-8B with WPO is preferred over GPT-4-turbo on a majority of Alpaca Eval 2 prompts.

Gemma-2-9b-it with ArmoRM: In a separate experiment using a stronger base model (Gemma-2-9b-it) and a better reward model (ArmoRM; Wang et al., 2024a,b) for preference annotation, WPO achieves a length-controlled win rate of 76.7% and a raw win rate of 77.8% on Alpaca Eval 2. This is reported as a new state-of-the-art result and establishes that WPO scales to stronger base models and better preference signals, though the improvement relative to a DPO baseline on Gemma-2 is not reported, making it difficult to isolate WPO's contribution from the base model and reward model improvements.

Comparison of RL Settings: Off-Policy, On-Policy, and Hybrid

Section 4.3 (Figure 3) systematically compares WPO performance across three data regimes:

Off-policy: only the binarized Ultrafeedback data (63k preference pairs from other models)
On-policy: data generated entirely from the SFT model (sampled outputs, scored by gpt-4-turbo)
Hybrid: both on-policy and off-policy data (the setting from Table 1)

The results are presented as bar charts in Figure 3, with the left panel showing Alpaca Eval 2 length-controlled win rate and the right panel showing MT-bench pairwise win rate relative to the off-policy setting.

For Mistral-7B, off-policy slightly outperforms on-policy. The off-policy setting produces the highest LC win rate on Alpaca Eval 2, with on-policy slightly lower and hybrid achieving the best results among all three. The authors attribute this reversal to the quality of the Mistral SFT model: "the sampled outputs are of lower quality, causing the preference optimization process to mimic sub-optimal outputs and leading to poorer results." This is a crucial nuance: WPO's simulation of on-policy behavior is only beneficial if the on-policy behavior is itself good. When the SFT model produces low-quality outputs, pure on-policy training can reinforce mediocrity, and high-quality off-policy data (from GPT-4, etc.) provides a stronger signal.

For Llama-3-8B-Instruct, on-policy outperforms off-policy. The expected pattern—on-policy > off-policy—holds for the higher-quality Llama-3 model, consistent with prior work (Tang et al., 2024a; Xu et al., 2024). This suggests that WPO's effectiveness is partly contingent on base model quality: models that already produce reasonable outputs benefit more from on-policy data reweighting, while weaker models may need the "lift" provided by high-quality off-policy examples.

Hybrid consistently achieves the best results for both models. This confirms Rosset et al. (2024)'s finding that combining high-quality off-policy outputs with on-policy outputs yields superior performance, and shows that WPO works additively with this data mixture strategy.

The Asymmetric Importance of On-Policy Preferred vs. Dispreferred Data

Figure 4 presents results from a targeted ablation that decomposes WPO's joint weighting $w(x, y_w) \cdot w(x, y_l)$ into separate weighting of the preferred output only (WPO-W) and the dispreferred output only (WPO-L), compared to full WPO and standard DPO. The results are shown as bar charts for Mistral-7B (left panel) and Llama-3-8B-Instruct (right panel), each across off-policy, on-policy, and hybrid settings.

WPO-L (weight only the dispreferred output) performs nearly identically to full WPO. Across both models and all three RL settings, WPO-L's LC win rate tracks full WPO closely, with only minor degradation. This is the paper's most surprising empirical finding: the benefits of on-policy reweighting are almost entirely attributable to the dispreferred half of the preference pair.

WPO-W (weight only the preferred output) substantially underperforms full WPO and often underperforms DPO. In most settings, weighting only $y_w$ produces worse results than standard DPO (uniform weighting), particularly for Llama-3-Instruct where WPO-W in the hybrid setting drops noticeably below the DPO baseline. This means that incorrectly weighting the preferred output (making it "more on-policy") is actively harmful compared to uniform weighting, while correctly weighting the dispreferred output provides nearly all the benefit of full WPO.

The mechanistic interpretation (from the gradient decomposition shown in Section 4.3) is that the dispreferred output controls the "move away from this behavior" part of the gradient. If $y_l$ is off-policy (an output the current model would never generate), penalizing it doesn't effectively steer the model away from its actual bad behaviors—it's penalizing a straw man. Only when $y_l$ is representative of what the model might actually produce does the repulsive gradient hit the right target. The preferred output $y_w$ , by contrast, can serve as a positive example even if off-policy, because the model can learn to move toward high-quality outputs without needing them to already be probable.

The finding is consistent across all three RL settings and both model families. In Figure 4, WPO-L ≈ WPO > DPO > WPO-W holds for Mistral-7B off-policy, on-policy, and hybrid, and for Llama-3-8B-Instruct in all three settings (though the DPO vs. WPO-W ordering varies slightly). This robustness across conditions supports the conclusion that the asymmetry is a general property of preference optimization, not an artifact of a particular model or data regime.

Integrating WPO with Alternative Loss Functions

Table 3 reports the results of applying WPO's reweighting to three non-DPO preference optimization objectives: IPO (paired data, squared loss), SimPO (paired data, reference-free reward), and KTO (unpaired data, prospect-theoretic loss). All experiments are on Mistral-7B in the off-policy setting.

WPO-IPO achieves 29.4% LC win rate, compared to 25.0% for standard IPO—a 4.4 percentage point gain. The raw win rate improves from 21.2% to 25.7%, and the MT-bench pairwise win rate vs. the baseline is 54.1% (i.e., WPO-IPO beats IPO in 54.1% of head-to-head comparisons). This is the largest absolute improvement among the three alternative losses, and WPO-IPO actually outperforms WPO-DPO (24.4% LC on Mistral-7B off-policy) by 5.0 percentage points, suggesting that IPO's squared loss may interact particularly well with the reweighting.

WPO-SimPO achieves 21.9% LC win rate, compared to 21.5% for standard SimPO—a modest 0.4 percentage point gain. The raw win rate improves from 21.4% to 24.6% (a larger 3.2 point gain on the non-length-controlled metric), and the pairwise MT-bench is 52.5%. The small LC improvement might reflect that SimPO already incorporates length normalization into its reward, partially addressing one source of distributional mismatch that WPO's weighting also targets.

WPO-KTO achieves 21.1% LC win rate, compared to 14.9% for standard KTO—a 6.2 percentage point gain, the largest relative improvement (though WPO-KTO's absolute performance is lower than WPO-IPO or WPO-DPO). The raw win rate improves from 12.3% to 20.3%, and the pairwise MT-bench is 60.0%—the highest pairwise win rate against the baseline among all three alternative losses. The large gain on KTO is particularly notable because KTO uses unpaired data, requiring a different weighting strategy (per-output weighting with separate normalization for desired and undesired outputs within each batch, as described in Section 3.4).

Consistent improvement across all loss functions. The uniformity of the gains—every WPO variant outperforms its corresponding baseline on every metric—is strong evidence that the distributional gap problem is orthogonal to the loss function. If the gains were specific to DPO's sigmoid cross-entropy, one would expect no improvement (or even degradation) when applied to squared-loss or prospect-theoretic objectives. The fact that all three show gains suggests that reweighting off-policy data by policy probability is a broadly applicable technique independent of how the preference signal is converted to a training loss.

WPO-IPO outperforms WPO-DPO on Mistral-7B off-policy. This is a secondary finding that the paper does not emphasize: WPO-IPO's 29.4% LC win rate is the highest off-policy result reported for Mistral-7B, exceeding WPO-DPO's 24.4% by a substantial margin. This raises the question of whether IPO is a better base loss than DPO when combined with reweighting—a question the paper does not explore systematically (no WPO-IPO results are reported for Llama-3 or hybrid settings).

Training Stability and Resistance to Overoptimization

Appendix A (Figure 5) compares DPO and WPO training dynamics over 5 epochs on Mistral-7B with Ultrafeedback. The left y-axis shows Alpaca Eval 2 LC win rate (bar chart), and the right y-axis shows MT-bench average score (line plot).

DPO peaks at epoch 2 and collapses. The LC win rate rises from the SFT baseline to approximately 20–21% at epoch 2, then drops sharply to roughly 10–12% by epoch 5. The MT-bench average score similarly declines from ~7.4 to ~5.5. This is characteristic of reward model overoptimization (Rafailov et al., 2024): the policy learns to exploit the implicit DPO reward signal in ways that increase the reward gap but degrade actual output quality.

WPO continues to improve through epoch 5. The LC win rate rises steadily from epoch 1 through epoch 5, reaching its maximum at the final epoch (approximately 24–25% on Alpaca Eval 2, consistent with the 24.4% reported in Table 1 at epoch 1—note that Table 1 uses 1 epoch of training, while Figure 5 extends to 5 epochs, and WPO's epoch-5 performance exceeds its epoch-1 performance). The MT-bench average score remains stable around 7.4–7.5. WPO's stability across epochs is attributed to the downweighting of off-policy pairs that provide spurious reward hacking opportunities.

WPO's peak exceeds DPO's peak. At DPO's best epoch (epoch 2), WPO already matches or exceeds DPO's performance. At WPO's best epoch (epoch 5), WPO substantially outperforms DPO's best epoch. This demonstrates that the two methods converge to different solutions, not merely the same solution at different rates—WPO finds a better policy that would not be reached by training DPO longer or with different learning rate schedules.

The figure does not report WPO performance beyond epoch 5, and it is unclear whether WPO would eventually overoptimize given enough epochs, or whether the reweighting provides permanent protection against overoptimization. The paper frames this as "better training stability" and "mitigating issues related to reward model overoptimization" (Appendix A), not as a complete solution to overoptimization.

Downstream Task Performance (OpenLLM Leaderboard)

Table 4 in Appendix A reports accuracy on six standard benchmarks (ARC, TruthfulQA, WinoGrande, GSM8k, HellaSwag, MMLU) for SFT, off-policy DPO, off-policy WPO, hybrid DPO, and hybrid WPO, for both Mistral-7B and Llama-3-8B-Instruct.

Preference optimization (DPO or WPO) generally outperforms SFT on the aggregate OpenLLM average, but the differences are small and inconsistent. For Mistral-7B, the SFT average is 59.95%; off-policy DPO is 61.94% and off-policy WPO is 61.76%. Hybrid DPO reaches 63.27% and hybrid WPO is 63.01%. For Llama-3-8B-Instruct, the pattern is similar: SFT is 68.35%, off-policy DPO is 71.63%, off-policy WPO is 70.19%, hybrid DPO is 70.42%, hybrid WPO is 69.03%.

There is no positive correlation between OpenLLM performance and instruction-following performance. Specifically, hybrid WPO on Llama-3-8B—which achieves the best Alpaca Eval 2 and MT-bench results—actually underperforms off-policy DPO on the OpenLLM average (69.03% vs. 71.63%). This is the "alignment tax" phenomenon (Askell et al., 2021): better alignment with human preferences (as measured by Alpaca Eval 2) can come at the cost of degraded performance on standard academic benchmarks.

GSM8k shows consistent degradation after preference optimization. For Mistral-7B, SFT achieves 38.89% on GSM8k, while all DPO and WPO variants score between 30.17% and 32.60%. For Llama-3-8B-Instruct, SFT achieves 75.82%, while DPO and WPO variants score 66.72% to 75.13%. This is a well-known issue with RLHF-style training: optimizing for human preference can degrade mathematical reasoning, likely because preference data emphasizes style, helpfulness, and safety over raw problem-solving accuracy.

MMLU is essentially unchanged. Across all methods and both models, MMLU scores cluster tightly around 59–60% for Mistral-7B and 65–66% for Llama-3-8B-Instruct, with no method showing a clear advantage. This is consistent with MMLU being largely insensitive to preference optimization, as it tests factual knowledge rather than stylistic or alignment-related qualities.

WPO does not systematically outperform DPO on downstream tasks. On the aggregate average and on most individual benchmarks, WPO and DPO are within 1–2 percentage points of each other, with the direction of the difference varying across settings. The paper does not claim improvements on these benchmarks—WPO is designed for preference alignment, not for improving factual accuracy or reasoning. The OpenLLM results are presented for completeness rather than as evidence for WPO's effectiveness, and the authors explicitly note the lack of correlation between these metrics and instruction-following quality.

Ablation Studies and Robustness Checks

Weight alignment method (Table 2): The choice of weight alignment strategy has a clear performance ordering: sampled alignment (24.4% LC win rate, 23.7% raw win rate vs. GPT-4, 60.1% pairwise MT-bench vs. DPO) > greedy alignment (23.0% LC, 21.4% raw, 57.9% pairwise) > no alignment (22.0% LC, 20.3% raw, 54.4% pairwise) > DPO baseline (20.6% LC, 18.6% raw, 50% pairwise, reference). The 4.4 LC percentage point gap between sampled alignment and no alignment confirms that weight calibration is an important component of WPO, not a minor detail. The ranking matches the concentration of weight distributions shown in Figure 2, where sampled alignment produces the tightest distribution of weights for on-policy outputs (clustered near 1.0), greedy alignment produces a moderately spread distribution, and no alignment produces a wide spread. This supports the paper's theoretical motivation: the closer the weight distribution matches the ideal of uniform weights for on-policy outputs, the better the downstream performance.
WPO-W vs. WPO-L decomposition (Figure 4): As discussed in the main results, weighting only the dispreferred output (WPO-L) achieves near-identical performance to full WPO, while weighting only the preferred output (WPO-W) often underperforms DPO. This ablation reveals that the mechanism driving WPO's gains is not symmetric—the dispreferred output selection is the bottleneck, and on-policy negatives are far more valuable than on-policy positives. This is consistent across Mistral-7B and Llama-3-8B-Instruct and across all three RL settings.
WPO applied to non-DPO loss functions (Table 3): WPO's reweighting improves IPO, SimPO, and KTO, with gains ranging from modest (+0.4 LC points for SimPO) to large (+6.2 LC points for KTO). This robustness check confirms that WPO's effectiveness is not an artifact of DPO's specific mathematical form. The varying magnitude of improvement—large for KTO (which uses unpaired data and a fundamentally different loss structure), moderate for IPO, small for SimPO (which already incorporates length normalization)—suggests that the distributional gap is more severe for some loss functions than others, and that WPO's benefits are largest when the base loss is most vulnerable to off-policy data mismatch.
Training epoch sensitivity (Figure 5): DPO performance degrades sharply after epoch 2, with Alpaca Eval 2 LC win rate and MT-bench scores both collapsing. WPO remains stable through epoch 5 and continues to improve. This ablation is important because it tests whether WPO's gains could be matched by simply training DPO for more epochs with careful early stopping—Figure 5 shows they cannot, as DPO's peak (epoch 2) is below WPO's epoch-2 performance and far below WPO's epoch-5 peak. The stability result also has practical implications: WPO is more robust to hyperparameter choices related to training duration, reducing the need for extensive epoch tuning.
Base model quality and RL setting interaction (Figure 3): The finding that off-policy slightly outperforms on-policy for Mistral-7B, while on-policy outperforms off-policy for Llama-3-8B-Instruct, is an important robustness check that contextualizes WPO's applicability. It suggests that WPO's simulation of on-policy data is most valuable when (a) the base model already produces reasonable outputs (so mimicking on-policy behavior is desirable) and (b) pure off-policy data would otherwise create a substantial distributional gap. When the base model is weak, high-quality off-policy data may provide a stronger signal than (simulated) on-policy data, and WPO's reweighting toward on-policy might actually be counterproductive (though the paper does not test this directly—WPO is always applied with reweighting turned on, so we cannot see a "WPO with reversed weighting" ablation).
+Ultrafeedback data augmentation (Table 1, hybrid setting): Adding back the Ultrafeedback prompts that were discarded during hybrid data construction (because no valid preference pair could be formed from the sampled outputs) provides an additional ~1 percentage point LC improvement for WPO on Mistral-7B (42.0% → 43.1%) and ~2.8 points for WPO on Llama-3-8B (45.8% → 48.6%). This shows that even off-policy data that couldn't be paired with on-policy samples is still valuable when weighted by WPO—the model benefits from more data, even if imperfect, as long as the weighting prioritizes the relevant portion.
Reward model quality (Gemma-2 experiment): Using ArmoRM (a more sophisticated reward model) instead of gpt-4-turbo for preference annotation, combined with the Gemma-2-9b-it base model, pushes the LC win rate to 76.7%. While this is not a clean ablation (both the base model and the reward model change), it demonstrates that WPO benefits from better preference signals and stronger base policies—the method does not saturate at the performance levels shown in the main experiments.

Critical Assessment

Claim 1: WPO outperforms DPO by up to 5.6% on Alpaca Eval 2 in the off-policy setting.

What the experiments show: Table 1 reports a 5.6 percentage point improvement on Llama-3-8B-Instruct (33.8% vs. 28.2% LC) and a 3.8 point improvement on Mistral-7B (24.4% vs. 20.6%). These gains are statistically significant at $p < 0.05$ across 5 runs. The claim of "up to 5.6%" is accurate for the conditions tested.

What qualifies this: The 5.6% figure is specific to Llama-3-8B-Instruct in the off-policy setting. For Mistral-7B, the gain is 3.8%. In the hybrid setting, the gap narrows to 1.6–4.4 points depending on data configuration. So the magnitude of WPO's benefit depends on both the base model and the data regime. The paper could be clearer about this contingency—the "up to 5.6%" framing in the abstract is technically correct but suggests a larger typical gain than most experimental conditions produce. Additionally, the 5-run standard deviations on Llama-3 are ±1.3 for WPO and ±0.5 for DPO, meaning the 5.6 point gap is roughly 4 standard deviations—robust, but the error bars are non-trivial.

Claim 2: WPO simulates on-policy learning without incurring additional costs.

What the experiments show: The paper reports comparable training times (Mistral: ~1.5 hours, Llama-3: ~4 hours on 8× H100 GPUs; Appendix C) but does not report DPO training times separately, making a direct "no additional cost" comparison impossible from the reported data alone. The theoretical argument for zero additional cost rests on the fact that WPO's weight computation reuses probabilities already computed during DPO's forward pass. This is a reasonable argument—the length normalization and exponentiation are O(1) operations per token, negligible next to the transformer forward/backward passes—but it is not empirically verified through wall-clock time measurements, FLOP counts, or memory usage comparisons.

What qualifies this: The "zero cost" claim must be understood as "zero additional forward passes through the policy model" (the dominant cost), not literally zero additional computation. The weight alignment adds per-token operations (computing $\max_{v \in \mathcal{V}} \pi_\theta(v|x, y_{<t})$ for greedy alignment or $\sum_v \pi_\theta(v)^2$ for sampled alignment) that require summing over the vocabulary. For a vocabulary of 32k–128k tokens, this is non-trivial—it's a vectorized sum operation per token position—though still small relative to the transformer's compute budget. The paper does not measure this overhead or confirm that it is negligible in practice. For small models or fast training setups where data loading rather than model computation is the bottleneck, the additional operations might become noticeable.

Claim 3: WPO establishes a new SOTA length-controlled winning rate of 76.7% on Alpaca Eval 2.

What the experiments show: The Gemma-2-9b-it model trained with WPO in a hybrid-like setup with ArmoRM reward modeling achieves 76.7% LC win rate. This is reported as a SOTA result.

What qualifies this: There is no reported DPO baseline on Gemma-2-9b-it with the same data and reward model, so this number cannot be attributed specifically to WPO versus any other preference optimization method that might be used on the same base model with the same data. The 76.7% reflects the combination of a strong base model (Gemma-2-9b-it is more capable than Mistral-7B or Llama-3-8B), a high-quality reward model (ArmoRM), and WPO—but we cannot disentangle these contributions from the reported data. The paper should either report a DPO baseline in the same setup or acknowledge that the SOTA claim is for the full pipeline, not for WPO specifically. This is a significant gap in the experimental evidence for what is presented as a headline result.

Claim 4: WPO provides universal improvements across different loss functions for preference optimization.

What the experiments show: Table 3 shows WPO-IPO > IPO, WPO-SimPO > SimPO, and WPO-KTO > KTO on all metrics (Alpaca Eval 2 LC, raw win rate, and MT-bench pairwise). The gains are consistent in direction but vary in magnitude.

What qualifies this: The experiments are limited to Mistral-7B in the off-policy setting. No results are reported for Llama-3-8B-Instruct or hybrid settings, so the "universal" claim is extrapolated from a single model/data combination per loss function. For SimPO, the LC improvement is only 0.4 percentage points (21.5% → 21.9%), which may not be statistically significant (standard deviations are not reported for Table 3) and might not replicate. The claim of "consistent improvements" in the abstract is supported by the direction of all six comparisons (three loss functions × two main metrics), but the magnitude evidence is thin for SimPO.

Claim 5: On-policy dispreferred data is more important than on-policy preferred data.

What the experiments show: Figure 4 demonstrates WPO-L ≈ WPO > DPO ≥ WPO-W across Mistral-7B and Llama-3-8B-Instruct in off-policy, on-policy, and hybrid settings. The consistency is striking.

What qualifies this: This is a well-executed ablation with clear results across model families and data regimes. The main qualification is that WPO-L and WPO-W are not the only possible decompositions—WPO applies multiplicative weights to the joint pair, and the additive decomposition in the gradient (quoted in Section 4.3) shows separate terms for increasing $\pi_\theta(y_w)$ and decreasing $\pi_\theta(y_l)$ , so the ablation does directly test the two gradient components. A potential confound: WPO-W's underperformance might be specific to the weight values produced by the current weighting scheme. If a different weight calibration were used, the WPO-W vs. WPO-L gap might narrow or shift. The paper does not explore this. Additionally, the finding is interpreted as evidence that dispreferred data should be on-policy, but the experiment only manipulates weighting, not the actual data content—WPO-L doesn't make the dispreferred outputs more on-policy in content, it only upweights pairs where $y_l$ already has high probability under $\pi_\theta$ . This is a subtle but important distinction: the experiment shows that weighting $y_l$ by its probability is beneficial, not that replacing off-policy $y_l$ with on-policy $y_l$ would be beneficial (though the interpretation suggests that conclusion).

Claim 6: WPO mitigates reward model overoptimization and improves training stability.

What the experiments show: Figure 5 (Appendix A) shows DPO collapsing after epoch 2 while WPO remains stable through epoch 5. This is a single experiment (Mistral-7B, off-policy, one set of hyperparameters).

What qualifies this: The experiment uses the same hyperparameters for DPO and WPO, but DPO and WPO may have different optimal hyperparameters (learning rate, β, batch size). It is possible that DPO's collapse could be prevented with a lower learning rate or stronger KL penalty, and that DPO with tuned hyperparameters would remain stable and potentially match WPO's performance. The paper does not report a hyperparameter sweep for DPO across epochs or test whether DPO's collapse is invariant to β. This is a significant omission: if DPO's collapse is simply a learning rate artifact, the stability claim would be weakened. A stronger demonstration would show that DPO overoptimizes across a range of β values and learning rates while WPO does not. The paper also does not test whether WPO eventually overoptimizes given enough epochs (e.g., 10 or 20), so the claim should be "WPO resists overoptimization longer than DPO under the tested hyperparameters," not "WPO solves overoptimization."

Missing Experiments and Weaknesses

No DPO baseline for Gemma-2-9b-it. The 76.7% headline number cannot be attributed to WPO without a controlled comparison.
No FLOPs or wall-clock comparison. The "zero additional cost" claim is theoretically grounded but empirically unverified.
Single off-policy dataset (Ultrafeedback). All results use the same 63k preference pairs. WPO's effectiveness on other off-policy data distributions (e.g., HH-RLHF, OpenAssistant, synthetic data from different model families) is untested. Ultrafeedback has specific properties (GPT-4-generated outputs, Llama-2 outputs, a particular annotation protocol) that may interact with WPO's weighting in ways that don't generalize.
Limited model scale range. Results are reported for 7B–9B models. At larger scales (30B, 70B), the distributional gap might behave differently—larger models might have more peaked output distributions, changing how weights distribute—or the gap might be less severe because larger models are closer to the data-generating distribution. At smaller scales, the gap might be more severe. Without scaling experiments, we don't know whether WPO's benefits grow, shrink, or plateau with model size.
No systematic hyperparameter sensitivity analysis. The paper uses one set of hyperparameters for Mistral-7B and a searched set for Llama-3-8B, but doesn't report how WPO's performance varies with β, learning rate, batch size, or the number of training epochs beyond the 1 vs. 5 comparison for Mistral. This matters for a method whose core contribution is a reweighting scheme that interacts with the loss function's sensitivity to β.
No analysis of which pairs get upweighted/downweighted. The paper does not provide examples of preference pairs that receive high vs. low weights under WPO, nor does it characterize the properties of these pairs (length, style, content overlap with policy outputs). Such an analysis would strengthen the mechanistic claim that WPO is really upweighting "on-policy" data—without it, we only know that the weighting correlates with better performance, not that the weighting identifies on-policy-relevant pairs in the way the paper claims.
MT-bench average score is insensitive. The paper acknowledges this and pivots to pairwise comparisons, but the fact that the primary automated metric for dialogue quality shows essentially zero differentiation (7.36 vs. 7.37 on Mistral-7B, 8.10 vs. 8.14 on Llama-3-8B) raises questions about whether any of these methods are making meaningful progress on multi-turn conversation quality, as opposed to single-turn instruction following measured by Alpaca Eval 2.
No human evaluation. All results are based on LLM-as-judge metrics (gpt-4-turbo and gpt-4-0613). While these are widely used and correlate with human preferences, the absence of any human judgment data means we cannot rule out the possibility that WPO is optimizing for evaluator-specific biases rather than genuine quality improvements—a concern that applies to the entire alignment benchmarking paradigm, not just this paper, but worth noting given the magnitude of the claimed improvements.

6. Limitations and Trade-offs

The Gap Between Off-Policy and True On-Policy Preference Optimization Persists

WPO is explicitly designed to reduce, not eliminate, the performance gap between off-policy and on-policy preference optimization. The paper is transparent about this in Section 5:

"Although WPO simulates on-policy RL with off-policy data, it does not fully bridge the performance gap between off-policy and on-policy RL. As shown in the results, even with WPO, off-policy methods may still underperform compared to on-policy and hybrid methods."

The consequence is that WPO should be understood as a mitigation strategy rather than a solution to the distributional gap. In Figure 3, the on-policy setting with Llama-3-8B-Instruct produces superior Alpaca Eval 2 LC win rates compared to the off-policy setting even with WPO, and the hybrid setting (which incorporates actual on-policy data) consistently outperforms pure off-policy WPO across both Mistral-7B and Llama-3-8B-Instruct. For practitioners, this means WPO does not eliminate the need for on-policy data collection—it only extracts more value from the off-policy data you already have. If compute budget permits online sampling, on-policy or hybrid approaches will likely outperform WPO on static off-policy data alone.

The paper does not measure how much of the on-policy/off-policy gap WPO closes versus how much remains—no experiment provides a direct decomposition. The authors frame this as a direction for future work (Section 5): "Future work will be on how to further reduce this performance gap without incurring additional training costs." This is a candid acknowledgment but also means that practitioners cannot currently estimate the ceiling of WPO's effectiveness for a given base model and data distribution.

Additionally, the finding in Section 4.3 (Figure 3) that the relative benefit of on-policy versus off-policy training depends on base model quality—off-policy slightly outperforms on-policy for Mistral-7B while the reverse holds for Llama-3-8B-Instruct—complicates the story. WPO's simulation of on-policy behavior is only beneficial when on-policy behavior is itself good. For weaker SFT models whose sampled outputs are low-quality, pushing the training distribution toward on-policy through reweighting could theoretically be counterproductive (upweighting pairs that reflect poor model behavior). The paper does not test a "reverse WPO" that downweights on-policy pairs for weak models, leaving open the question of whether WPO's weighting direction is universally correct or should be adapted based on SFT quality.

The Weight Alignment Mechanism Introduces Undocumented Computational Overhead That Is Not Empirically Measured

The paper claims that WPO "enhances the optimization process without incurring additional costs" (Abstract) and positions the weight computation as reusing probabilities already calculated during DPO's forward pass. However, the weight alignment methods described in Section 3.3 require non-trivial additional computation that is never benchmarked.

For sampled alignment (the default method), computing the denominator $\sum_{v \in \mathcal{V}} \pi_\theta(v \mid x, y_{<t})^2$ requires summing over the entire vocabulary at every token position for both $y_w$ and $y_l$ . For a vocabulary of 32k–128k tokens (standard for models like Mistral and Llama-3) and sequences that may be hundreds of tokens long, this represents a substantial number of operations: the model must compute $\pi_\theta(v|x, y_{<t})$ for every token in the vocabulary at each position—something that is not required by standard DPO (which only needs the probabilities of the actually observed tokens). For greedy alignment, computing $\max_{v \in \mathcal{V}} \pi_\theta(v \mid x, y_{<t})$ requires identifying the argmax token at each position, which similarly requires examining the full token distribution.

The consequence is that WPO's "zero additional cost" claim is accurate with respect to generating new outputs (the dominant cost in on-policy methods), but misleading with respect to the actual per-step training cost compared to standard DPO. The weight alignment computation adds operations that scale with vocabulary size × sequence length × batch size. For a batch of 128 preference pairs with average output length of 200 tokens and a 32k vocabulary, sampled alignment requires approximately 128 × 2 × 200 × 32,000 ≈ 1.6 billion additional operations per training step (for the vocabulary sum alone, before the exponentiation and length normalization). This is small relative to a 7B-parameter transformer's forward pass (which involves trillions of operations), but it is not zero, and the paper provides no wall-clock timing comparison between DPO and WPO training to verify that the overhead is negligible in practice.

The training times reported in Appendix C—approximately 1.5 hours for Mistral-7B on 8× H100 GPUs and 4 hours for Llama-3-8B-Instruct—are given as absolute numbers for WPO, with no corresponding DPO timing, making a direct comparison impossible. For practitioners deciding between WPO and DPO, the question "how much slower is WPO per training step?" remains unanswered by the empirical data in the paper. The theoretical argument suggests the overhead should be small (a few percent), but without measurement, the paper's central economic claim—that WPO costs nothing beyond DPO—rests on an assertion rather than evidence.

The paper does not acknowledge this as a limitation, does not provide profiling data, and does not discuss the vocabulary-sum computation in Section 3.3 or Appendix C. The mitigation is implicit: one could use greedy alignment (which only requires argmax, not full vocabulary sum) at the cost of slightly lower performance (Table 2 shows a 1.4 LC percentage point drop), or simply omit weight alignment entirely (a 2.4 point drop). The optimal trade-off between computational overhead and alignment method performance is not characterized.

The 76.7% SOTA Result Cannot Be Attributed to WPO Specifically

The paper reports a headline result in Section 4.2: WPO with Gemma-2-9b-it and the ArmoRM reward model achieves a length-controlled win rate of 76.7% on Alpaca Eval 2. This is presented as a demonstration that "WPO can produce better LLMs" and implicitly as evidence for WPO's effectiveness at scale.

The fundamental problem is that no comparative baseline is reported for this experiment. There is no DPO Gemma-2-9b-it + ArmoRM result, no WPO with a different reward model on Gemma-2, and no Gemma-2-9b-it with no preference optimization at all. The 76.7% number conflates at least three factors: (1) the Gemma-2-9b-it base model, which is substantially more capable than Mistral-7B or Llama-3-8B-Instruct (Gemma-2 models were released after those and represent a newer generation of training); (2) the ArmoRM reward model, which is more sophisticated than the gpt-4-turbo scoring used in the main experiments and may produce higher-quality preference pairs; and (3) WPO's reweighting. Without a DPO baseline in the same setup, we cannot estimate what fraction of the 76.7% is attributable to WPO versus the base model and reward model improvements.

The consequence is that the paper's strongest numerical result—the one highlighted in the abstract as "establishing a remarkable length-controlled winning rate against GPT-4-turbo of 76.7% based on Gemma-2-9b-it"—provides no evidence specific to WPO. A practitioner reading the abstract might reasonably infer that WPO contributed decisively to this number, but the experimental design does not support that inference. The 76.7% could be achievable with standard DPO (or even SFT alone followed by best-of-N decoding) on the same base model with the same reward model. The paper's decision to report this as a WPO result without a DPO control undermines the strength of the SOTA claim.

The paper does not acknowledge this as a limitation. The closest it comes is describing the setup as "In a setup similar to our hybrid approach, we sample five outputs from Gemma and one additional output from gpt-4-turbo" (Section 4.2), but this describes the data construction, not the missing baseline. The only mitigation would be to run the DPO-controlled experiment, which is absent. For a paper whose primary contribution is a specific method (weighted preference optimization) rather than a system pipeline, the conflation of method and base model improvements in the strongest reported result is a significant evidential weakness.

The Method Has Only Been Validated on a Single Off-Policy Dataset and Two Model Families at a Single Scale

All off-policy experiments in the paper use the binarized Ultrafeedback dataset (Cui et al., 2023), which contains 63k preference pairs generated primarily by GPT-4 and Llama-2 models and annotated through a specific protocol. All experiments use models in the 7B–9B parameter range from three families: Mistral, Llama-3, and Gemma-2. This narrow empirical scope raises several distinct concerns about generalization, each with different consequences.

Dataset specificity. Ultrafeedback has specific distributional properties: the outputs were generated by strong models (GPT-4, Llama-2-70B-chat) and the preference annotations were produced by GPT-4, which may have systematic biases (e.g., preferring longer outputs, certain stylistic conventions, or particular reasoning formats). WPO's reweighting operates on the probability of these outputs under the current policy. If the off-policy outputs in Ultrafeedback are all from models substantially stronger than the policy being trained (as is the case when training a 7B model on GPT-4-generated outputs), the probability gap between on-policy and off-policy outputs will be systematically large—most pairs will receive low weights, and the effective training signal may come from a small subset of the data. On a dataset where the off-policy outputs are from models closer in capability to the policy being trained (e.g., a 7B model's outputs used to train another 7B model), the distribution of weights would be different, and WPO's benefits might shrink because the uniform-weighted data is already closer to on-policy. The paper provides no evidence about this contingency.

Model family and scale specificity. WPO's weighting depends on the policy model's probability estimates, which are influenced by model architecture, training data, and scale. Larger models tend to have sharper output distributions (higher confidence), which would produce systematically higher raw weights before alignment. The weight alignment mechanism is designed to correct for this, but whether it successfully calibrates across a wide range of model scales (e.g., 1B vs. 70B) is untested. Similarly, different model families may have different calibration properties—some models may be consistently overconfident or underconfident—which would change how the weights distribute and whether the alignment methods work as intended. The paper's primary comparison across model families (Mistral vs. Llama-3) shows WPO working for both, which is encouraging but limited to two families at similar scales.

Task domain specificity. All evaluation is on instruction-following benchmarks (Alpaca Eval 2 and MT-bench). While these are standard in the alignment literature, they represent a specific task domain: single-turn (MT-bench includes some multi-turn) open-ended instruction following. The paper provides no results for other alignment-sensitive tasks such as summarization (where faithfulness and conciseness trade off against length), coding assistance (where correctness is more important than style), or safety-critical refusal (where the preference structure is asymmetric—refusing is sometimes correct and sometimes incorrect). WPO's reweighting might interact differently with these domains. For example, in safety tasks where the preferred output is a refusal and the dispreferred output is a harmful response, the policy's probability of generating the harmful response may be very low (since SFT models are typically already safety-trained). WPO would downweight such pairs, potentially underinvesting in safety training relative to standard DPO. This is speculative—the paper does not study safety—but the absence of any domain variation limits confidence in WPO's universality.

The paper does not discuss these generalization concerns as limitations. The evaluation section describes the benchmarks used but does not acknowledge the narrowness of the empirical scope. The authors do state in the limitations (Section 5) that "the goal of our experiments is to compare WPO with other preference optimization algorithms, not to provide a comprehensively aligned LLM" and that Ultrafeedback "does not include safety aspects," which addresses part of the concern (task domain) but not the dataset or model scale specificity. No mitigation is proposed beyond suggesting that future work "should involve collecting more comprehensive preference datasets."

The Paper Provides Only Indirect Evidence That Reweighting Actually Identifies On-Policy-Relevant Data

WPO's central mechanistic claim is that reweighting preference pairs by $w(x, y_w) \cdot w(x, y_l)$ causes the training distribution to resemble what would be obtained by resampling on-policy data. The paper supports this through a thought experiment (Section 3.2) and through downstream performance improvements (Tables 1–3). However, it provides no direct evidence that the weights actually correlate with "on-policy-ness" in the way the theory assumes.

The only analysis of the weights themselves is Figure 2, which shows the distribution of $w(x, y)$ for outputs sampled from the policy model under different alignment methods. This demonstrates that weight alignment makes on-policy outputs receive weights concentrated near 1.0, which is consistent with the theory—but it does not show that off-policy outputs receive systematically lower weights, which is the other half of the mechanism. The paper never presents a scatter plot or histogram of weights for off-policy outputs versus on-policy outputs, nor does it characterize which properties of outputs (length, style, content overlap with policy outputs, lexical features) correlate with high vs. low weights. Without this evidence, it is possible that WPO's weights are doing something other than what the theory says—for example, they might be upweighting shorter sequences (which have higher per-token probabilities on average), or upweighting high-confidence prompts, or upweighting a particular subset of the data that happens to be easier to learn from, without actually implementing on-policy simulation in any meaningful sense.

The consequence is that the paper's central conceptual contribution—"simulating on-policy learning with off-policy data"—remains an interpretation rather than a verified mechanism. The performance improvements are real and well-measured, but they could result from a different mechanism entirely. For instance: (1) the weights might simply implement a form of curriculum learning, where the model first learns from "easy" pairs (high probability, close to its current behavior) and gradually incorporates harder ones; (2) the weights might act as a regularizer by downweighting outlier pairs that would otherwise cause large gradient updates; (3) the length normalization might be the primary benefit, with the policy-probability weighting playing a secondary role. The ablation in Table 2 shows that removing weight alignment hurts performance, but this only tells us that better weight calibration helps—not why. The WPO-W vs. WPO-L decomposition (Figure 4) provides some mechanistic insight by isolating the dispreferred output weighting, but still doesn't verify that high-weight pairs are actually "on-policy."

A diagnostic experiment that could clarify this would be: take the off-policy preference dataset, sort pairs by their WPO weight, and evaluate DPO trained on the top-k weighted pairs versus the bottom-k weighted pairs. If on-policy simulation is the mechanism, the top-k model should substantially outperform the bottom-k model, and the top-k model should perform similarly to a model trained on actually on-policy data of the same size. Neither experiment is reported.

The paper does not acknowledge this limitation. The thought experiment in Section 3.2 is presented as a derivation, with the step from "the occurrence rate would be proportional to $\pi_\theta(y_w|x)\pi_\theta(y_l|x)p(x)$ " to "this is equivalent to reweighting the original dataset" treated as an equivalence, not as a hypothesis requiring validation. For practitioners, the gap between "performance improved" and "we know why it improved" matters: if the mechanism is not what the paper claims, the method might fail in predictable ways that users cannot anticipate, and principled extensions (e.g., to multi-turn dialogue or to different data distributions) would be harder to design. The mitigation is partial: the consistent improvement across loss functions (Table 3) and across model families (Table 1) provides some evidence that the benefit is not a fluke of DPO's specific loss landscape, but it doesn't validate the specific claim of on-policy simulation.

The Revision Model or Online Sampling Loop That Would Maximize WPO's Benefits Is Not Developed

WPO is fundamentally a static reweighting method: it takes a fixed off-policy preference dataset and adjusts the importance of existing pairs based on the current policy's probabilities. This approach has an inherent ceiling: it can only upweight pairs that already exist in the off-policy dataset. If the current policy's output distribution differs from the off-policy data-generating distribution in ways that mean certain important preference comparisons are entirely absent from the dataset, no amount of reweighting can create them. The distributional gap has two components—data that is present but irrelevant (which WPO addresses by downweighting) and data that is relevant but absent (which WPO cannot address at all).

This limitation is most visible in the hybrid setting results (Table 1). Adding even a small amount of on-policy data to the training mixture (the "hybrid" condition) provides a large boost over pure off-policy WPO: Mistral-7B WPO goes from 24.4% LC (off-policy) to 42.0% LC (hybrid), and Llama-3-8B WPO goes from 33.8% to 45.8%. The gap between off-policy WPO and hybrid WPO is larger than the gap between off-policy DPO and off-policy WPO, indicating that having some on-policy data is substantially more valuable than optimally reweighting off-policy data alone. This is not a failure of WPO—the method was designed for pure off-policy settings—but it does clarify the practical ceiling: WPO is most useful when you cannot afford any on-policy data collection at all, and its benefits diminish as you add on-policy data to the mix.

The paper does not explore whether WPO's weights could be used to guide a limited on-policy sampling budget—for instance, by identifying which prompts or which types of preference pairs are most underweighted (indicating a large gap between off-policy data and current policy) and spending a small online sampling budget to fill those gaps. This would be a natural extension that combines WPO's diagnostic capability with targeted on-policy data collection, potentially achieving most of the hybrid performance gain at a fraction of the full on-policy sampling cost. The paper identifies this implicitly (the hybrid results show the value of mixing on-policy and off-policy data) but does not propose a principled method for deciding how much on-policy data to collect or which prompts to prioritize.

The authors acknowledge the persistence of the performance gap in the limitations (Section 5): "on-policy preference data remains important." However, they do not frame the absence of on-policy data for certain regions of output space as a fundamental ceiling on WPO's effectiveness, nor do they discuss active sampling strategies as a mitigation. For practitioners with a limited compute budget, the practical question is: "Given that I can afford either (a) pure off-policy WPO with no online sampling, or (b) a small amount of online sampling combined with off-policy DPO, which is better?" The paper provides data suggesting the answer depends on the specific budget and model quality (Figure 3 shows that pure on-policy can underperform pure off-policy for weak SFT models), but it does not characterize the trade-off curve between online sampling budget and performance when combined with WPO. This is a significant gap for deployment decision-making.

7. Implications and Future Directions

How This Work Changes the Landscape

WPO introduces an architecturally minimal change—multiplying each preference pair's loss contribution by a detached weight computed from the policy's own probabilities—that produces a disproportionately large empirical effect (up to 5.6 percentage points improvement on Alpaca Eval 2 length-controlled win rate over DPO, Table 1). The magnitude of this effect, given the simplicity of the intervention, forces a reevaluation of where the field should invest its optimization effort in preference-based alignment.

The dominant research thrust in preference optimization since DPO's introduction has been loss function design: IPO's squared loss, SimPO's reference-free reward with length normalization, KTO's prospect-theoretic formulation, ORPO's combined SFT+preference objective. Each paper asked, in effect, "What is the correct mathematical form for converting a preference pair into a training signal?" WPO demonstrates that this question, while important, is incomplete. A preference pair is not just a mathematical object to be plugged into a loss function—it is a sample from some data-generating distribution that may or may not be relevant to the policy being trained. The relevance of the data matters at least as much as the formula applied to it, and the two are orthogonal axes of improvement.

This is not a paradigm shift—WPO does not replace DPO, propose a new loss function, or change the fundamental preference learning framework. It is better understood as a reframing: the field has been optimizing the "how" of preference optimization (the loss) while largely ignoring the "what" (which data matters). WPO demonstrates that the "what" is not just a data collection problem to be solved once with better datasets, but a dynamic property that changes during training as the policy evolves. A pair that was highly relevant at the start of training becomes less relevant as the policy moves away from the data-generating distribution, and vice versa. This reframing opens a new axis of research orthogonal to loss function design.

The paper also resolves a subtle contradiction in the iterative DPO and self-play literature. Prior work had shown that iteratively refreshing the preference data with on-policy samples improves performance (Xu et al., 2023; Rosset et al., 2024; Chen et al., 2024), but it was unclear whether this benefit came from (a) having more data, (b) having data from a distribution closer to the current policy, or (c) some interaction between the data distribution and the DPO loss's optimization dynamics. The WPO-W vs. WPO-L ablation (Figure 4) provides a surprisingly clean answer: the benefit is asymmetric, concentrated almost entirely in the dispreferred outputs. Making $y_l$ closer to on-policy (via reweighting) recovers nearly all of WPO's gains, while making $y_w$ on-policy often provides no benefit or is actively harmful. This finding retroactively explains why self-play methods (which naturally generate on-policy dispreferred outputs by using the model's own previous responses as negatives) are effective: they provide exactly the ingredient that matters most. It also explains why simply adding high-quality off-policy preferred outputs (e.g., from GPT-4) can be beneficial without corresponding on-policy negatives—the preferred outputs don't need to be on-policy; the dispreferred ones do.

Perhaps the most consequential reframing is around reward overoptimization. Prior work (Rafailov et al., 2024; Gao et al., 2023) treated reward overoptimization as a problem of the reward model's fidelity: the implicit DPO reward diverges from the true preference signal, and the policy exploits this divergence. WPO's training dynamics (Figure 5, Appendix A) show that DPO collapses after two epochs while WPO remains stable through five, suggesting that the distributional gap itself is a contributing cause of overoptimization. The mechanism is plausible: off-policy preference pairs contain spurious patterns (formatting artifacts, length cues, stylistic tics from the data-generating models) that the policy can learn to exploit to inflate the implicit reward gap without improving genuine output quality. By downweighting these pairs, WPO removes some of the "reward hacking surface area." If this mechanism is correct, it implies that improving data relevance is a more direct path to training stability than improving reward model accuracy—a claim with significant implications for how alignment research allocates effort between reward modeling and data curation.

The practical consequence is a shift in the cost-benefit calculus for alignment practitioners. Before WPO, the choice was binary: pay the computational cost of on-policy sampling (PPO, iterative DPO, self-play) for better alignment, or accept lower quality in exchange for the efficiency of off-policy DPO. WPO demonstrates that a substantial fraction of the on-policy benefit—specifically, the benefit of having on-policy dispreferred outputs—can be achieved through post-hoc reweighting at negligible additional cost. This doesn't eliminate the need for on-policy data (the hybrid results in Table 1 show that real on-policy data still helps), but it changes the question from "can I afford on-policy training?" to "how much on-policy data do I actually need, and where should I spend my sampling budget?" The answer, based on Figure 4, is that sampling budget should be concentrated on generating dispreferred outputs, not preferred ones—a non-obvious allocation that prior work had not identified.

Follow-Up Research This Work Enables

Characterizing what WPO upweights: a content analysis of high-weight versus low-weight preference pairs. The paper's central mechanistic claim—that WPO's weights correlate with on-policy relevance—is supported only indirectly through performance improvements. A direct validation would analyze the actual preference pairs that receive high versus low weights under a trained WPO policy. Concrete experiment: train WPO on Mistral-7B with Ultrafeedback, freeze the policy at epoch 1, and compute weights for all 63k pairs. Extract the top-1% and bottom-1% weighted pairs and categorize them by: (a) output length relative to policy-generated outputs, (b) lexical overlap with policy-generated outputs on the same prompts, (c) presence of formatting artifacts from specific data-generating models (e.g., GPT-4's characteristic bullet-point style), (d) semantic similarity to policy outputs measured by embedding distance, and (e) the policy's own probability of generating each output at temperature 1.0. If WPO's mechanism is as claimed, high-weight pairs should be systematically shorter (or length-matched), lexically similar, free of foreign-model artifacts, and semantically close to on-policy outputs. A null result—no systematic difference between high and low-weight pairs on these dimensions—would indicate that the performance gains come from a mechanism other than on-policy simulation (e.g., curriculum learning or variance reduction), requiring a substantial reinterpretation of the method.

Testing whether WPO eliminates the need for early stopping across hyperparameter ranges. Figure 5 shows WPO resisting DPO's overoptimization collapse on one hyperparameter configuration (Mistral-7B, Ultrafeedback, β=0.01, lr=5e-7). However, DPO's collapse point is known to depend on the KL penalty coefficient β and the learning rate—stronger KL regularization can delay or prevent overoptimization. The open question is whether WPO's stability advantage persists across a range of β values or whether it is equivalent to simply training DPO with a larger β. Concrete experiment: sweep β ∈ {0.001, 0.01, 0.1, 1.0} and learning rate ∈ {1e-7, 5e-7, 1e-6} for both DPO and WPO, train for 5 epochs, and measure Alpaca Eval 2 LC win rate at each epoch. If DPO with β=0.1 matches WPO's stability and peak performance, then WPO's stability benefit reduces to implicit KL regularization through reweighting, which is a different mechanism than the paper claims. If WPO maintains stability even at low β where DPO collapses, the distributional gap is confirmed as an independent contributor to overoptimization. Additionally, extend training to 10+ epochs for WPO to determine whether it eventually overoptimizes—Figure 5 shows stability through epoch 5, but the ceiling is unknown. A finding that WPO eventually collapses at epoch 8 would define the practical limits of the method's stability benefit.

Measuring WPO's effectiveness as a function of the off-policy data's divergence from the current policy. WPO's thought experiment (Section 3.2) conceptualizes the distributional gap, but the paper never varies the gap systematically to measure WPO's sensitivity. The Ultrafeedback dataset has a particular divergence profile (GPT-4 and Llama-2 outputs, high quality, certain stylistic properties). How does WPO's benefit change when the off-policy data is: (a) generated by a much weaker model than the policy being trained (e.g., using a 1B model's outputs to train a 7B model—the reverse of the paper's setup), (b) generated by the SFT checkpoint that initializes the policy (making the initial gap zero and growing during training), (c) synthetically perturbed to be systematically longer, shorter, or stylistically distinct from the policy's natural outputs? Concrete experiment: take the Mistral-7B SFT model, generate an off-policy preference dataset by sampling from itself (so the initial gap is zero), train WPO and DPO, and compare. Then, repeat with off-policy data from Mistral-7B-Instruct, Llama-2-7B, Llama-2-70B, and GPT-4—each representing a progressively larger expected distributional gap. The hypothesis: WPO's benefit relative to DPO should grow monotonically with the gap size until a point where the gap is so large that even reweighting cannot salvage the data (all weights near zero, effectively training on noise). Characterizing this inverted-U curve would tell practitioners when WPO is worth applying versus when they should discard the off-policy data entirely.

Combining WPO with targeted on-policy data collection informed by the weights. WPO provides a per-pair relevance signal at zero sampling cost. This signal could be used to decide which prompts or outputs to spend a limited on-policy sampling budget on. The paper shows (Table 1, hybrid setting) that adding on-policy data substantially improves performance, but it adds on-policy data uniformly—all prompts get the same number of sampled outputs. The hypothesis is that prompts whose off-policy preference pairs receive the lowest WPO weights (indicating the largest distributional gap) are the ones where on-policy data would be most valuable. Concrete experiment: given a fixed budget of sampling on-policy outputs for B prompts (e.g., B=1000 out of 63k), compare three strategies: (a) uniform random selection of prompts, (b) selection of prompts with the lowest average WPO weight (largest distributional gap), (c) selection of prompts with the highest average WPO weight (smallest gap—a baseline for directionality). Generate on-policy preference pairs for the selected prompts, mix with off-policy WPO on the remaining prompts, and measure Alpaca Eval 2 performance. If strategy (b) outperforms (a) by a meaningful margin, WPO's weights serve a dual purpose: improving off-policy training and guiding efficient on-policy data collection. This turns WPO from a static reweighting method into an active data acquisition policy, bridging off-policy and on-policy approaches.

Validating WPO on safety-critical alignment tasks where the preference structure is asymmetric. The paper's evaluation is entirely on general instruction following, where "preferred" and "dispreferred" are symmetric in the sense that both outputs are plausible responses to the prompt. In safety alignment, the preference structure is asymmetric: for a harmful prompt, the preferred output is a refusal, and the dispreferred output is a harmful response. The policy's probability of generating the harmful response may already be very low (since SFT models typically refuse harmful requests), so WPO would assign low weights to these pairs—potentially downweighting the most safety-critical training examples. Conversely, for benign prompts where the policy sometimes over-refuses, the dispreferred output (a refusal when help was needed) might also receive low weight if the refusal style differs from the policy's typical outputs. The result could be that WPO systematically underinvests in safety training relative to standard DPO. Concrete experiment: train WPO and DPO on a safety-focused preference dataset (e.g., Anthropic's HH-RLHF or a dataset with explicit harm/refusal annotations) and evaluate on both safety metrics (refusal rate on harmful prompts, over-refusal rate on benign prompts) and helpfulness metrics. A negative result where WPO degrades safety performance would define a critical boundary condition on the method's applicability and would motivate a "safety-aware" variant that upweights safety-critical pairs regardless of their on-policy probability.

Scaling WPO to larger models (30B–70B) to determine whether the benefit persists, grows, or shrinks. The paper evaluates on 7B–9B models. Larger models have different properties that could affect WPO: (a) their output distributions may be sharper (higher confidence per token), producing systematically higher raw weights that might saturate the weighting signal; (b) their SFT performance is higher, so the distributional gap to off-policy data (which comes from other strong models) may be smaller, reducing WPO's headroom; (c) their training is more expensive, making WPO's "zero additional cost" claim more sensitive to any hidden overhead from the weight alignment computation. Concrete experiment: reproduce the off-policy setting on Llama-3-70B-Instruct (or another available 70B model with a public SFT checkpoint) using the same Ultrafeedback dataset and hyperparameter protocol, measuring Alpaca Eval 2 LC win rate for DPO vs. WPO. If the gap shrinks from 5.6 points (8B) to <1 point (70B), the practical value of WPO is limited to smaller models where the distributional gap is large. If the gap persists or grows, WPO is a scaling-compatible method that becomes more valuable as models get larger and training gets more expensive—a strong argument for adoption. If WPO degrades performance at scale (weights cause instability or overconfidence), that's a critical limitation requiring architectural modification.

Practical Applications and Downstream Use Cases

Cost-constrained alignment for teams without on-policy sampling infrastructure. The primary deployment scenario for WPO is any setting where running online sampling loops during training is impractical—limited GPU budget, no reward model serving infrastructure, or rapid experimentation cycles where iterating on data construction is affordable but iterating on training requires expensive generation. The paper's off-policy results (Table 1) show that WPO improves over DPO by 3.8–5.6 LC percentage points on Alpaca Eval 2 with no additional generation cost over standard DPO training. For a team fine-tuning a 7B–8B model on a static preference dataset, this is a drop-in improvement requiring only a loss function change in their training code—the weights are computed from probabilities the model already calculates. The training time overhead, while not measured in the paper, is bounded by the vocabulary-sum operations in the weight alignment (likely 1–5% for typical vocabulary sizes and sequence lengths), making this a near-zero-cost upgrade. The practical recipe: (1) take an existing DPO training pipeline, (2) replace the DPO loss with the WPO loss using sampled alignment (Equation 2), (3) ensure the weights are detached from the gradient graph, (4) use the same hyperparameters as the DPO baseline initially, then tune β and learning rate if needed. The paper's results on Mistral-7B (1.5 hours on 8× H100s) and Llama-3-8B (4 hours) confirm that WPO training is fast enough for rapid experimentation.

Improving data mixture strategies for hybrid off-policy/on-policy training with limited sampling budgets. The hybrid results in Table 1 show that adding on-policy data to off-policy WPO yields large gains—from 24.4% to 42.0% LC win rate on Mistral-7B, and from 33.8% to 45.8% on Llama-3-8B. But collecting on-policy data is expensive: sampling outputs, scoring them with a reward model, and forming preference pairs. The WPO-W vs. WPO-L ablation (Figure 4) provides a specific, actionable data collection strategy: invest the on-policy sampling budget in generating dispreferred outputs, not preferred ones. Since weighting the dispreferred output drives nearly all of WPO's benefit, practitioners constructing hybrid datasets should: (1) use off-the-shelf high-quality outputs (from GPT-4, existing datasets) as preferred outputs, (2) sample the model's own outputs as dispreferred candidates, (3) pair them and apply WPO's reweighting. This strategy produces the highest-value on-policy data (dispreferred outputs that represent what the model might actually generate) while saving the cost of generating on-policy preferred outputs (which Figure 4 shows provide little additional benefit). For a team with a budget of N on-policy generations, this reallocates the budget from a 50/50 preferred/dispreferred split to an 80/20 or 100/0 split favoring dispreferred outputs, potentially achieving similar performance to the hybrid setting at a fraction of the generation cost. The paper doesn't directly test this allocation strategy, but the WPO-L result strongly implies it would work.

Stabilizing long-duration preference optimization runs without extensive hyperparameter tuning. The training dynamics comparison in Figure 5 (Appendix A) shows DPO performance collapsing after epoch 2 while WPO remains stable and continues improving through epoch 5. For practitioners running multi-epoch preference optimization, this has a concrete operational benefit: WPO reduces the need for careful epoch-by-epoch validation and early stopping. With DPO, a team must evaluate checkpoints at each epoch (or more frequently) on a held-out set to catch the collapse point, and the optimal stopping epoch may vary with model scale, dataset, and hyperparameters. WPO's stability means practitioners can simply train for the scheduled number of epochs and take the final checkpoint, confident that performance hasn't degraded—or even train longer than initially planned if compute budget permits. The paper only demonstrates stability through epoch 5, so this recommendation is bounded, and practitioners should still validate at the final epoch. But the reduction in early-stopping sensitivity directly saves researcher time and compute spent on checkpoint evaluation. Additionally, the finding that WPO's peak performance (epoch 5) exceeds DPO's peak (epoch 2) means that switching to WPO provides both a higher ceiling and a flatter optimization trajectory—a rare combination that makes hyperparameter tuning more forgiving.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023· training-methods← LLM-judged similar
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback2023· alignment← LLM-judged similar
- Fine-Tuning Language Models with Just Forward Passes2023· training-methods← in-domain predecessor
- DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization2025· training-methods← similar topic
Offline Regularised Reinforcement Learning for Large Language Models Alignment2024· rl-training← in-domain predecessor
- Super-Exponential Regret for UCT, AlphaGo and Variants2024· rl-training← in-domain predecessor
- Conservative Q-Learning for Offline Reinforcement Learning2020· rl-training← similar topic
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline2024· evaluation← topic thread
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference2024· evaluation← LLM-judged similar
- OLMES: A Standard for Language Model Evaluations2024· evaluation← in-domain predecessor
- Bootstrapping Language Models with DPO Implicit Rewards2024· alignment← topic thread
- Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs2026· evaluation← similar topic
- Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation2025· evaluation← similar topic
Statistical Rejection Sampling Improves Preference Optimization2023· rl-training← similar topic
- Gated recurrent neural networks discover attention2023· rl-training← in-domain predecessor
- Stabilizing RLHF through Advantage Model and Selective Rehearsal2023· alignment← topic thread

What it influenced

Later papers in the same chain. Direction reversed.

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code2024· rl-training→ in-domain predecessor
- REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards2025· reasoning→ LLM-judged similar
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions2024· rl-training→ in-domain predecessor
Towards a Unified View of Preference Learning for Large Language Models: A Survey2024· alignment→ topic thread
- Language Models Learn to Mislead Humans via RLHF2024· alignment→ in-domain predecessor
- Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics2026· alignment→ similar topic
Statistical Rejection Sampling Improves Preference Optimization2023· rl-training→ similar topic
- Contrastive Prefence Learning: Learning from Human Feedback without RL2023· rl-training→ in-domain predecessor
- Efficient RLHF: Reducing the Memory Usage of PPO2023· training-methods→ topic thread
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization2026· rl-training→ similar topic