Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

ArXiv: 2605.20834

🎯 Pitch

DPO is not universally equivalent to RLHF—the derivation silently assumes the optimal policy already prefers human-chosen responses, an assumption that crumbles when the reference model is misaligned, causing DPO to optimize relative advantage rather than true preference satisfaction. The authors expose a pathological solution space where DPO loss decreases while dispreferred responses gain likelihood, and introduce CPO, a constrained variant that provably escapes this trap and achieves new state‑of‑the‑art performance.

---

1. Executive Summary

This paper analyzes the theoretical equivalence between Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF), proving that the claimed equivalence is conditional rather than universal — it depends on an implicit assumption (that the RLHF-optimal policy must prefer human-preferred responses over dispreferred ones) frequently violated in practice when the reference policy is sufficiently misaligned. The analysis uses Llama-3-8B-Instruct and the UltraFeedback preference dataset, characterizing failure modes through a newly defined undesirable solution space (policies that simultaneously decrease DPO loss while preferring dispreferred responses) and demonstrating that DPO effectively implements soft margin ranking with a potentially negative target margin (δ_ref < 0) that allows pathological convergence. To address this, the authors introduce Constrained Preference Optimization (CPO) and its conservative variant E-CPOC, which augment RLHF with explicit constraints guaranteeing absolute advantage (δ_π > 0 for preferred responses) and achieving state-of-the-art performance on AlpacaEval 2 (25.15% win rate, +0.55% over DPO) and Arena-Hard (32.6% win rate, +2.6% over the closest competitor), establishing that DPO inherits RLHF's algebraic structure through reward reparameterization but does not inherit its alignment guarantees except when the reference policy is already well-aligned with human preferences.

2. Context and Motivation

The Core Problem: DPO's Theoretical Foundations Are Fragile

The fundamental question this paper asks is: does Direct Preference Optimization (DPO) actually optimize the same objective as RLHF? This is not a minor implementation concern — it cuts to the heart of whether DPO, widely adopted precisely for its claimed equivalence to RLHF, inherits the alignment guarantees that make RLHF theoretically sound. The paper identifies a gap between the algebraic manipulation used to derive DPO and the semantic conditions under which that manipulation is valid.

The distinction matters because DPO's popularity rests on a specific promise: that it achieves what RLHF achieves (a policy aligned with human preferences via reward maximization under KL regularization) without the computational overhead of training a separate reward model or running reinforcement learning. This promise appears in the very title of Rafailov et al. (2023): "Your Language Model is Secretly a Reward Model." But the current paper demonstrates that this "secret" — the reward reparameterization that eliminates the reward model from the objective — carries with it a hidden condition. The reparameterization substitutes the RLHF-optimal policy $\pi^*$ into the Bradley-Terry preference model to obtain a loss that depends only on $\pi_\theta$ and $\pi_\text{ref}$. This substitution, however, assumes that $\pi^*$ already respects human preferences — that $\pi^*(y_w|x) > \pi^*(y_l|x)$ whenever humans prefer $y_w$ over $y_l$. If this assumption does not hold, the two optimization landscapes diverge, and DPO optimizes something fundamentally different from what it claims.

Why This Matters: Theoretical and Practical Stakes

Theoretical Significance. The paper's title — "Conditional Equivalence" — positions it as a corrective to what has become received wisdom in the alignment community. Rafailov et al. (2023) stated that DPO "optimizes the same objective as RLHF." The current paper proves this is true if and only if the reference policy is already well-aligned with human preferences (Theorem 3.5). When the reference policy is misaligned, DPO optimizes for relative advantage over the reference policy ($\delta_\pi - \delta_{\pi_\text{ref}}$) rather than absolute alignment with human preferences ($r^*$). This is a sharp theoretical distinction: equivalence is conditional on reference policy quality, not an inherent property of the derivation.

Practical Stakes. The assumption violation is not a corner case. Using Llama-3-8B-Instruct (an instruction-tuned model, not a random initialization) on the UltraFeedback dataset, the paper measures that 45.5% of preference pairs violate Assumption 3.1 (Figure 1, Appendix A.1). The reference policy's $\delta_{\pi_\text{ref}}$ has a standard deviation of 46.69, while the reward correction $\Delta r^*/\beta$ has a mean of only 0.20 (at $\beta = 0.1$). This means the reward signal often cannot compensate for the reference policy's systematic biases — the KL penalty inherited from the reference dominates. DPO is frequently deployed precisely in this regime: starting from a supervised fine-tuned model that needs alignment, where the reference may not already prefer human-preferred responses. The 45.5% violation rate on an instruction-tuned model demonstrates that the pathology is pervasive in realistic settings.

Deployment Implications. The failure mode is particularly insidious because it is silent. The paper proves that policies can decrease DPO loss while systematically preferring dispreferred responses (Definition 3.3: the undesirable solution space $\mathcal{U} = \{\pi : \delta_\pi < 0 \text{ and } \delta_\pi > \delta_{\pi_\text{ref}}\}$). A practitioner monitoring the training loss curve sees improvement; a user interacting with the deployed model experiences deterioration. Standard monitoring cannot detect this misalignment because the DPO loss measures $\delta_\pi - \delta_{\pi_\text{ref}}$ (how much better the policy is than the reference), not $\delta_\pi$ (whether the policy actually prefers the right responses). The paper provides explicit gradient analysis (Section 3.2) showing that the gradient magnitude $\sigma(-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}))$ becomes progressively weaker as $\delta_{\pi_\theta}$ approaches 0 from below, creating a practical optimization barrier where policies become trapped in $\mathcal{U}$ while the loss continues to decrease.

Where Prior Approaches Fall Short

The paper does not identify DPO's limitation in isolation — it connects to a growing body of work that has documented discrepancies between DPO and RLHF without identifying the root cause.

Implicit reward overfitting. Fisch et al. (2024) showed that DPO's implicit rewards ($r_\theta = \beta \log \frac{\pi_\theta}{\pi_\text{ref}}$) overfit and trend toward infinite magnitude, yielding degenerate policies where even preferred responses receive near-zero probability. The current paper provides a mechanistic explanation: when $\delta_{\pi_\text{ref}}$ is negative, the DPO gradient pushes $\delta_{\pi_\theta}$ to infinity (to maximize $\sigma(\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}))$) while RLHF's optimal $\delta_{\pi^*}$ remains negative due to KL regularization. The divergence between finite RLHF optimum and infinite DPO optimum is a direct consequence of the objective mismatch, not an independent phenomenon.

Poor generalization of implicit rewards. Lin et al. (2024) demonstrated that DPO's implicit reward model generalizes significantly worse than explicit reward models under distribution shift. The current paper's framework suggests a structural explanation: DPO's implicit rewards are optimized for relative advantage, not absolute reward accuracy. Under distribution shift, the reference policy's biases change, but the relative advantage objective provides no mechanism for recalibration — unlike an explicit reward model that learns generalizable features of response quality.

Representational capacity limitations. Im & Li (2024) examined how performance gaps emerge when reward and policy models have different representational capacities. The current paper identifies an additional, orthogonal source of gap: even with unlimited capacity, DPO and RLHF optimize different objectives when Assumption 3.1 is violated.

Value misalignment despite decreasing loss. Shi et al. (2025) revealed that DPO prioritizes statistically distinguishable behaviors over value-aligned ones, potentially causing misalignment despite decreasing loss. This is the closest antecedent to the current paper's core finding. The current paper provides the precise mathematical condition under which this occurs ($\delta_{\pi_\text{ref}} \leq -\Delta r^*/\beta$), characterizes the resulting undesirable convergence space, and offers a constructive solution with provable guarantees.

IPO and other DPO variants. IPO (Azar et al., 2024a) was proposed for better calibration and robustness to noise by replacing the sigmoid loss with a squared loss. However, IPO modifies the loss function rather than the underlying RLHF objective and does not address the conditional equivalence issue — it changes what DPO optimizes rather than ensuring it optimizes the same thing as RLHF. KTO (Ethayarajh et al., 2024) extends DPO to binary feedback but retains the same structural dependence on reference policy quality that creates the assumption violation. ORPO (Hong et al., 2024) combines preference optimization with supervised fine-tuning but inherits DPO's reference-dependent reparameterization. The fundamental gap that the current paper identifies — that DPO's derivation assumes $\pi^*$ respects human preferences — applies to all methods that use the reward reparameterization to eliminate the reward model while claiming equivalence to RLHF.

How This Paper Positions Itself

The paper frames its contribution as addressing a fundamental open problem (explicitly stated in Section 1):

"Under what conditions can DPO be derived through RLHF?"

This is a sharper question than "does DPO work?" or "is DPO equivalent to RLHF?" — it asks for the boundary condition separating regimes where DPO inherits RLHF's guarantees from regimes where it does not. The paper's answer is both precise and constructive:

1. Characterization: DPO and RLHF are equivalent if and only if $\delta_{\pi_\text{ref}} > -\Delta r^*/\beta$ for all preference pairs (Theorem 3.5, Equation 15). This condition depends on both the reference policy's quality ($\delta_{\pi_\text{ref}}$) and the strength of KL regularization ($\beta$).

2. Mechanism: When the condition is violated, the two methods optimize fundamentally different objectives. RLHF optimizes $\mathbb{E}[r] - \beta \text{KL}(\pi \| \pi_\text{ref})$, converging to $\pi^*$ with $\delta_{\pi^*} = \delta_{\pi_\text{ref}} + \Delta r^*/\beta$ (which may be $\leq 0$). DPO optimizes $\mathbb{E}[\log \sigma(\beta(\delta_\pi - \delta_{\pi_\text{ref}}))]$, maximizing $\delta_\pi - \delta_{\pi_\text{ref}}$ and pushing $\delta_\pi$ to $+\infty$ (an objective disconnected from human preferences).

3. Solution: The paper proposes not to abandon the reward-reparameterization approach but to fix its foundations. CPO augments RLHF with an explicit constraint term $\gamma \mathbb{E}[\delta_\pi]$ that directly encourages the policy to prefer human-preferred responses, then derives a DPO-like loss from this augmented objective. E-CPOC goes further, using hard constraints via KKT conditions to guarantee $\delta_\pi \geq \gamma > 0$, establishing provable equivalence to explicitly constrained RLHF under standard statistical learning assumptions.

Positioning relative to RLHF. The paper does not claim RLHF is perfect — it explicitly acknowledges that when $\delta_{\pi_\text{ref}} \leq -\Delta r^*/\beta$, RLHF also produces a policy preferring dispreferred responses (Appendix F). But this is correct behavior for RLHF: it transparently optimizes $\mathbb{E}[r] - \beta \text{KL}$, and if the KL penalty is too strong relative to the reward signal, the optimal trade-off may indeed favor the reference policy's biases over weak reward signals. The problem is not that RLHF can produce misaligned policies — it's that DPO silently optimizes something else entirely while claiming to be equivalent. RLHF's misalignment can be diagnosed (by examining the reward model and KL penalty), while DPO's misalignment is invisible to standard monitoring because the loss function itself has shifted.

Positioning relative to line work on training dynamics. Separately, the paper connects preference learning to the learning-to-rank literature through the margin ranking perspective (Section 5). This is not the primary theoretical contribution but provides geometric intuition for why the failure occurs: DPO implements soft margin ranking with target margin $m = \delta_{\pi_\text{ref}}$, which can be negative. A negative target margin means the optimization can achieve zero loss while still preferring dispreferred responses (any $\delta_{\pi_\theta} > \delta_{\pi_\text{ref}}$ suffices, even if $\delta_{\pi_\theta} < 0$). CPO corrects this by ensuring a non-negative effective margin, guaranteeing the optimization only stops when the policy genuinely prefers the correct response. This perspective connects alignment theory to decades of ranking research and provides an intuitive diagnostic: if your reference policy systematically disfavors winners ($\delta_{\pi_\text{ref}} < 0$ on many pairs), DPO's implicit margin is negative and alignment is not guaranteed.

A practical note on the "surprising" violation rate. The 45.5% violation rate on an instruction-tuned model may seem counterintuitive — why would an instruction-tuned model systematically disfavor preferred responses? The explanation lies in the nature of instruction tuning. An instruction-tuned model learns to produce helpful, harmless responses broadly, but the preference pairs in alignment datasets are often fine-grained: two responses may both be reasonable, but one is marginally better (more concise, more informative, better formatted). The instruction-tuned reference can easily have $\pi_\text{ref}(y_l|x) > \pi_\text{ref}(y_w|x)$ because both are plausible outputs and the reference hasn't been optimized for these specific distinctions — that's precisely why alignment is needed. The assumption violation is not a sign of a broken reference policy but a natural consequence of the alignment problem structure: the reference is being asked to make distinctions it wasn't trained for, and the reward signal at typical $\beta$ values may be too weak to override its default preferences.

3. Technical Approach

3.1 Reader Orientation

This paper develops a constrained optimization framework for aligning language models with human preferences that fixes a fundamental flaw in Direct Preference Optimization (DPO) — the approach adds explicit constraints to the RLHF objective to guarantee that the learned policy always prefers human-chosen responses over rejected ones, then derives DPO-like loss functions from this constrained objective that retain DPO's simplicity (no separate reward model, no RL training) while inheriting the alignment guarantees that standard DPO silently loses when the reference policy is misaligned.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

1. Implicit Assumption Detector (Section 3) — an analytical framework that tests whether the DPO derivation's hidden assumption ($\pi^*$ must prefer $y_w$ over $y_l$) holds for a given reference policy and preference dataset. When violated, DPO and RLHF are proven to optimize fundamentally different objectives.

2. Constrained RLHF Objective (Section 4.2) — the core innovation: the standard RLHF objective ($\mathbb{E}[r] - \beta\text{KL}$) augmented with an explicit preference alignment term $\gamma\mathbb{E}[\delta_\pi]$ that encourages the policy to assign higher log-probability to preferred responses. This creates a new optimal policy that satisfies $\delta_{\pi^*} > 0$ for appropriately chosen $\gamma$, closing the assumption gap.

3. Loss Derivation Pipeline (Sections 4.3–4.5) — a mathematical machinery that converts the constrained RLHF optimal policy condition into a stationary loss function suitable for gradient descent. This involves: (a) deriving the first-order optimality condition for constrained RLHF, (b) reparameterizing through the Bradley-Terry model to eliminate the reward function, and (c) approximating policy-dependent terms with reference-policy-dependent terms to maintain stationarity. Two variants emerge: CPO (soft constraint, constant margin) and E-CPOC (hard constraint via KKT conditions, adaptive margin, no reward model needed).

4. Geometric Interpretation Layer (Section 5) — a reformulation of DPO, CPO, and E-CPOC as soft margin ranking losses, revealing that DPO implements margin ranking with potentially negative target margins ($m = \delta_{\pi_\text{ref}}$) while CPO and E-CPOC guarantee non-negative margins through their constraint terms. This provides intuitive diagnostics and connects alignment theory to the learning-to-rank literature.

Information flows through these components: the user provides a preference dataset and reference policy → the assumption detector checks whether the implicit assumption holds → if violated, the constrained RLHF objective is constructed with an appropriate $\gamma$ → the loss derivation pipeline produces either CPO or E-CPOC loss → gradient descent optimizes the policy parameters → the geometric interpretation layer verifies that margins are non-negative → the trained policy is guaranteed to prefer human-chosen responses.

3.3 Roadmap for the Deep Dive

• First, the implicit assumption and its violation mechanics (Sections 3.1–3.3): understanding exactly what assumption DPO's derivation hides and how its violation transforms the optimization objective. This is the diagnostic foundation — everything else is the treatment.

• Second, the undesirable solution space $\mathcal{U}$ and gradient vanishing (Section 3.2): characterizing where DPO optimization gets trapped and why standard gradient descent cannot escape. This operationalizes the theoretical violation into a concrete optimization pathology.

• Third, the constrained RLHF framework (Section 4.2): the mathematical augmentation of RLHF that eliminates the assumption dependency. Here we derive the first-order optimality condition for the constrained problem — this is the theoretical engine that drives both CPO and E-CPOC.

• Fourth, the CPO loss derivation (Section 4.3): converting the constrained RLHF optimum into a stationary loss function, including the critical approximation step (using $\pi_\text{ref}$ instead of $\pi^*$ in the margin term) and its error bound. This is the practical recipe — the DPO-like loss that practitioners actually implement.

• Fifth, the E-CPOC loss derivation and equivalence proof (Section 4.5, Appendices J, K, L): the hard-constraint variant that eliminates the approximation via KKT conditions and a conservative bound, achieving provable equivalence to explicitly constrained RLHF under standard assumptions. This is the theoretically strongest result.

• Sixth, the margin ranking geometric interpretation (Section 5): reframing all losses as margin ranking to provide intuition for why DPO fails and how CPO/E-CPOC succeed. This is the conceptual bridge that makes the theory accessible.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical analysis paper with a constructive algorithmic contribution. The core idea is that DPO's equivalence to RLHF is conditional on an assumption about reference policy quality (Assumption 3.1), and when this assumption is violated, augmenting the RLHF objective with explicit preference constraints — then deriving DPO-like losses from this augmented objective — restores the alignment guarantees that DPO silently loses.

---

The Implicit Assumption: Where DPO's Derivation Goes Wrong

The paper's first technical contribution is identifying exactly what assumption is buried in DPO's derivation. The standard DPO derivation (Rafailov et al., 2023) proceeds in two steps: (1) solve the RLHF objective analytically to obtain $\pi^*(y|x) = \frac{1}{Z(x)}\pi_\text{ref}(y|x)\exp(r(x,y)/\beta)$, and (2) rearrange this to express the reward difference in terms of policy log-ratios, then substitute into the Bradley-Terry model to obtain a loss depending only on $\pi_\theta$ and $\pi_\text{ref}$.

The rearrangement from step (1) yields:

$r(x, y_w) - r(x, y_l) = \beta\left(\log\frac{\pi^*(y_w|x)}{\pi_\text{ref}(y_w|x)} - \log\frac{\pi^*(y_l|x)}{\pi_\text{ref}(y_l|x)}\right)$

which simplifies using the log-probability ratio notation:

$\delta_\pi(x, y_w, y_l) := \log\pi(y_w|x) - \log\pi(y_l|x)$

to:

$r(x, y_w) - r(x, y_l) = \beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}})$

What this computes: the reward difference between any two responses equals $\beta$ times the difference between (a) how much the optimal policy's log-probability ratio exceeds the reference policy's log-probability ratio.

Why this form: this algebraic identity is exact and unconditional — it follows directly from the closed-form solution to RLHF and holds for any $\pi^*$, $\pi_\text{ref}$, $r$, and $\beta$. There is no assumption here.

The assumption enters at step (2), when this expression is substituted into the Bradley-Terry model:

$p^*(y_w \succ y_l) = \sigma(r^*(x, y_w) - r^*(x, y_l))$

Substituting the reward reparameterization yields:

$p^*(y_w \succ y_l) = \sigma(\beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}))$

The critical observation: for the Bradley-Terry model to be semantically consistent, if $y_w \succ y_l$ (humans prefer $y_w$), then we must have $r^*(y_w) > r^*(y_l)$, which implies $\beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}) > 0$, which implies $\delta_{\pi^*} > \delta_{\pi_\text{ref}}$. However, the Bradley-Terry model does not require $\delta_{\pi^*} > 0$ — it only requires the reward difference to be positive. The substitution $r^*(y_w) - r^*(y_l) \to \beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}})$ is algebraically valid, but semantically, it encodes human preference $y_w \succ y_l$ as $\delta_{\pi^*} > \delta_{\pi_\text{ref}}$, not necessarily $\delta_{\pi^*} > 0$.

Assumption 3.1 (DPO's Implicit Assumption): For all preference data $(x, y_w, y_l) \in \mathcal{D}$ where $y_w \succ y_l$, the RLHF-optimal policy satisfies:

$\pi^*(y_w|x) > \pi^*(y_l|x) \iff \delta_{\pi^*} > 0$

What this says in plain language: the RLHF-optimal policy must assign higher probability to the human-preferred response than to the dispreferred one. This is NOT the same as $\delta_{\pi^*} > \delta_{\pi_\text{ref}}$ (which DPO's loss actually optimizes).

Why this matters: DPO's loss is $-\log\sigma(\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}))$. This loss is minimized when $\delta_{\pi_\theta} \to +\infty$, regardless of whether $\delta_{\pi_\text{ref}}$ is positive or negative. The loss does not directly encourage $\delta_{\pi_\theta} > 0$; it encourages $\delta_{\pi_\theta} \gg \delta_{\pi_\text{ref}}$. If $\delta_{\pi_\text{ref}} = -10$, then $\delta_{\pi_\theta} = -9$ achieves lower loss than $\delta_{\pi_\theta} = -11$, but both violate human preferences ($\delta_{\pi_\theta} < 0$ means $\pi_\theta(y_l|x) > \pi_\theta(y_w|x)$). DPO's loss landscape is shifted by $\delta_{\pi_\text{ref}}$, but the alignment target (preferring $y_w$) is fixed at $\delta_{\pi_\theta} > 0$. When $\delta_{\pi_\text{ref}}$ is negative, these two targets diverge.

---

The Necessary Condition and Why It Fails

Proposition 3.2 derives the necessary condition for Assumption 3.1 to hold:

$\delta_{\pi_\text{ref}} > -\frac{r^*(y_w) - r^*(y_l)}{\beta} \quad \forall (x, y_w, y_l) \in \mathcal{D}$

Derivation: From the RLHF optimal policy's log-ratio:

$\delta_{\pi^*} = \delta_{\pi_\text{ref}} + \frac{r^*(y_w) - r^*(y_l)}{\beta}$

For Assumption 3.1 ($\delta_{\pi^*} > 0$), we need:

$\delta_{\pi_\text{ref}} + \frac{r^*(y_w) - r^*(y_l)}{\beta} > 0$

which rearranges to the condition above. Since $r^*(y_w) - r^*(y_l) > 0$ by the Bradley-Terry model (humans prefer $y_w$), the right-hand side $-\Delta r^*/\beta$ is negative. The condition says: the reference policy can disfavor $y_w$ (have $\delta_{\pi_\text{ref}} < 0$), but not by more than the reward difference (scaled by $1/\beta$) can compensate.

What this reveals operationally: two forces determine whether the assumption holds:

1. Reference policy quality ($\delta_{\pi_\text{ref}}$): if $\pi_\text{ref}$ already prefers $y_w$ ($\delta_{\pi_\text{ref}} > 0$), the condition is automatically satisfied since the RHS is negative.
2. KL regularization strength ($\beta$): larger $\beta$ means weaker reward influence, making the RHS closer to zero and the condition harder to satisfy. As $\beta \to \infty$, the condition requires $\delta_{\pi_\text{ref}} > 0$ for every pair — the reference must already be perfectly aligned.

Why this fails in practice (Section 3.2): when $\delta_{\pi_\text{ref}} \leq -\Delta r^*/\beta$, we have:

$\delta_{\pi^*} = \delta_{\pi_\text{ref}} + \frac{\Delta r^*}{\beta} \leq 0$

This means the RLHF-optimal policy $\pi^*$ actually prefers the dispreferred response ($\pi^*(y_l|x) \geq \pi^*(y_w|x)$). This is not a bug in RLHF — it's the correct optimum of $\mathbb{E}[r] - \beta\text{KL}$ when the KL penalty of deviating from $\pi_\text{ref}$ outweighs the reward gain. But it breaks DPO's derivation because DPO's loss, derived by substituting this $\pi^*$ into the Bradley-Terry model, would encode $y_l$ as preferred over $y_w$ (since $\delta_{\pi^*} \leq 0$), contradicting the ground-truth preference data.

Crucial nuance in the paper's claim: the authors clarify (Appendix D header) that the algebraic substitution in Equation (7) "holds regardless of the sign of $\delta_{\pi^*}$." The problem is not that the substitution is mathematically invalid — it's that the resulting DPO loss optimizes a different objective than RLHF when Assumption 3.1 is violated. The DPO loss $-\log\sigma(\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}))$ is monotonically decreasing in $\delta_{\pi_\theta}$, so it always pushes $\delta_{\pi_\theta}$ upward. But the RLHF optimal $\delta_{\pi^*}$ may be negative. In that case, DPO's optimum ($\delta_{\pi_\theta} \to +\infty$) diverges from RLHF's optimum ($\delta_{\pi^*} < 0$), and the two methods optimize fundamentally different things.

---

The Undesirable Solution Space and Gradient Vanishing

Section 3.2 introduces the paper's operational characterization of DPO's failure mode.

Definition 3.3 (Undesirable Solution Space):

$\mathcal{U} := \{\pi : \delta_\pi < 0 \text{ and } \delta_\pi > \delta_{\pi_\text{ref}}\}$

What this defines: the set of policies that simultaneously (a) violate human preferences ($\delta_\pi < 0$, meaning $\pi(y_l|x) > \pi(y_w|x)$), and (b) satisfy DPO's relative advantage criterion ($\delta_\pi > \delta_{\pi_\text{ref}}$, meaning the policy is "better" than the reference by DPO's metric).

Why this is insidious: for any policy in $\mathcal{U}$, the DPO loss is actually lower than for a policy with the same absolute preference violation but no relative improvement. Specifically, if $\delta_{\pi_\text{ref}} = -10$, then $\pi$ with $\delta_\pi = -5$ achieves $\mathcal{L}_\text{DPO} = -\log\sigma(\beta(-5 + 10)) = -\log\sigma(5\beta) \to 0$ (low loss), despite still preferring $y_l$ over $y_w$. Meanwhile, $\pi$ with $\delta_\pi = -15$ achieves $\mathcal{L}_\text{DPO} = -\log\sigma(\beta(-15 + 10)) = -\log\sigma(-5\beta) \to 5\beta$ (high loss). DPO's loss landscape has a valley in $\mathcal{U}$ — policies can slide deeper into $\mathcal{U}$ while the loss decreases.

Proposition 3.4 (Non-emptiness and Gradient Vanishing): when $\delta_{\pi_\text{ref}} < -\Delta r^*/\beta$, $\mathcal{U}$ is non-empty. Moreover, for any policy $\pi \in \mathcal{U}$, the DPO gradient magnitude becomes progressively weaker as $\delta_\pi$ approaches 0 from below.

Gradient analysis: the DPO gradient (derived in Section 3.2, Equation 13) is:

$\nabla_\theta \mathcal{L}_\text{DPO} = -\beta\mathbb{E}\left[\sigma(-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}})) \cdot (\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x))\right]$

What each term means:

• $\sigma(-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}))$ is the gradient weight — a scalar between 0 and 1 that controls how strongly the gradient pushes $\delta_{\pi_\theta}$ upward.
• $\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x)$ is the gradient direction — it points toward increasing $\delta_{\pi_\theta}$ (making $y_w$ more probable relative to $y_l$).
• The product means the gradient always pushes in the right direction (increase $\delta_{\pi_\theta}$), but the magnitude depends on how much room for improvement remains.

Why gradient vanishing occurs in $\mathcal{U}$: For $\pi \in \mathcal{U}$, we have $\delta_{\pi_\theta} > \delta_{\pi_\text{ref}}$ (relative advantage achieved), so $\delta_{\pi_\theta} - \delta_{\pi_\text{ref}} > 0$, making $-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}) < 0$, and thus $\sigma(-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}})) < 0.5$. As $\delta_{\pi_\theta}$ increases toward 0 (the boundary of preference satisfaction), $\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}$ grows larger (since $\delta_{\pi_\text{ref}}$ is fixed and negative), making $-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}})$ more negative, and $\sigma(\cdot) \to 0$ exponentially. The gradient — which always points in the correct direction — becomes exponentially weaker as the policy approaches the $\delta_\pi = 0$ boundary. The policy can get "stuck" with $\delta_\pi \approx -0.1$, where the gradient is so small that practical optimization cannot push it across to positive territory, despite the loss being very low.

This is NOT a local minimum: the loss continues to decrease as $\delta_\pi \to +\infty$ (the global optimum of DPO). The problem is a practical optimization barrier — the gradient becomes so weak near $\delta_\pi = 0$ that standard gradient descent with finite learning rates and limited iterations cannot traverse the region. The policy appears to converge (loss stops improving significantly), but it has converged to a point in $\mathcal{U}$ rather than a point that respects human preferences.

---

Conditional Equivalence Theorem (Formal Guarantee of Divergence)

Theorem 3.5 (Conditional Equivalence of DPO and RLHF): DPO and RLHF optimize equivalent objectives if and only if:

$\delta_{\pi_\text{ref}}(x, y_w, y_l) > -\frac{r^*(y_w) - r^*(y_l)}{\beta} \quad \forall (x, y_w, y_l) \in \mathcal{D}$

The "if" direction (sufficiency, proved in Appendix D.3): when the condition holds, $\delta_{\pi^*} > 0$ for all pairs (RLHF optimum respects preferences), and the Bradley-Terry substitution $p^* = \sigma(\beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}))$ is semantically consistent — higher probability corresponds to human preference. Under this condition, $\pi^*$ is a global optimum of the DPO loss (proved via strict convexity of $-\log\sigma$ in $\delta_\pi$), so DPO converges to the same policy as RLHF.

The "only if" direction (necessity, proved via contrapositive): when the condition is violated for some pair $(x_0, y_w^0, y_l^0)$, the paper constructs an explicit policy $\pi_\epsilon$ that achieves lower DPO loss than $\pi^*$, proving $\pi^*$ is not the DPO optimum. The construction perturbs $\pi^*$ to increase $\delta_{\pi^*}$ at the violating pair by $2\epsilon$ while keeping other pairs unchanged. Since $-\log\sigma$ is strictly decreasing, this reduces the loss. Therefore, DPO's optimum $\pi_\text{DPO} \neq \pi^*$ — the methods optimize different objectives.

What this theorem establishes: the claimed equivalence in Rafailov et al. (2023) is conditional, not universal. It depends on reference policy quality and KL regularization strength. When the reference is misaligned or $\beta$ is large, DPO silently optimizes a different objective than RLHF, with no indication in the loss curve. This is the paper's primary theoretical contribution.

---

Constrained RLHF: Augmenting the Objective to Fix the Assumption

Section 4.2 introduces the paper's constructive solution. Rather than abandoning the RLHF framework, the authors augment it with an explicit constraint that guarantees the implicit assumption holds.

Definition 4.7 (Constrained RLHF):

$\max_\pi \mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi(\cdot|x)}[r(x, y)] - \beta\text{KL}(\pi\|\pi_\text{ref}) + \gamma\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}[\delta_\pi]$

where $\delta_\pi = \log\pi(y_w|x) - \log\pi(y_l|x)$ is the log-probability ratio.

What each term means:

• $\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi(\cdot|x)}[r(x, y)]$ is the expected reward — encourages generating high-quality responses as judged by the reward model. This is the standard RLHF reward maximization term.
• $-\beta\text{KL}(\pi\|\pi_\text{ref})$ is the KL penalty — prevents the policy from deviating too far from the reference. $\beta > 0$ controls the trade-off: larger $\beta$ means stronger regularization (stay closer to $\pi_\text{ref}$).
• $\gamma\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}[\delta_\pi]$ is the new constraint term — directly encourages the policy to assign higher log-probability to $y_w$ than to $y_l$ across the preference dataset. $\gamma \geq 0$ controls the strength of this preference alignment pressure.

What changes from standard RLHF: when $\gamma = 0$, this reduces exactly to standard RLHF. When $\gamma > 0$, the objective explicitly penalizes policies that disfavor human-preferred responses in log-probability space. The constraint operates on $\delta_\pi$, not on $\pi(y_w|x)$ directly — it's the ratio that matters, not the absolute probabilities.

Why this form fixes the assumption: the additional term $\gamma\mathbb{E}[\delta_\pi]$ directly pushes $\delta_\pi$ upward. For sufficiently large $\gamma$, this term can overcome the KL penalty's tendency to keep $\pi$ close to a misaligned $\pi_\text{ref}$, ensuring $\delta_{\pi^*} > 0$ for the constrained optimum. The paper characterizes exactly how large $\gamma$ needs to be (Theorem 4.9).

Why not just use a larger $\beta$? Larger $\beta$ in standard RLHF weakens the reward signal, making $\pi^*$ stay closer to $\pi_\text{ref}$ — which is the opposite of what we want when $\pi_\text{ref}$ is misaligned. The constraint term is a directed force that specifically targets the preference direction, whereas $\beta$ is an omnidirectional regularizer.

---

The Constrained RLHF First-Order Optimality Condition

Because the constrained RLHF objective adds a non-standard term involving log-probability ratios, its optimum cannot be expressed in closed form (unlike standard RLHF's $\pi^* \propto \pi_\text{ref}\exp(r/\beta)$). The paper instead characterizes it through first-order conditions.

Derivation setup (Theorem 4.8, proved in Appendix D.4): the Lagrangian for a fixed prompt $x$ is:

$\mathcal{L}_x = \sum_y \pi(y|x)r(x, y) - \beta\sum_y \pi(y|x)\log\frac{\pi(y|x)}{\pi_\text{ref}(y|x)} + \sum_y c(x, y)\log\pi(y|x) - \lambda(x)\left(\sum_y \pi(y|x) - 1\right)$

where $c(x, y)$ is the aggregated margin coefficient defined as:

$c(x, y) = \gamma\sum_{(y_w, y_l)\in\mathcal{P}(x)} p(y_w, y_l|x)(\mathbb{I}[y = y_w] - \mathbb{I}[y = y_l])$

What $c(x, y)$ computes: for a given response $y$, it sums $\gamma$ with positive sign for every preference pair where $y$ is the winner, and with negative sign for every pair where $y$ is the loser, weighted by the probability $p(y_w, y_l|x)$ of that pair occurring in the data. If $y$ appears only as $y_w$ in a single pair, $c(x, y) = \gamma$. If $y$ appears only as $y_l$, $c(x, y) = -\gamma$. If $y$ appears in multiple pairs, the contributions sum.

Why this aggregation: the constraint term $\gamma\mathbb{E}_{(x,y_w,y_l)}[\log\pi(y_w|x) - \log\pi(y_l|x)]$ can be regrouped by response — each $\log\pi(y|x)$ term gets a coefficient equal to $\gamma$ times (number of times $y$ appears as winner minus number of times $y$ appears as loser). This regrouping is key because it converts a pairwise constraint into per-response reward augmentations in the first-order conditions.

Taking $\partial\mathcal{L}_x/\partial\pi(y|x) = 0$ yields the first-order optimality condition:

$\beta\log\frac{\pi^*(y|x)}{\pi_\text{ref}(y|x)} = r(x, y) + \frac{c(x, y)}{\pi^*(y|x)} - \beta - \lambda(x)$

What this says: at optimality, the log-ratio between the optimal policy and reference policy, scaled by $\beta$, equals the reward plus a constraint-induced correction $c(x, y)/\pi^*(y|x)$, minus constants. The term $c(x, y)/\pi^*(y|x)$ is the key novelty — it's an inverse-probability-weighted constraint that pushes harder when $\pi^*(y|x)$ is small (making the constraint term larger and the gradient stronger for low-probability responses).

For a preference pair $(y_w, y_l)$ under the single-pair convention (Remark D.1), subtracting the conditions for $y_w$ and $y_l$ yields:

$\beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}) = r(y_w) - r(y_l) + \gamma\left(\frac{1}{\pi^*(y_w|x)} + \frac{1}{\pi^*(y_l|x)}\right)$

What this computes: the reward difference equals $\beta$ times the policy's log-ratio improvement over the reference, minus an adaptive margin $\gamma(1/\pi^*(y_w|x) + 1/\pi^*(y_l|x))$. The subtraction is critical — it means the constrained RLHF optimum satisfies a modified reward-policy relationship where the effective reward is reduced by the constraint margin.

Why this form emerges: compared to standard RLHF's $r(y_w) - r(y_l) = \beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}})$, the constraint adds the term from $c(x, y_w)/\pi^*(y_w|x) - c(x, y_l)/\pi^*(y_l|x) = \gamma/\pi^*(y_w|x) + \gamma/\pi^*(y_l|x)$, since $c(x, y_w) = \gamma$ (winner) and $c(x, y_l) = -\gamma$ (loser). The $1/\pi^*$ factors come from differentiating $\log\pi(y|x)$ in the Lagrangian.

---

From Constrained RLHF to the CPO Loss

Section 4.3 derives the practical loss function. Rearranging the paired first-order condition:

$r(y_w) - r(y_l) = \beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}) - \tilde{\gamma}^*(x, y_w, y_l)$

where the adaptive margin is:

$\tilde{\gamma}^*(x, y_w, y_l) = \gamma\left(\frac{1}{\pi^*(y_w|x)} + \frac{1}{\pi^*(y_l|x)}\right)$

Key insight: substituting this into the Bradley-Terry model $p^*(y_w \succ y_l) = \sigma(r^*(y_w) - r^*(y_l))$ gives:

$p^*(y_w \succ y_l) = \sigma\left(\beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}) - \tilde{\gamma}^*(x, y_w, y_l)\right)$

The approximation problem: $\tilde{\gamma}^*$ depends on $\pi^*$ (the optimal policy), which we don't know during training. If we replace $\pi^*$ with $\pi_\theta$ (the current policy), the loss becomes non-stationary — the target margin changes as $\theta$ changes, violating standard assumptions for gradient descent convergence.

The CPO approximation (Equation 24): replace $\pi^*$ with $\pi_\text{ref}$ in the margin term:

$\tilde{\gamma}^*(x, y_w, y_l) \approx \tilde{\gamma}_\text{ref}(x, y_w, y_l) = \gamma\left(\frac{1}{\pi_\text{ref}(y_w|x)} + \frac{1}{\pi_\text{ref}(y_l|x)}\right)$

Why this is reasonable: the KL penalty in RLHF keeps $\pi^*$ close to $\pi_\text{ref}$ (Proposition H.2 quantifies this: under bounded rewards and moderate $\gamma$, $|\pi^*(y|x) - \pi_\text{ref}(y|x)| = O(\sqrt{\tilde{R}_\text{max}/\beta})$). The approximation error $|\tilde{\gamma}^* - \tilde{\gamma}_\text{ref}|$ is bounded by $O(\gamma\sqrt{\tilde{R}_\text{max}/\beta}/q_0^2)$ where $q_0 = p_\text{min}e^{-2R_\text{max}/\beta}$ is a lower bound on policy probabilities. The error vanishes as $\beta \to \infty$ (strong regularization). Critically, using $\pi_\text{ref}$ makes $\tilde{\gamma}_\text{ref}$ a constant with respect to $\theta$, yielding a stationary loss.

The CPO loss (Equation 25):

$\mathcal{L}_\text{CPO}(\pi_\theta) = -\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}) - \tilde{\gamma}_\text{ref}(x, y_w, y_l)\right)\right]$

What this computes: standard binary cross-entropy for preference prediction, but with the logit $\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}})$ reduced by a sample-adaptive margin $\tilde{\gamma}_\text{ref}$. The margin is largest for pairs where $\pi_\text{ref}$ assigns low probability to both responses (hard pairs) and smallest for high-confidence pairs.

Why this fixes DPO: when $\delta_{\pi_\text{ref}}$ is negative (reference disfavors $y_w$), DPO's logit $\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}})$ can be positive even when $\delta_{\pi_\theta}$ is slightly negative — DPO accepts $\delta_{\pi_\theta} = -1$ if $\delta_{\pi_\text{ref}} = -10$ because the logit is $9\beta > 0$. CPO's margin $\tilde{\gamma}_\text{ref}$ shifts the logit downward by a data-dependent amount: $\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}) - \tilde{\gamma}_\text{ref} > 0$ requires $\delta_{\pi_\theta} > \delta_{\pi_\text{ref}} + \tilde{\gamma}_\text{ref}/\beta$. For sufficiently large $\gamma$, this threshold exceeds 0, ensuring the loss only decreases when the policy genuinely prefers $y_w$.

The CPO gradient (Equation 28):

$\nabla_\theta\mathcal{L}_\text{CPO} = -\beta\mathbb{E}\left[\sigma\left(\beta(\delta_{\pi_\text{ref}} - \delta_{\pi_\theta}) + \tilde{\gamma}_\text{ref}\right) \cdot (\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x))\right]$

Key difference from DPO's gradient: the sigmoid weight argument is shifted by $+\tilde{\gamma}_\text{ref}$. When $\delta_{\pi_\text{ref}}$ is negative and $\delta_{\pi_\theta}$ is stuck near 0, DPO's weight $\sigma(-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}})) \approx \sigma(-\beta(0 - (-10))) = \sigma(-10\beta) \approx 0$ (gradient vanishes). CPO's weight is $\sigma(\beta(-10 - 0) + \tilde{\gamma}_\text{ref}) = \sigma(-10\beta + \tilde{\gamma}_\text{ref})$, which remains substantial if $\tilde{\gamma}_\text{ref}$ is large enough. The margin term explicitly counteracts the gradient vanishing, enabling escape from $\mathcal{U}$.

---

The Absolute Advantage Guarantee (Theorem 4.9)

The paper proves that CPO can guarantee $\delta_{\pi^*} > 0$ (the policy prefers $y_w$) for all preference pairs, provided $\gamma$ is chosen appropriately.

Lemma D.3 (Sample-level guarantee): for a single preference pair, if:

$\gamma > \beta \cdot \frac{\max\left\{0, -\delta_{\pi_\text{ref}}(x, y_w, y_l) - \frac{r^*(y_w) - r^*(y_l)}{\beta}\right\}}{\frac{1}{\pi_\text{ref}(y_w|x)} + \frac{1}{\pi_\text{ref}(y_l|x)}}$

then CPO guarantees $\delta_{\pi^*_\text{CPO}}(x, y_w, y_l) > 0$.

What the numerator measures: how much the reference policy's misalignment ($-\delta_{\pi_\text{ref}}$) exceeds the reward correction ($\Delta r^*/\beta$). If $-\delta_{\pi_\text{ref}} \leq \Delta r^*/\beta$, the max is 0 — no correction needed. Otherwise, the numerator is the shortfall that $\gamma$ must compensate.

What the denominator does: scales the required $\gamma$ inversely with reference policy confidence. When $\pi_\text{ref}(y_w|x)$ and $\pi_\text{ref}(y_l|x)$ are both small (say, $10^{-5}$ each), the denominator is $\sim 2 \times 10^5$ — the required $\gamma$ is tiny because the constraint term $\gamma/\pi$ amplifies the signal for low-probability responses. When both are large (say, 0.1 each), the denominator is $\sim 20$ — more $\gamma$ is needed because the constraint's per-unit effect is weaker.

Theorem 4.9 (Dataset-level guarantee): setting:

$\gamma^* = \max_{(x, y_w, y_l) \in \mathcal{D}} \beta \cdot \frac{\max\left\{0, -\delta_{\pi_\text{ref}}(x, y_w, y_l) - \frac{r^*(y_w) - r^*(y_l)}{\beta}\right\}}{\frac{1}{\pi_\text{ref}(y_w|x)} + \frac{1}{\pi_\text{ref}(y_l|x)}}$

and choosing any $\gamma \geq \gamma^*$ guarantees $\delta_{\pi^*_\text{CPO}} > 0$ for all pairs in the dataset.

What this means practically: $\gamma^*$ is the worst-case required constraint strength — computed as the maximum over all pairs of the per-pair requirement. It depends on: (a) how misaligned the reference is ($\delta_{\pi_\text{ref}}$), (b) how strong the reward signal is ($\Delta r^*/\beta$), and (c) how confident the reference is ($\pi_\text{ref}$ probabilities). The paper notes that in practice, $\gamma$ is treated as a hyperparameter tuned on validation data, not computed exactly from $\gamma^*$ (which would require knowing $r^*$).

Theorem 4.10 (CPO Avoids Pathological Convergence): when $\gamma \geq \gamma^*$, CPO does not converge to the undesirable solution space $\mathcal{U}$. The proof is direct: since $\delta_{\pi^*_\text{CPO}} > 0$, $\pi^*_\text{CPO} \notin \mathcal{U}$ (which requires $\delta_\pi < 0$). Under standard smoothness assumptions, gradient descent converges to $\pi^*_\text{CPO}$, avoiding $\mathcal{U}$ entirely.

Why this is more than just a larger margin: the adaptive margin $\tilde{\gamma}_\text{ref} = \gamma(1/\pi_\text{ref}(y_w|x) + 1/\pi_\text{ref}(y_l|x))$ is NOT a constant — it scales with $1/\pi_\text{ref}$. This means CPO automatically applies stronger correction to pairs where $\pi_\text{ref}$ is uncertain (low probabilities), which is precisely where the reference is most likely to be misaligned. For pairs where $\pi_\text{ref}$ is confident and already correct, the margin is small, avoiding unnecessary regularization. This adaptive behavior emerges from the first-order conditions of constrained RLHF, not from heuristic design.

---

E-CPOC: Hard Constraints and Provable Equivalence Without a Reward Model

Section 4.5 introduces the paper's strongest theoretical result: a variant that enforces preference alignment through hard constraints and achieves provable equivalence to explicitly constrained RLHF under standard assumptions, without requiring a reward model.

Definition 4.11 (Explicitly Constrained RLHF):

$\max_\pi \mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi(\cdot|x)}[r(x, y)] - \beta\text{KL}(\pi\|\pi_\text{ref})$

$\text{s.t. } \delta_\pi(x, y_w, y_l) \geq \gamma(x, y_w, y_l) \quad \forall (x, y_w, y_l) \in \mathcal{D}$

Key difference from CPO: CPO uses a soft penalty term $\gamma\mathbb{E}[\delta_\pi]$ that encourages large $\delta_\pi$. E-CPOC uses hard inequality constraints $\delta_\pi \geq \gamma$ that require the policy to prefer $y_w$ over $y_l$ by at least margin $\gamma$ in log-probability space. The constraint is enforced via Lagrange multipliers in the KKT conditions, not via an additive term in the objective.

Theorem 4.12 (Log-probability Ratio of Optimal Policy for EC-RLHF):

$\delta_{\pi^*} = \delta_{\pi_\text{ref}} + \frac{\Delta r}{\beta} + \Phi(\delta_{\pi_\text{ref}}, \Delta r; \gamma, \tau)$

where the adaptive margin function is:

$\Phi(\delta_{\pi_\text{ref}}, \Delta r; \gamma, \tau) = \frac{1}{\tau}\log\left(1 + \exp\left(\tau\left(\gamma - \delta_{\pi_\text{ref}} - \frac{\Delta r}{\beta}\right)\right)\right)$

What $\Phi$ computes: a smooth approximation to $\max\{0, \gamma - \delta_{\pi_\text{ref}} - \Delta r/\beta\}$. When the constraint is satisfied without enforcement ($\delta_{\pi_\text{ref}} + \Delta r/\beta \geq \gamma$), $\Phi \approx 0$ (no correction needed). When the constraint would be violated, $\Phi \approx \gamma - \delta_{\pi_\text{ref}} - \Delta r/\beta$ (exactly the amount needed to satisfy $\delta_{\pi^*} \geq \gamma$). The parameter $\tau > 0$ controls smoothness: larger $\tau$ makes the transition sharper, approaching a hard ReLU as $\tau \to \infty$.

Why this form emerges (derived in Appendix J): the KKT conditions for the constrained optimization problem yield Lagrange multipliers $\mu^*$ for each constraint. Complementary slackness ($\mu^*(\delta_{\pi^*} - \gamma) = 0$) means $\mu^* > 0$ only when the constraint is binding ($\delta_{\pi^*} = \gamma$). The effective margin contribution $\mu^*(1/\pi^*(y_w|x) + 1/\pi^*(y_l|x))$ admits a closed form independent of $\pi^*$ (Proposition J.6): it equals $\beta\max\{0, \gamma - \delta_{\pi_\text{ref}} - \Delta r/\beta\}$. The softplus $\Phi$ is the smooth relaxation of this hard max.

The $\Delta r$ dependence problem: $\Phi$ depends on the true reward difference $\Delta r = r^*(y_w) - r^*(y_l)$, which is unknown without a reward model. This would defeat the purpose of DPO-like methods.

The E-CPOC solution (Conservative Bound): exploit the monotonicity of $\Phi$ in $\Delta r$ (Proposition E.1):

$\frac{\partial\Phi}{\partial(\Delta r)} = -\frac{1}{\beta}\sigma\left(\tau\left(\gamma - \delta_{\pi_\text{ref}} - \frac{\Delta r}{\beta}\right)\right) \leq 0$

Since $\Phi$ is non-increasing in $\Delta r$, and human preference data satisfies $\Delta r^* > 0$ (by the Bradley-Terry model), the maximum value of $\Phi$ over all possible $\Delta r^*$ is achieved at $\Delta r \to 0^+$:

$\Phi_\text{cons}(\delta_{\pi_\text{ref}}) := \Phi(\delta_{\pi_\text{ref}}, 0) = \frac{1}{\tau}\log\left(1 + \exp\left(\tau(\gamma - \delta_{\pi_\text{ref}})\right)\right) \geq \Phi(\delta_{\pi_\text{ref}}, \Delta r^*)$

for all $\Delta r^* > 0$.

What this means operationally: using $\Phi_\text{cons}$ (which assumes $\Delta r = 0$, the worst case) instead of $\Phi(\delta_{\pi_\text{ref}}, \Delta r^*)$ (which uses the unknown true reward) produces a larger margin correction. This makes the resulting policy more conservative — it enforces a stronger preference for $y_w$ than strictly necessary. Critically, $\Phi_\text{cons}$ requires no knowledge of rewards, maintaining DPO's key advantage.

The E-CPOC loss (Equation 33):

$\mathcal{L}_\text{E-CPOC}(\pi_\theta) = -\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}) - \beta\Phi_\text{cons}(\delta_{\pi_\text{ref}})\right)\right]$

What this computes: same structure as CPO but with the adaptive margin $\beta\Phi_\text{cons}(\delta_{\pi_\text{ref}})$ instead of $\tilde{\gamma}_\text{ref} = \gamma(1/\pi_\text{ref}(y_w|x) + 1/\pi_\text{ref}(y_l|x))$. The margin is now a function of $\delta_{\pi_\text{ref}}$ alone, not of individual response probabilities.

Key properties of $\Phi_\text{cons}$ (Propositions E.4, L.16):

1. Non-negativity: $\Phi_\text{cons}(\delta_{\pi_\text{ref}}) \geq 0$ for all $\delta_{\pi_\text{ref}}$ — the margin never reduces the logit below the DPO baseline.
2. Monotonicity: $\partial\Phi_\text{cons}/\partial\delta_{\pi_\text{ref}} = -\sigma(\tau(\gamma - \delta_{\pi_\text{ref}})) < 0$ — as the reference becomes better aligned (larger $\delta_{\pi_\text{ref}}$), the margin decreases.
3. Sample-adaptive behavior:
   • For difficult samples ($\delta_{\pi_\text{ref}} \ll 0$): $\Phi_\text{cons} \approx \gamma - \delta_{\pi_\text{ref}}$ (strong correction, margin grows with misalignment).
   • For easy samples ($\delta_{\pi_\text{ref}} \gg \gamma$): $\Phi_\text{cons} \approx 0$ (no correction needed, recovers DPO).
   • For neutral samples ($\delta_{\pi_\text{ref}} \approx \gamma$): $\Phi_\text{cons} \approx \log 2 / \tau$ (smooth transition).

What E-CPOC achieves that CPO doesn't: E-CPOC's margin $\beta\Phi_\text{cons}$ is derived from the KKT conditions of the hard-constrained problem and uses a conservative bound to eliminate reward dependence. CPO's margin $\tilde{\gamma}_\text{ref}$ comes from a soft-penalty formulation with an approximation ($1/\pi^* \approx 1/\pi_\text{ref}$). The hard-constraint approach eliminates the approximation error entirely (Appendix J, Proposition J.6): the effective margin $\mu^*(1/\pi^*(y_w|x) + 1/\pi^*(y_l|x))$ admits a closed form $\beta\max\{0, \gamma - \delta_{\pi_\text{ref}} - \Delta r/\beta\}$ where the $1/\pi^*$ factors are absorbed by the Lagrange multipliers — they never need to be approximated because the KKT conditions relate $\mu^*$ and $\pi^*$ in a way that makes their product independent of $\pi^*$.

---

The E-CPOC Equivalence Theorem (Theorem L.17)

The paper's most technically sophisticated result establishes that E-CPOC is provably equivalent to explicitly constrained RLHF under four standard assumptions (Section 4.1):

Assumption 4.1 (Bradley-Terry Model): $p^*(y_w \succ y_l|x) = \sigma(r^*(x, y_w) - r^*(x, y_l))$ — preferences follow a logistic model with latent rewards.

Assumption 4.2 (Approximate Realizability): the policy class can approximate the target log-probability ratios to within $\epsilon_\text{approx}$.

Assumption 4.3 (Finite-Sample Data): the dataset has $N$ i.i.d. samples with statistical error $\epsilon_\text{stat}(N) = O(1/\sqrt{N})$.

Assumption 4.4 ($\ell_2$-$\delta$-Proximity): the returned policy satisfies $\frac{1}{N}\sum_{i=1}^N (\delta_{\hat{\pi}, i} - \delta_{\pi^*_\text{MLE}, i})^2 \leq \epsilon^2_\text{opt,2}$ — the mean-square error in $\delta$-space between the trained policy and the class-optimal MLE policy is bounded.

What makes this assumption set "mild" (the paper's claim): all four are standard in statistical learning. No global optimality, exact realizability, or pointwise ($\ell_\infty$) optimization conditions are required. The $\ell_2$-$\delta$-proximity condition (Assumption 4.4) is strictly weaker than the pointwise condition $\max_i |\delta_{\hat{\pi}, i} - \delta^*_i| \leq \epsilon_\text{opt}$ — it permits larger deviations on a few difficult pairs as long as the average is controlled. Moreover, Assumption 4.4 admits a verifiable sufficient condition via the Loss-to-Delta bridge (Proposition L.3): under Assumption 4.5 (non-degenerate preferences, $\kappa_0 > 0$), a training loss gap $\epsilon_\text{loss}$ implies $\ell_2$-$\delta$-proximity with $\epsilon_\text{opt,2} = \sqrt{2\epsilon_\text{loss}/(\beta^2\kappa_0)}$, and this bound is independent of $N$.

Theorem L.17 (E-CPOC Upper Bound Equivalence): the E-CPOC optimal policy satisfies (up to combined error $\epsilon_\text{approx} + \epsilon_\text{opt} + L_\sigma^{-1}\epsilon_\text{stat}$, where $\epsilon_\text{opt} = \sqrt{N}\epsilon_\text{opt,2}$ via Lemma L.1):

$\delta_{\pi^*_\text{E-CPOC}} = \delta_{\pi_\text{ref}} + \frac{\Delta r^*}{\beta} + \Phi_\text{cons}(\delta_{\pi_\text{ref}})$

which implies:

$\delta_{\pi^*_\text{E-CPOC}} \geq \delta_{\pi^*_\text{EC-RLHF}}(\Delta r^*)$

for any true reward difference $\Delta r^* > 0$.

What this equality means: the E-CPOC policy can be decomposed as the worst-case EC-RLHF policy (with $\Delta r = 0$) plus the true reward gap $\Delta r^*/\beta$. Since $\Delta r^*/\beta > 0$, E-CPOC is strictly more conservative than EC-RLHF at $\Delta r = 0$, which is itself the most conservative possible EC-RLHF solution.

Absolute advantage from the theorem: using the strict inequality $\Phi_\text{cons}(\delta_{\pi_\text{ref}}) > \gamma - \delta_{\pi_\text{ref}}$ (softplus strictly dominates identity), we get:

$\delta_{\pi^*_\text{E-CPOC}} = \delta_{\pi_\text{ref}} + \frac{\Delta r^*}{\beta} + \Phi_\text{cons}(\delta_{\pi_\text{ref}}) > \delta_{\pi_\text{ref}} + \frac{\Delta r^*}{\beta} + (\gamma - \delta_{\pi_\text{ref}}) = \gamma + \frac{\Delta r^*}{\beta} > \gamma > 0$

This bound is exact (non-asymptotic) and holds for any $\gamma > 0$ — no minimum threshold is theoretically required. The recommendation to set $\gamma \geq \gamma^*_\text{cons} := \max_{(x, y_w, y_l)}\{-\delta_{\pi_\text{ref}}(x, y_w, y_l)\}$ ensures the constraint is meaningful (binding for the hardest samples), but any positive $\gamma$ guarantees $\delta_{\pi^*} > 0$.

The Loss-to-Delta Bridge (Proposition L.3, Corollary L.18): the paper provides a verifiable pathway from training diagnostics to the equivalence guarantee. The bridge shows:

$\epsilon_\text{loss} := \mathcal{L}(\hat{\pi}) - \inf_\theta \mathcal{L}(\pi_\theta) \implies \epsilon_\text{opt,2} = \sqrt{\frac{2\epsilon_\text{loss}}{\beta^2\kappa_0}}$

where $\kappa_0 = \min_i \sigma(g_i(\delta^*_i))(1 - \sigma(g_i(\delta^*_i))) > 0$ is a curvature measure (how decisive the preferences are at the optimum). The practitioner monitors the training loss gap $\epsilon_\text{loss}$ (directly observable), and under the mild condition that preferences are non-degenerate ($\kappa_0 > 0$), the $\ell_2$-$\delta$-proximity bound follows automatically. The bridge constant $\kappa_0$ is N-independent; the $\sqrt{N}$ factor appears only in the $\ell_2 \to \ell_\infty$ conversion (Lemma L.1), making the dependence on dataset size transparent.

Why this matters for deployment: the paper doesn't just prove that an ideal optimizer would achieve equivalence — it provides a practical criterion (training loss convergence) that, when satisfied, guarantees the equivalence bound holds for the actual returned policy. This bridges the gap between asymptotic theory and finite-training reality.

---

Implementation: The CPO and E-CPOC Algorithms

Algorithm 1 (CPO) — from Section 4.3 and the algorithmic listing:

1. Precompute for each preference pair in the dataset:
   • $\delta^{(i)}_\text{ref} = \log\pi_\text{ref}(y_w|x) - \log\pi_\text{ref}(y_l|x)$ (reference policy's log-ratio)
   • $\tilde{\gamma}^{(i)}_\text{ref} = \gamma \cdot (1/\pi_\text{ref}(y_w|x) + 1/\pi_\text{ref}(y_l|x))$ (adaptive margin)
2. Training loop: for each batch:
   • Compute $\delta_\theta = \log\pi_\theta(y_w|x) - \log\pi_\theta(y_l|x)$ (current policy's log-ratio)
   • Compute logits = $\beta(\delta_\theta - \delta^{(i)}_\text{ref}) - \tilde{\gamma}^{(i)}_\text{ref}$ (margin-adjusted preference score)
   • Compute loss = $-\log\sigma(\text{logits})$ (binary cross-entropy)
   • Gradient descent on $\theta$

Algorithm 2 (E-CPOC) — from Appendix E.2:

1. Precompute for each preference pair:
   • $\delta^{(i)}_\text{ref} = \log\pi_\text{ref}(y_w|x) - \log\pi_\text{ref}(y_l|x)$
   • $\Phi^{(i)} = \frac{1}{\tau}\log(1 + \exp(\tau(\gamma - \delta^{(i)}_\text{ref})))$ (conservative adaptive margin function)
   • $\Psi^{(i)} = \beta\Phi^{(i)}$ (scaled margin)
2. Training loop: for each batch:
   • Compute $\delta_\theta$ as above
   • Compute logits = $\beta(\delta_\theta - \delta^{(i)}_\text{ref}) - \Psi^{(i)}$
   • Compute loss = $-\log\sigma(\text{logits})$
   • Gradient descent

Computational overhead: both algorithms have essentially identical cost to DPO. Standard DPO already requires precomputing $\delta^{(i)}_\text{ref}$ via a forward pass over the dataset. CPO/E-CPOC reuse the same forward pass to additionally compute the margin term (two scalar divisions and one addition per sample for CPO; one softplus evaluation per sample for E-CPOC). During training, the only difference from DPO is subtracting the precomputed margin from the logits — one extra scalar operation per sample. No additional forward/backward passes, no reward model inference, no architectural changes.

---

Geometric Interpretation: Preference Learning as Margin Ranking

Section 5 reframes all losses through the lens of soft margin ranking, providing geometric intuition.

Standard margin ranking loss (hinge):

$\mathcal{L}_\text{hinge}(s_w, s_l; m) = \max(0, m - (s_w - s_l))$

where $s_w, s_l$ are scores for winner and loser, and $m \geq 0$ is the target margin. The loss is zero when $s_w - s_l \geq m$ (winner outscores loser by at least $m$); otherwise it incurs a linear penalty.

Proposition 5.1 (DPO as Soft Margin Ranking): in the high-temperature limit $\beta \to \infty$:

$\lim_{\beta\to\infty} \frac{1}{\beta}\mathcal{L}_\text{DPO}(\pi_\theta) = \max(0, \delta_{\pi_\text{ref}} - \delta_{\pi_\theta})$

What this reveals: DPO implements margin ranking with target margin $m = \delta_{\pi_\text{ref}}$ — and this target can be negative. When $\delta_{\pi_\text{ref}} = -10$, the loss is zero whenever $\delta_{\pi_\theta} > -10$, which includes policies that prefer $y_l$ ($\delta_{\pi_\theta} < 0$). The "margin" is not a requirement that $\delta_{\pi_\theta}$ exceed 0, but only that it exceed $\delta_{\pi_\text{ref}}$. A negative target margin means the optimization can succeed without achieving alignment.

Theorem 5.2 (CPO as Corrected Soft Margin Ranking):

$\lim_{\beta\to\infty} \frac{1}{\beta}\mathcal{L}_\text{CPO}(\pi_\theta) = \max\left(0, \delta_{\pi_\text{ref}} + \frac{2\gamma}{\beta} - \delta_{\pi_\theta}\right)$

with guaranteed non-negative margin $m^*_\text{eff} = \delta_{\pi_\text{ref}} + 2\gamma^*/\beta \geq 0$ when $\gamma \geq \gamma^*$ as in Theorem 4.9.

What changed: the target margin is shifted upward by $2\gamma/\beta$. When $\gamma$ is chosen according to Theorem 4.9, this shift is sufficient to make the target margin non-negative for all pairs, ensuring the optimization only stops when $\delta_{\pi_\theta} > 0$.

Theorem 5.3 (E-CPOC as Adaptive Margin Ranking):

$\lim_{\beta\to\infty} \frac{1}{\beta}\mathcal{L}_\text{E-CPOC}(\pi_\theta) = \max(0, \delta_{\pi_\text{ref}} + \Phi_\text{cons}(\delta_{\pi_\text{ref}}) - \delta_{\pi_\theta})$

with guaranteed non-negative margin $m^*(\delta_{\pi_\text{ref}}) = \delta_{\pi_\text{ref}} + \Phi_\text{cons}(\delta_{\pi_\text{ref}})$.

What's different from CPO: the margin is sample-adaptive — it depends on $\delta_{\pi_\text{ref}}$. For difficult samples ($\delta_{\pi_\text{ref}} \ll 0$): $m^* \approx \gamma > 0$ (flat positive margin). For easy samples ($\delta_{\pi_\text{ref}} \gg 0$): $m^* \approx \delta_{\pi_\text{ref}} > 0$ (margin tracks reference quality). This adaptive behavior means E-CPOC applies strong correction precisely where needed and minimal correction where the reference is already aligned.

The unified geometric picture: all three methods can be understood as soft margin ranking in log-probability space. DPO's margin = $\delta_{\pi_\text{ref}}$ (reference-dependent, can be negative). CPO's margin = $\delta_{\pi_\text{ref}} + 2\gamma/\beta$ (reference plus constant shift). E-CPOC's margin = $\delta_{\pi_\text{ref}} + \Phi_\text{cons}(\delta_{\pi_\text{ref}})$ (reference plus adaptive shift that guarantees non-negativity). The softplus function $\log(1 + e^x)$ provides smooth, differentiable approximations to the hard max in hinge loss, with $\beta$ controlling the transition sharpness. This perspective connects preference optimization to decades of ranking research and provides an immediate diagnostic: check whether your reference policy's $\delta_{\pi_\text{ref}}$ is negative for many pairs. If so, DPO's implicit margin is negative and alignment is not guaranteed.

---

Training Details and Hyperparameters

The paper's experiments (Section 6) use:

• Base model: Llama-3-8B-Instruct (Dubey et al., 2024)
• Preference dataset: princeton-nlp/llama3-ultrafeedback-armorm
• Hyperparameters for CPO: $\beta = 0.1$ (temperature, same as DPO baseline), $\gamma = 0.25$ (chosen via sensitivity analysis in Appendix A.3, Table 5). The sensitivity analysis shows robust performance for $\gamma \in [0.2, 0.4]$ (win rate 26–28%, length-controlled 31–34%), with peak at $\gamma = 0.25$.
• Sensitivity of $\gamma$: performance drops notably below 0.2 (WR 20.87% at $\gamma = 0.15$), confirming that insufficient margin correction indeed hurts performance — consistent with the theory that weak $\gamma$ fails to escape $\mathcal{U}$.

The Appendix A.1 measurement (Figure 1) quantifies the assumption violation on this exact setup: 45.5% of pairs violate Assumption 3.1 ($\beta = 0.1$). The reference's $\delta_{\pi_\text{ref}}$ has mean 1.35 and standard deviation 46.69, while $\Delta r^*/\beta$ has mean 0.20 and standard deviation 0.16. The reward correction is two orders of magnitude smaller than the reference policy's variability, explaining why nearly half of all pairs fall into the violation regime.

4. Key Insights and Innovations

Innovation 1: Reframing DPO's Equivalence from "Algebraic Identity" to "Conditional Semantic Guarantee"

The dominant narrative since Rafailov et al. (2023) has been that DPO "is" RLHF — that the reward reparameterization provides an algebraic shortcut that eliminates the reward model while preserving the optimization objective. This paper makes a fundamental conceptual move: it separates the algebraic validity of the reparameterization from its semantic validity. The substitution $r^*(y_w) - r^*(y_l) = \beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}})$ is algebraically exact — it follows from the closed-form RLHF solution with no assumptions. But encoding this as the Bradley-Terry probability $\sigma(\beta(\delta_{\pi^*} - \delta_{\pi_\text{ref}}))$ implicitly asserts that $\pi^*$ encodes human preference in its log-probability ratio. The paper identifies this as a separate claim — one that DPO's derivation never verifies.

What makes this reframing distinctive is that it converts what appeared to be a mathematical identity into a falsifiable condition. Before this work, questioning DPO's equivalence meant questioning the derivation itself (which is mathematically sound). The paper shows the derivation is correct but incomplete — it's missing a premise. This is a more precise critique than prior work that identified discrepancies between DPO and RLHF (Fisch et al., 2024; Lin et al., 2024; Shi et al., 2025). Those papers documented empirical or theoretical gaps; this paper identifies the specific condition under which the gap appears and proves that it's the only condition (Theorem 3.5: the "if and only if"). The contribution is not that DPO sometimes fails — several papers had shown that — but that the failure is systematic, predictable, and structural: it occurs exactly when $\delta_{\pi_\text{ref}} \leq -\Delta r^*/\beta$, a condition that can be checked on any dataset and reference policy.

The evidence that makes this more than a theoretical curiosity is the 45.5% violation rate measured on Llama-3-8B-Instruct (Figure 1, Appendix A.1). This is not a corner case or a toy example — it's a standard model and dataset in production alignment pipelines. The field has been deploying DPO under the assumption of equivalence while nearly half of the training examples violate the condition required for that equivalence to hold. The paper's reframing changes the question from "does DPO work?" to "under what conditions does DPO inherit RLHF's alignment guarantees, and are those conditions met in practice?" This is a sharper, more actionable question that should influence how practitioners select and monitor alignment methods.

Innovation 2: The Undesirable Solution Space $\mathcal{U}$ as an Operational Diagnostic for Silent Misalignment

The paper invents a new concept — the undesirable solution space $\mathcal{U} = \{\pi : \delta_\pi < 0 \text{ and } \delta_\pi > \delta_{\pi_\text{ref}}\}$ — that transforms the theoretical violation of Assumption 3.1 into an operationally measurable pathology. Prior work on DPO's failures (Fisch et al., 2024; Shi et al., 2025) characterized problems in terms of loss behavior (overfitting, implicit reward divergence) or output quality (degenerate policies). These are symptoms. $\mathcal{U}$ is the mechanism: it defines the exact region in policy space where DPO's loss decreases but human preferences are violated.

What makes $\mathcal{U}$ a genuine innovation rather than a restatement of the assumption violation is that it provides a training-time diagnostic. The paper shows that for policies in $\mathcal{U}$, the DPO loss can be arbitrarily low (approaching 0 as $\delta_\pi - \delta_{\pi_\text{ref}}$ grows) while the policy still prefers dispreferred responses. A practitioner monitoring the loss curve sees convergence; the geometry of $\mathcal{U}$ reveals that this convergence may be toward a region of systematic preference violation. The training dynamics visualization in Figure 2 (Appendix A.2) operationalizes this: the paper tracks the fraction of training samples in $\mathcal{U}$ over training steps, showing that DPO policies enter $\mathcal{U}$ and become trapped there (fraction plateaus at ~30–40%) while CPO policies escape $\mathcal{U}$ (fraction rapidly drops to near zero). This is not a theoretical possibility — it's a measured training dynamic.

The gradient vanishing analysis (Proposition 3.4) provides the mechanistic explanation for why $\mathcal{U}$ is an attractor. The DPO gradient weight $\sigma(-\beta(\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}))$ becomes exponentially small as $\delta_{\pi_\theta}$ approaches 0 from below, because $\delta_{\pi_\theta} - \delta_{\pi_\text{ref}}$ grows large (since $\delta_{\pi_\text{ref}}$ is fixed and negative). The gradient always points in the correct direction — it's not that DPO is optimizing the wrong thing locally — but the magnitude becomes so small near the $\delta_\pi = 0$ boundary that practical optimization cannot cross it. This explains the empirical observation, noted in multiple prior works, that DPO-trained models sometimes show limited improvement or even regression on certain preference distinctions: the models may be stuck in $\mathcal{U}$ on those specific pairs, with vanishing gradients preventing escape.

The significance of this diagnostic extends beyond DPO. Any preference optimization method that optimizes relative advantage over a reference policy — rather than absolute preference satisfaction — is potentially susceptible to a $\mathcal{U}$-like pathology. The concept provides a vocabulary and a measurement protocol (fraction of pairs with $\delta_\pi < 0$ yet $\delta_\pi > \delta_{\pi_\text{ref}}$) for evaluating whether a given alignment method successfully escapes the relative-advantage trap.

Innovation 3: Constructive Solution Through Constrained RLHF — Proving That the Fix Preserves DPO's Simplicity

The paper could have stopped at identifying the assumption violation and characterizing $\mathcal{U}$. That would have been a valuable theoretical contribution: a precise statement of when and why DPO's equivalence fails. What elevates the work is the constructive solution — and specifically, the demonstration that the fix can be achieved without abandoning the reward-reparameterization approach that makes DPO attractive.

Prior work that identified problems with DPO generally responded by proposing fundamentally different loss functions. IPO (Azar et al., 2024a) replaces the sigmoid with a squared loss. KTO (Ethayarajh et al., 2024) switches to binary feedback. SimPO (Meng et al., 2024) removes the reference policy entirely. These are reasonable responses to observed failures, but they abandon DPO's conceptual framework — the elegant reparameterization that connects preference learning to the Bradley-Terry model and RLHF. The field was drifting toward the conclusion that the DPO-RLHF connection might need to be discarded.

This paper takes the opposite approach: it shows that the connection is salvageable with a principled modification. By augmenting RLHF with a constraint term $\gamma\mathbb{E}[\delta_\pi]$ and re-deriving the DPO-like loss from this augmented objective, CPO retains the core structure of DPO — a loss depending only on $\pi_\theta$ and $\pi_\text{ref}$ through log-ratio differences — while inheriting the absolute advantage guarantee that standard DPO silently loses. The key insight is that the constraint term propagates through the derivation to produce an adaptive margin $\tilde{\gamma}_\text{ref}$ in the loss, and this margin can be precomputed from $\pi_\text{ref}$ alone, maintaining the stationarity required for gradient descent.

What makes this intellectually distinctive is that it treats the assumption violation not as a reason to abandon DPO but as an under-specification — a missing term in the objective that should have been there all along. Standard RLHF's objective $\mathbb{E}[r] - \beta\text{KL}$ can produce $\delta_{\pi^*} < 0$ when KL dominates reward. This is correct behavior for that objective, but it breaks the Bradley-Terry encoding that DPO relies on. CPO's insight is that the objective itself should be modified — not the loss derivation — to ensure $\delta_{\pi^*} > 0$, after which the standard reparameterization machinery works correctly.

E-CPOC deepens this insight by showing that even the approximation $1/\pi^* \approx 1/\pi_\text{ref}$ (which CPO requires for stationarity) can be eliminated. By formulating the constraint as a hard inequality and solving through KKT conditions, the effective margin $\beta\Phi_\text{cons}$ achieves a closed form that absorbs the $1/\pi^*$ factors into the Lagrange multipliers — they appear in the KKT equations but cancel out in the final expression. This is a genuinely elegant theoretical result: the constraint that DPO's derivation was implicitly assuming can be explicitly enforced, and the resulting loss requires no approximations, no reward model, and no additional computational cost beyond a precomputed margin term.

The practical significance is captured in the experimental results (Table 2): CPO achieves 25.15% win rate on AlpacaEval 2 (vs. DPO's 24.60%) and 32.6% on Arena-Hard (vs. DPO's 28.9%). The gains are modest on AlpacaEval 2 (+0.55%) but substantial on Arena-Hard (+3.7%), consistent with the theoretical prediction: the assumption violation matters most on difficult, discriminative prompts where the reference policy's biases are most pronounced relative to the reward signal. The fact that these gains come with no architectural changes, no additional models, and trivial computational overhead makes the contribution practically compelling — it's a drop-in replacement for DPO with stronger theoretical foundations.

Innovation 4: The Margin Ranking Geometry as a Unifying Diagnostic Framework

The paper's reframing of DPO, CPO, and E-CPOC as soft margin ranking losses (Section 5) is conceptually powerful in a way that goes beyond providing intuition. It connects preference optimization to the learning-to-rank literature (Burges et al., 2005; Cao et al., 2007; Schroff et al., 2015) and provides a unified diagnostic criterion: the sign of the effective target margin determines whether the optimization can converge to a preference-violating policy.

The key finding (Proposition 5.1) is that DPO's effective target margin is $m = \delta_{\pi_\text{ref}}$ — a quantity that can be negative and that is purely a property of the reference policy, not of the preference data or the optimization process. A margin ranking loss with negative target margin $m < 0$ achieves zero loss whenever $\delta_{\pi_\theta} > m$, which can be satisfied while $\delta_{\pi_\theta} < 0$ — the optimization has succeeded by its own criterion but failed to achieve alignment. This is the geometric essence of the $\mathcal{U}$ pathology: the loss landscape has a flat valley extending into the region $\delta_\pi < 0$, with zero loss achievable anywhere in $\delta_{\pi_\text{ref}} < \delta_{\pi_\theta} < 0$.

The reframing yields an immediate practical diagnostic that requires no training: compute $\delta_{\pi_\text{ref}}$ for all preference pairs in the dataset. Any pair with $\delta_{\pi_\text{ref}} < 0$ has a negative DPO margin — DPO can achieve low loss on that pair without the policy preferring the correct response. The fraction of such pairs is a direct measure of DPO's exposure to the pathology. This is simpler and more interpretable than monitoring implicit reward divergence or policy degeneracy, and it can be computed before training begins.

The constructive contribution of the geometric framing is showing that CPO and E-CPOC fix the problem by ensuring non-negative effective margins. CPO adds a constant shift $2\gamma/\beta$ to the target margin; E-CPOC adds an adaptive shift $\Phi_\text{cons}(\delta_{\pi_\text{ref}})$ that guarantees the margin is at least $\gamma > 0$ for all pairs. The geometric picture makes the mechanism transparent: both methods shift the loss landscape upward so that the zero-loss region lies entirely in $\delta_\pi > 0$. The difference between CPO and E-CPOC is whether the shift is constant (simpler but over-corrects on easy pairs) or adaptive (more efficient but more complex).

This perspective also explains why SimPO (which removes the reference policy) sidesteps the assumption violation (as noted in Appendix B): without $\pi_\text{ref}$ in the loss, there is no $\delta_{\pi_\text{ref}}$ to create a negative target margin. But as the paper argues, SimPO achieves this by abandoning the RLHF-BT framework entirely, sacrificing formal equivalence to any reward-maximizing objective. The margin ranking view clarifies the trade-off: SimPO is safe from this specific pathology but operates outside the theoretical framework that connects preference learning to utility maximization. CPO and E-CPOC stay within that framework while eliminating the pathology.

Innovation 5: The Provable Equivalence Theorem with Verifiable Conditions

The E-CPOC equivalence result (Theorem L.17, Appendix L) represents a significant theoretical advance in how alignment methods can be formally verified. The theorem establishes that E-CPOC is equivalent to explicitly constrained RLHF under four assumptions — Bradley-Terry preferences, approximate realizability, finite-sample data, and $\ell_2$-$\delta$-proximity — none of which requires global optimality, exact realizability, pointwise optimization guarantees, or a reward model. What makes this result structurally novel is that the $\ell_2$-$\delta$-proximity condition (the optimization quality requirement) admits a verifiable sufficient condition (Proposition L.3, Corollary L.18): the training loss gap $\epsilon_\text{loss}$ (directly observable) implies the required $\delta$-space proximity under a mild non-degeneracy condition on preference probabilities ($\kappa_0 > 0$). This creates a bridge from a quantity the practitioner monitors during training to the formal guarantee — something absent from prior preference optimization theory.

The significance of this bridge extends beyond E-CPOC. Prior theoretical analyses of preference optimization (Azar et al., 2024a; Munos et al., 2024) typically prove properties of the population optimum or assume exact optimization. These guarantees are asymptotic or idealized — they describe what would happen with infinite data and perfect optimization. The E-CPOC theorem, through the Loss-to-Delta bridge, describes what happens with the returned policy from finite training: if the loss converged (verifiable) and preferences are not deterministic (mild), then the equivalence bound holds with explicit constants. The $\ell_2$ formulation (rather than $\ell_\infty$) is deliberate — it permits larger errors on a few difficult pairs while controlling the average, reflecting realistic training outcomes where not every example is optimized perfectly.

The conservative bound strategy — using $\Phi_\text{cons}(\delta_{\pi_\text{ref}}) = \Phi(\delta_{\pi_\text{ref}}, 0)$ as an upper bound for all $\Delta r^* > 0$ — is another distinctive conceptual move. Rather than estimating the reward difference (which would require a reward model), E-CPOC assumes the worst case ($\Delta r \to 0^+$) and applies the maximum possible margin correction. This guarantees $\delta_{\pi^*_\text{E-CPOC}} \geq \delta_{\pi^*_\text{EC-RLHF}}(\Delta r^*)$ for all true $\Delta r^*$ — the method is provably conservative: it never under-corrects. The decomposition $\delta_{\pi^*_\text{E-CPOC}} = \delta_{\pi^*_\text{EC-RLHF}}(\Delta r = 0) + \Delta r^*/\beta$ (Theorem L.17) reveals the structure of this conservatism: E-CPOC uses the margin of the worst-case EC-RLHF solution plus the true reward advantage, making it strictly stronger than necessary but provably safe. This is an elegant theoretical resolution to the tension between eliminating the reward model (DPO's advantage) and maintaining formal guarantees (RLHF's advantage): conservatism replaces estimation, at the cost of potentially over-correcting on easy pairs.

The practical implication is that E-CPOC provides something no prior preference optimization method has offered: a formal alignment guarantee whose premises can be partially verified from training diagnostics alone, without requiring a reward model or assuming perfect optimization. Whether this matters in practice depends on the stringency of the non-degeneracy condition ($\kappa_0 > 0$) and the achievable $\epsilon_\text{loss}$ in specific deployments, but the architectural contribution — a verifiable bridge from observable loss to formal guarantee — is a template that future alignment methods could adopt.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the princeton-nlp/llama3-ultrafeedback-armorm dataset for preference alignment training. This dataset provides pairwise preference data (chosen vs. rejected responses) suitable for training DPO and its variants. Evaluation is conducted on two benchmarks: AlpacaEval 2 (Li et al., 2023) and Arena-Hard (Li et al., 2024), both of which evaluate conversational abilities on real-life queries. The paper also includes IFEval (Zhou et al., 2023) results in Appendix A.4 as an additional instruction-following benchmark.

• Base model(s). The primary model is Llama-3-8B-Instruct (Dubey et al., 2024), an instruction-tuned 8-billion-parameter model. This model was chosen because it is "representative of the capabilities of many contemporary LLMs" and sits in the regime where preference optimization is commonly applied — already instruction-tuned but not yet aligned with fine-grained human preferences. For the misaligned reference experiments (Appendix A.2), the authors construct corrupted references by fine-tuning Llama-3-8B-Instruct on rejected responses from varying fractions (R ∈ {0.2, 0.3, 0.4}) of the dataset.

• Metrics. Three primary metrics are reported:

  • Win Rate (WR%): the percentage of time the evaluated model's response is preferred over a reference response, as judged by a GPT-4-based evaluator. On AlpacaEval 2, the original evaluator gpt-4-1106-preview was used for main results (Table 2); the misalignment experiments (Appendix A.2) and sensitivity analysis (Appendix A.3) used gpt-4.1 due to deprecation of the original evaluator.
  • Length-Controlled Win Rate (LC%): a variant of win rate that adjusts for response length to penalize models that win simply by being more verbose.
  • Average Length: the mean response length in tokens, reported alongside WR and LC to contextualize whether gains come from quality improvements or verbosity. On Arena-Hard, 90% confidence intervals are reported alongside the win rate. On IFEval, both Strict Accuracy and Loose Accuracy are reported (Table 6).

• Baselines. The paper compares against a comprehensive set of preference optimization methods:

  • SFT-Base (Dubey et al., 2024): the base instruction-tuned model without any alignment process.
  • DPO (Rafailov et al., 2023): the standard Direct Preference Optimization.
  • SLiC-HF (Zhao et al., 2023): Sequence Likelihood Calibration with Human Feedback.
  • Contrastive-PO (Xu et al., 2024): Contrastive Preference Optimization.
  • RRHF (Yuan et al., 2023): Rank Responses to align language models with Human Feedback.
  • RDPO (Park et al., 2024): DPO with length disentanglement.
  • ORPO (Hong et al., 2024): Monolithic preference optimization without reference model.
  • KTO (Ethayarajh et al., 2024): Prospect-theoretic optimization extending DPO to binary feedback.
  • IPO (Azar et al., 2024b): Identity Preference Optimization with squared loss.
  • SimPO (Meng et al., 2024): Simple Preference Optimization with reference-free reward.

• Generation budget / compute accounting. The paper does not explicitly compare methods at fixed generation budgets since the evaluation metrics (AlpacaEval 2, Arena-Hard) measure downstream task performance rather than scaling behavior under varying inference compute. Instead, fairness is ensured through consistent training protocols: all methods use the same base model, same dataset, and comparable training resources. The key efficiency claim — that CPO's precomputation and per-iteration costs are essentially identical to DPO (Appendix I) — is argued analytically (one extra scalar operation per sample during training, reused forward pass for margin precomputation) rather than measured through wall-clock timing experiments. No FLOPs-matched comparisons between CPO and baselines are reported, which differs from the FLOPs-matched analysis style in the reference example paper.

• Cross-validation / statistical protocol. No formal cross-validation is used for strategy selection, as CPO does not involve choosing among multiple strategies per prompt (unlike the compute-optimal scaling in the reference example). Hyperparameter γ is selected via a sensitivity analysis on AlpacaEval 2 (Appendix A.3, Table 5), sweeping γ ∈ {0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40} and selecting γ = 0.25 based on peak LC performance (33.97%). Arena-Hard reports 90% confidence intervals (Table 2), providing some statistical rigor, though the method for computing these intervals is not described. The IFEval results (Table 6) report point estimates without confidence intervals.

---

Main Quantitative Results

Aggregate Performance on Standard Benchmarks

Table 2 (Main Results) reports the core comparison. On AlpacaEval 2:

• CPO achieves 25.15% WR (win rate), outperforming DPO at 24.60% by +0.55 percentage points. The gap is modest but consistent.
• CPO achieves 26.57% LC (length-controlled win rate), the highest among all methods. This surpasses SimPO's 25.91% by +0.66 percentage points and DPO's 25.09% by +1.48 percentage points. The length-controlled metric is particularly important because it penalizes verbosity — a common failure mode where models achieve high win rates simply by generating longer responses. CPO's strong LC performance (while maintaining an average length of 1879 tokens, comparable to DPO's 1896 and RDPO's 1895) suggests that the gains come from quality improvements rather than increased verbosity.
• CPO's average length of 1879 tokens is similar to strong baselines like DPO (1896) and RDPO (1895), and notably shorter than methods with lower LC scores like IPO (1993, LC 20.41%) and Contrastive-PO (2169, LC 15.95%). This indicates CPO achieves its high LC without sacrificing conciseness.

On Arena-Hard:

• CPO achieves 32.6% WR with a 90% confidence interval of (−1.9 / +2.4). This represents a +2.6 percentage point gain over SimPO (30.0%, the previous runner-up) and a +3.7 percentage point gain over DPO (28.9%). The advantage on Arena-Hard is substantially larger than on AlpacaEval 2, which is consistent with the paper's theoretical framework: Arena-Hard consists of "difficult, discriminative prompts where length bias and subtle preference distinctions matter most" — precisely the regime where the assumption violation (δ_ref < 0 for many pairs) is expected to be most consequential. DPO's relative-advantage optimization becomes most problematic on these hard distinctions, while CPO's absolute advantage guarantee provides the strongest benefit.
• The confidence interval is relatively wide (spanning ~4.3 percentage points), reflecting the inherent variability in LLM-based evaluation. However, the lower bound of CPO's interval (30.7%) is still above SimPO's point estimate (30.0%), suggesting the improvement is likely real.

On IFEval (Appendix A.4, Table 6):

• CPO achieves 35.12% Strict Accuracy and 43.99% Loose Accuracy, outperforming all baselines. The improvement over DPO is 1.11 percentage points on Strict (DPO: 34.01%) and 3.32 percentage points on Loose (DPO: 40.67%). The larger gain on Loose Accuracy — which gives partial credit for partially following instructions — suggests CPO improves the model's overall instruction-following tendency rather than just getting exact matches. This extends the evidence beyond conversational preference benchmarks to a more structured capability evaluation.

Key pattern across benchmarks: CPO's advantage over DPO is smallest on AlpacaEval 2 (+0.55% WR, +1.48% LC), larger on IFEval (+1.11% Strict, +3.32% Loose), and largest on Arena-Hard (+3.7% WR). This gradient correlates with benchmark difficulty and discriminativeness: AlpacaEval 2 contains broader, easier comparisons; Arena-Hard is specifically designed to be challenging. This pattern supports the paper's theoretical claim that the assumption violation matters most when preference distinctions are difficult and the reference policy's biases are most consequential.

Misaligned Reference Policy Experiments

The paper constructs misaligned references by fine-tuning Llama-3-8B-Instruct on rejected responses from varying fractions R ∈ {0.2, 0.3, 0.4} of the dataset, then training DPO and CPO on clean preference data starting from these corrupted references (Appendix A.2). This directly tests the central claim: when the reference policy is misaligned, DPO should degrade while CPO remains robust.

Table 3 establishes the misalignment levels:

• At R = 0.2: 53.2% of pairs have δ_ref < 0; 52.9% violate Assumption 3.1.
• At R = 0.3: 56.9% have δ_ref < 0; 56.8% violate Assumption 3.1.
• At R = 0.4: 60.1% have δ_ref < 0; 60.0% violate Assumption 3.1. The near-identical numbers between "δ_ref < 0" and "Assumption 3.1 violated" confirm that the reward correction Δr*/β is small (as quantified in Appendix A.1: mean 0.20, std 0.16) — reference misalignment dominates.

Table 4 (AlpacaEval 2 under misaligned references) reports:

• DPO degrades under increasing corruption:
  • R = 0.2: LC = 17.23%
  • R = 0.3: LC = 15.48%
  • R = 0.4: LC = 15.98% There is no monotonic trend in the corruption ratio — the degradation is present at all levels — but all values are substantially below DPO's performance from a clean reference (Table 2: LC = 25.09%). The fact that even R = 0.2 (only 20% corruption in reference training data) causes a drop from 25.09% to 17.23% LC supports the claim that DPO is sensitive to reference policy quality.
• CPO remains robust:
  • R = 0.2: LC = 27.60%
  • R = 0.3: LC = 27.35%
  • R = 0.4: LC = 24.34% At R = 0.2 and R = 0.3, CPO's LC actually exceeds its clean-reference performance (Table 2: LC = 26.57%), which is notable but not explained — it may reflect that the misaligned reference training inadvertently creates a larger gap between δ_ref and the desired δ_π, providing a stronger training signal once CPO's margin correction is applied. At R = 0.4, performance drops but remains substantially above DPO at the same corruption level (24.34% vs. 15.98%) and only ~2 percentage points below CPO from a clean reference. The average response length for CPO (1699-1714 tokens) is consistently shorter than DPO (1894-1958 tokens), suggesting CPO's robustness is not achieved through verbosity.

Figure 2 (Fraction in U over training steps):

• DPO's fraction in U rises then plateaus at approximately 0.28-0.36 across all corruption ratios. After an initial increase (as the policy improves relative to the reference but hasn't yet crossed δ_π = 0), DPO policies become trapped — the fraction remains elevated throughout training, directly visualizing the gradient vanishing predicted by Proposition 3.4.
• CPO's fraction in U rises then rapidly drops to near zero. The initial increase follows the same pattern (policies enter U as they improve relative to reference), but CPO's margin-corrected gradients provide sufficient strength to push past δ_π = 0. By the end of training, almost all samples have escaped U. The escape is faster and more complete at lower corruption ratios, consistent with the theory: larger margins (needed for more misaligned references) require more optimization steps to overcome.

These experiments provide the most direct validation of the paper's central theoretical claim. They demonstrate that: (1) DPO's performance degrades when the reference policy is misaligned (as predicted by Theorem 3.5), (2) CPO remains robust under the same conditions (as predicted by Theorem 4.9 and Theorem 4.10), and (3) the mechanism of failure is the undesirable solution space U — DPO policies become trapped there while CPO policies escape (as predicted by Proposition 3.4 and the gradient analysis).

Computational Efficiency

The paper argues analytically that CPO and E-CPOC have essentially identical computational cost to DPO (Appendix I). Standard DPO already requires a forward pass over the entire dataset to precompute δ^(i)_ref = log π_ref(y_w|x) − log π_ref(y_l|x) before training. CPO reuses this same forward pass to additionally compute ˜γ^(i)_ref = γ(1/π_ref(y_w|x) + 1/π_ref(y_l|x)) — two scalar divisions and one addition per sample. During training, the only difference from DPO is subtracting the precomputed margin from the logits (Algorithm 1, line 11: logits ← β(δ_θ − δ^(i)_ref) − ˜γ^(i)_ref), one extra scalar operation per sample.

No explicit timing comparisons or FLOPs measurements are reported in the paper. The efficiency claim is therefore theoretical rather than empirical — it rests on the argument that the additional operations are of negligible cost relative to the forward/backward passes through the language model. This is almost certainly true at the scales studied (8B parameters), but the absence of any measured wall-clock times or memory overhead comparisons is a gap in the experimental validation, particularly for practitioners evaluating deployment costs.

---

Ablation Studies and Robustness Checks

Sensitivity to γ (Appendix A.3, Table 5): The hyperparameter γ controls the strength of the constraint term in CPO. A sweep over γ ∈ {0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40} on AlpacaEval 2 reveals:

• Performance peaks at γ = 0.25 (WR = 28.36%, LC = 33.97%) and remains strong in the range [0.20, 0.40], with LC consistently above 30%.
• Below 0.20, performance drops notably: at γ = 0.15, LC = 26.32%; at γ = 0.10, LC = 26.46%. This supports the theoretical prediction that insufficient margin correction (γ too small) fails to escape the undesirable solution space U, as the constraint term is too weak to overcome the KL penalty's pull toward a misaligned reference.
• The performance at γ = 0.10 is actually slightly better than at γ = 0.15 (LC 26.46% vs. 26.32%), which may reflect noise or the fact that at very low γ, CPO approximates DPO — which itself achieves reasonable performance on this clean reference (Table 2: DPO LC = 25.09%).
• The peak is not sharp — performance is similar across γ ∈ [0.20, 0.30] — suggesting CPO is not highly sensitive to exact γ choice within a reasonable range, which is practically important for hyperparameter tuning.
• Note: this sensitivity analysis uses gpt-4.1 as evaluator (due to deprecation of gpt-4-1106-preview), so the absolute numbers are not directly comparable to Table 2 (which used the original evaluator). The relative trends within Table 5 are the relevant signal.

SFT-Base and clean-reference baselines (Table 2): The SFT-Base (no alignment) achieves 14.22% WR and 13.47% LC — all preference optimization methods provide substantial improvements (10-12 percentage points LC), confirming that the alignment task is non-trivial and that the reference model is not already well-aligned. This context is important because DPO's implicit assumption (Assumption 3.1) is most problematic precisely when the reference needs alignment — the very scenario where these methods are deployed.

Comparison with clipped-reference baseline (Appendix A.5, Table 7): A natural question is whether CPO's adaptive margin provides benefits beyond simply preventing negative margins. The authors compare CPO against a "clipped-reference" baseline that clips δ_ref to be non-negative (i.e., max(0, δ_ref)) before applying standard DPO. Results:

• Clipped-reference: LC = 23.86%, WR = 17.91%, average length = 1586.
• CPO: LC = 33.97%, WR = 28.36%, average length = 1702. CPO substantially outperforms the clipped baseline, with a 10.11 percentage point LC advantage. This demonstrates that the adaptive margin ˜γ_ref provides benefits beyond simply preventing negative δ_ref — the sample-adaptive weighting (larger margins for low-probability pairs) and the connection to constrained RLHF optimality conditions matter in practice.

Violation frequency measurement (Appendix A.1, Figure 1): On the clean Llama-3-8B-Instruct reference with β = 0.1, 45.5% of preference pairs violate Assumption 3.1. Key statistics:

• δ_ref distribution: mean = 1.35, standard deviation = 46.69, median = 2.73. The large standard deviation (relative to the mean) indicates that while the average pair slightly favors y_w, there is enormous variation — many pairs have strongly negative δ_ref.
• Δr*/β distribution: mean = 0.20, standard deviation = 0.16. The reward correction is two orders of magnitude smaller than the reference policy's variability, explaining why the violation rate is high: the reward signal cannot compensate for reference misalignment at this β.
• 54.2% of pairs have δ_ref > 0, but this only slightly exceeds the 50% random baseline, confirming that the instruction-tuned reference is not strongly aligned with the fine-grained preferences in the dataset.

Why is there no ablation over β? The paper uses β = 0.1 throughout (matching the standard DPO configuration) but never ablates this choice. This is notable because Proposition 3.2 shows that larger β makes Assumption 3.1 harder to satisfy (the RHS −Δr*/β becomes less negative, requiring δ_ref to be less negative to compensate). At larger β, the violation rate would be even higher, and CPO's advantage over DPO should be larger. At smaller β, the reward signal dominates, potentially reducing the violation rate and narrowing the gap. A β sweep — comparing DPO vs. CPO at different KL regularization strengths — would have directly tested this predicted interaction and strengthened the paper's theoretical claims. Its absence is a limitation.

No direct comparison between CPO and E-CPOC: The experiments exclusively evaluate CPO. E-CPOC — the theoretically stronger variant with hard constraints, provable equivalence, and adaptive margins — is never evaluated empirically. The paper acknowledges this limitation explicitly (Section 7: "the performance of E-CPOC should also be validated in experiments beyond the theoretical aspects"). This is a significant gap: the paper's strongest theoretical results concern E-CPOC, but there is no evidence that its additional complexity (the softplus margin function Φ_cons, the smoothness parameter τ) translates to practical gains over CPO's simpler constant-γ approach.

No ablation over τ for E-CPOC: Since E-CPOC is not evaluated, the smoothness parameter τ (which controls how closely the softplus approximates the hard max in the constraint) is never studied. The theoretical limit τ → ∞ recovers the hard constraint, but finite τ affects the gradient behavior — this trade-off is unexplored.

No training dynamics beyond fraction in U: The misalignment experiments (Figure 2) track fraction in U over training, which directly validates the theoretical mechanism. However, the paper does not provide other training diagnostics that would be informative: loss curves for DPO vs. CPO (to test the prediction that DPO loss decreases while the policy remains in U), δ_π distributions before and after training (to visualize the escape from negative values), or gradient norm comparisons (to verify the vanishing gradient prediction in Proposition 3.4). These would have provided additional mechanistic validation.

---

Critical Assessment

The experimental results consistently show that CPO outperforms DPO and other baselines on standard benchmarks (AlpacaEval 2, Arena-Hard, IFEval), and the misaligned reference experiments (Appendix A.2) demonstrate that CPO is robust to reference policy quality in ways that DPO is not. However, the experiments demonstrate something narrower than the paper's full theoretical claims, and several important gaps exist between what was proved theoretically and what was validated empirically.

What the experiments DO demonstrate:

1. CPO achieves modest but consistent improvements over DPO on standard benchmarks from a clean reference (+0.55% WR on AlpacaEval 2, +3.7% on Arena-Hard, +1.1% Strict on IFEval). This validates that the method works in practice and is not purely a theoretical exercise.
2. When the reference policy is deliberately corrupted, DPO degrades substantially while CPO maintains performance (Table 4). This directly supports the paper's core claim that DPO's effectiveness depends on reference policy quality, while CPO's constraint term provides robustness.
3. The fraction of samples in the undesirable solution space U follows the predicted dynamics: DPO policies become trapped (fraction plateaus), while CPO policies escape (fraction drops to near zero; Figure 2). This provides direct mechanistic evidence for the theoretical framework.
4. The gains are largest on the most difficult benchmark (Arena-Hard: +3.7% over DPO; AlpacaEval 2: +0.55% over DPO), consistent with the prediction that the assumption violation matters most on hard, discriminative prompts.

What the experiments do NOT demonstrate:

The claim that DPO and RLHF are conditionally equivalent (Theorem 3.5) is never experimentally verified via direct RLHF comparison. The theoretical core of the paper is that DPO and RLHF optimize different objectives when Assumption 3.1 is violated — but the experiments never train an actual RLHF model (with a separate reward model and PPO) for comparison. All baselines are DPO variants or DPO-adjacent methods. While the misaligned reference experiments demonstrate that DPO degrades relative to CPO, they do not directly demonstrate that DPO diverges from RLHF under assumption violation — because RLHF's behavior in this regime is never measured. The theoretical claim that "DPO optimizes a fundamentally different objective than RLHF" (Theorem 3.5) is supported by mathematical proof but not by experimental evidence showing RLHF and DPO producing different policies from the same misaligned reference.

The E-CPOC variant — the paper's strongest theoretical result — is never evaluated. The provable equivalence theorem (Theorem L.17), the conservative bound strategy, and the verifiable Loss-to-Delta bridge are all developed for E-CPOC. None of these theoretical advances are validated empirically. The experiments exclusively use CPO (soft constraint, constant γ), which lacks the hard constraints, adaptive margin function Φ_cons, and formal equivalence guarantees of E-CPOC. This means there is no evidence that the additional theoretical properties of E-CPOC (conservatism, sample-adaptivity through Φ_cons, provable upper bound on EC-RLHF) translate to practical improvements over the simpler CPO.

The single-model, single-dataset scope limits generalizability. All experiments use Llama-3-8B-Instruct. While this is a reasonable choice, the paper's theoretical claims are model-agnostic. Without experiments on other model families (e.g., Mistral, Qwen, Gemma) or scales (e.g., 70B models), it is impossible to assess whether the 45.5% violation rate and the CPO-vs-DPO gap are consistent across model architectures and sizes. Model-specific factors — calibration, output distribution entropy, in-context learning capabilities — could affect both the violation rate and the effectiveness of the margin correction.

The experiments on IFEval add instruction-following evidence but the claim that CPO "achieves state-of-the-art performance" (abstract) requires careful interpretation. CPO achieves the highest numbers among the reported methods on the specific benchmarks tested. However, the paper does not compare against the strongest possible configurations of competing methods — for instance, larger-scale DPO variants, ensembles, or methods that combine DPO with other techniques (e.g., iterative DPO, DPO + rejection sampling). "State-of-the-art" within the reported comparison set is accurate; whether it holds against the broader literature is untested.

The sensitivity analysis for γ (Appendix A.3) shows robust performance for γ ∈ [0.2, 0.4], but the lower bound of this range is already well above γ = 0. At γ = 0.10, performance drops to near-DPO levels. This suggests γ is not just a "strength of correction" parameter but effectively a binary switch: below some threshold, the constraint is too weak to overcome the KL penalty, and CPO behaves similarly to DPO. The paper's Theorem 4.9 provides a formula for γ* (the minimum γ that guarantees absolute advantage), but this formula depends on the unknown true reward r*. In practice, γ must be tuned. The sensitivity analysis shows this is feasible (the peak is not sharp), but the dependence on γ for escaping U means that practitioners using CPO without tuning could inadvertently select a γ that provides minimal benefit.

The misaligned reference experiments are compelling but artificial. Corrupting 20-40% of the training data and SFT-ing on rejected responses is a clean experimental manipulation that produces systematically misaligned references. Real-world reference policies may be misaligned in more complex ways — for instance, biases that are correlated with specific topics, response styles, or demographic factors, rather than uniformly favoring rejected responses across a fraction of the data. Whether CPO's robustness transfers to these more realistic misalignment patterns is not tested.

Missing experiments that would have strengthened the paper:

1. Direct DPO vs. RLHF comparison under assumption violation: Train an explicit reward model, run PPO-based RLHF, and compare the resulting policies to DPO-trained policies from the same misaligned reference. This would directly validate Theorem 3.5's claim that the methods optimize different objectives.

2. E-CPOC evaluation: The paper's strongest theoretical results concern E-CPOC, but it is never run. Even a single experiment (e.g., AlpacaEval 2 comparison between CPO and E-CPOC) would indicate whether the additional theoretical properties translate to practical gains.

3. β sweep: Since the assumption violation depends on β (larger β = stronger KL = harder to satisfy Assumption 3.1), comparing DPO and CPO across multiple β values would test whether CPO's advantage grows as predicted.

4. Multiple model families and scales: Replicating the key results (violation rate, DPO vs. CPO performance, misaligned reference robustness) on models from different families (Mistral, Qwen) and at different scales (1B, 70B) would establish generalizability.

5. Training dynamics beyond fraction in U: Loss curves, gradient norm comparisons, and δ_π distribution shifts during training would provide additional mechanistic evidence for the theoretical framework.

6. Compute-matched comparisons with baselines: Wall-clock timing, memory usage, and FLOPs comparisons between CPO and DPO would validate the analytical claim of negligible overhead.

In summary: The experiments convincingly demonstrate that CPO works, that it improves over DPO on standard benchmarks, and that it is robust to deliberately introduced reference policy misalignment. The mechanistic evidence (fraction in U trajectories) provides direct support for the theoretical explanation of why CPO works. However, the gap between the paper's theoretical scope (conditional equivalence, provable guarantees for E-CPOC, detailed characterization of the DPO-RLHF relationship) and its experimental scope (CPO on a single model and dataset, without RLHF comparisons, without E-CPOC evaluation) is substantial. The experiments validate CPO as a practical improvement over DPO; they do not fully validate the broader theoretical framework that motivates it.

6. Limitations and Trade-offs

The γ Boundary Between Success and Failure Is Unknown Without Ground-Truth Rewards

The paper's central practical claim — that CPO guarantees alignment when γ is sufficiently large — hinges on Theorem 4.9, which provides an explicit formula for the minimum required γ^*. That formula depends on the true reward difference r^*(y_w) - r^*(y_l), which is the latent quantity that preference data is supposed to reveal but that DPO-like methods are designed to avoid modeling explicitly. The paper states this formula (Equation 27) without then providing a way to estimate γ^* from available data alone.

The consequence is that a practitioner must treat γ as a hyperparameter to be tuned on validation performance, rather than computed from first principles. The sensitivity analysis in Appendix A.3 (Table 5) confirms that this matters: at γ = 0.15, length-controlled win rate drops to 26.32%; at γ = 0.25, it rises to 33.97% — a ~7.6 percentage point gap. Below some threshold (roughly γ ≈ 0.2), CPO appears to provide minimal benefit over DPO, consistent with the theoretical prediction that insufficient margin correction fails to escape the undesirable solution space U. The paper does not characterize how this threshold relates to properties of the reference policy or preference data that a practitioner could measure before training, making γ tuning a trial-and-error process whose outcome determines whether CPO meaningfully outperforms DPO at all.

The paper partially mitigates this by reporting that the peak is not sharp — performance is similar for γ ∈ [0.2, 0.4] (Table 5) — suggesting that within a reasonable range, CPO is not highly sensitive. But the existence of a sharp drop below the threshold means the tuning process must discover a γ on the correct side of this boundary, and there is no diagnostic (short of training multiple models and evaluating) to determine which side a candidate γ falls on. The paper does not address this gap directly, and it is a consequential open problem for anyone deploying CPO on a new dataset or model.

---

E-CPOC — The Theoretically Strongest Variant — Is Entirely Untested

The paper's most sophisticated theoretical contributions concern E-CPOC: the hard-constraint formulation via KKT conditions (Appendix J), the conservative bound strategy that eliminates reward model dependence (Section 4.5), the provable equivalence theorem to explicitly constrained RLHF under verifiable assumptions (Theorem L.17, Appendix L), and the Loss-to-Delta bridge that connects training loss to formal guarantees (Proposition L.3, Corollary L.18). These results occupy roughly half of the paper's technical appendices and constitute its strongest claim to theoretical novelty.

Yet E-CPOC is never evaluated in any experiment. All empirical results — Table 2 (main benchmarks), Table 4 (misaligned references), Table 5 (γ sensitivity), Table 6 (IFEval), Figure 2 (fraction in U) — use CPO, the soft-constraint variant with constant γ. The paper acknowledges this explicitly in Section 7:

"the performance of E-CPOC should also be validated in experiments beyond the theoretical aspects"

The consequence is that there is no evidence that E-CPOC's additional theoretical properties translate to practical benefits. E-CPOC introduces several design choices absent from CPO: the smoothness parameter τ (controlling how closely the softplus Φ_cons approximates the hard max), the adaptive margin βΦ_cons(δ_ref) (which depends on δ_ref alone rather than on 1/π_ref factors), and the conservative bound strategy (assuming Δr → 0^+ rather than using the unknown true reward). Any of these could interact with optimization dynamics, gradient behavior, or final policy quality in ways that the theory does not predict. For instance, the conservative bound makes E-CPOC more conservative than necessary — using the worst-case Δr = 0 margin for every pair — which might over-regularize on pairs where the true reward difference is large, potentially reducing the policy's ability to learn strong preference signals in those cases. The adaptive margin Φ_cons also changes the gradient weighting compared to CPO's γ̃_ref (Propositions E.5–E.6), and whether this improves or degrades optimization is unknown without experiments.

A practitioner deciding between CPO and E-CPOC has no empirical basis for that decision: the simpler method (CPO) is validated; the theoretically stronger method (E-CPOC) is not. This is a significant gap between the paper's theoretical scope and its experimental coverage.

---

The 45.5% Violation Rate Is Measured on One Model, One Dataset, One β

The paper's headline empirical finding — that 45.5% of preference pairs violate Assumption 3.1 on Llama-3-8B-Instruct with β = 0.1 (Figure 1, Appendix A.1) — provides crucial motivation for the entire work. It demonstrates that the implicit assumption is not a theoretical curiosity but a pervasive condition in a realistic alignment setting. However, this measurement is a single data point.

The violation rate depends on three interacting factors: the reference policy's preference distribution (δ_ref), the true reward differences in the preference data (Δr^*), and the KL regularization strength (β). The paper demonstrates (Proposition 3.2, Equation 10) that larger β makes the condition stricter — the right-hand side −Δr^*/β approaches zero, so even mildly negative δ_ref values cause violations. Smaller β has the opposite effect. The paper never ablates β experimentally, so we do not know whether the 45.5% rate is specific to β = 0.1 (a common but not universal choice in DPO training) or representative across typical β ranges. At β = 0.5, the violation rate would almost certainly be higher; at β = 0.01, it might be substantially lower. Since practitioners often tune β as a hyperparameter, the relevance of the 45.5% figure to a specific deployment depends on the chosen β.

The violation rate also depends on the reference model and dataset. The paper uses an instruction-tuned reference (Llama-3-8B-Instruct) — models at different stages of alignment (base pretrained, SFT-only, RLHF-tuned) would produce different δ_ref distributions. The dataset (UltraFeedback) captures specific types of preferences; datasets with different annotation protocols, difficulty distributions, or domain coverage might yield different violation patterns. The paper does not explore this dependence, making the 45.5% figure — while striking — an existence proof rather than a characterization of the violation rate's expected range across common deployment scenarios.

The misaligned reference experiments (Appendix A.2) partially address the model/dataset dependence by constructing several deliberately corrupted references and showing that DPO degrades while CPO remains robust. This demonstrates the qualitative dependence on reference quality that the theory predicts. But it does not quantify the quantitative range of violation rates in natural (non-corrupted) settings, which is what a practitioner would need to assess whether their specific reference-model-dataset-β combination is likely to fall in the regime where CPO provides meaningful benefit.

---

Hard Problems Cannot Be Addressed — The Method Amplifies, Does Not Create

CPO and E-CPOC, like DPO, derive from the KL-regularized RLHF framework (Equations 2, 17, 29). The constraint term ensures that the optimal policy prefers y_w over y_l for pairs in the training data, but it does not change the fundamental ceiling imposed by the KL penalty: the learned policy cannot deviate arbitrarily far from the reference policy. The constraint term γE[δ_π] or the hard constraint δ_π ≥ γ provides a directional push (toward preferring winners), but it operates within the same KL-regularized optimization landscape.

For preference pairs where the reference policy is extremely misaligned — δ_ref is very negative, and the preferred response y_w has near-zero probability under π_ref — even CPO's corrected gradients may struggle. The adaptive margin γ̃_ref = γ(1/π_ref(y_w|x) + 1/π_ref(y_l|x)) grows large when π_ref(y_w|x) is tiny (Appendix H.3 discusses this as a "self-correcting" property), which strengthens the gradient signal. However, the policy must still actually increase π_θ(y_w|x) from near-zero, which requires navigating the parameter space away from a region where the reference policy assigns negligible mass to a token sequence. The paper's theory guarantees that an optimal solution with δ_π > 0 exists (Theorems 4.9, L.17), but it does not guarantee that gradient descent from the reference initialization will find it within practical compute budgets — the optimization landscape may have barriers that the theory does not capture.

This limitation is analogous to a finding in the reference example paper (Section 7, Figure 9): on the hardest problems (difficulty bin 5), no amount of test-time compute helps, because the base model cannot produce correct solutions at any non-trivial rate. Similarly, CPO's guarantees apply to the optimal policy of the constrained objective; whether optimization reaches a policy close to that optimum from a severely misaligned initialization is a separate question that the paper does not theoretically analyze. The misaligned reference experiments (Table 4) provide some evidence: at R = 0.4 (60% violation rate), CPO's LC drops to 24.34% (from 27.60% at R = 0.2), suggesting that extreme misalignment does degrade CPO's performance, though it remains far above DPO (15.98%). But these experiments use corrupted references trained on only 20-40% rejected responses — more extreme misalignment (e.g., a randomly initialized model used as reference, or a domain shift where π_ref assigns essentially zero probability to all preferred responses) is not tested.

The paper does not explicitly discuss this limitation. The theoretical guarantees are about the existence and properties of the optimal policy, not about the optimization trajectory from initialization, and the gap between these is an implicit scope limitation of the analysis.

---

Single Benchmark Family for Standard Evaluation; Corrupted-Reference Experiments Are Artificial

The paper's evaluation of CPO against standard baselines uses AlpacaEval 2, Arena-Hard, and IFEval (Table 2, Table 6). While these are widely used and reasonably diverse — AlpacaEval 2 tests general conversational quality, Arena-Hard focuses on difficult discriminative prompts, IFEval measures instruction-following — they all evaluate text generation quality as judged by LLM-based evaluators. None of these benchmarks measures factual accuracy, reasoning capability, safety, calibration, or downstream task performance (e.g., coding, mathematics, summarization metrics like ROUGE). The paper demonstrates that CPO improves preference alignment on conversation-focused benchmarks, but provides no evidence about whether the constraint term has side effects on other capabilities that practitioners care about.

This matters because CPO's mechanism — explicitly pushing the policy to increase log π_θ(y_w|x) - log π_θ(y_l|x) — is a form of targeted optimization pressure on specific response pairs. It is plausible that this pressure could cause overfitting to the training preference pairs at the expense of the policy's general language modeling quality, or that it could interact with other desiderata (factual correctness, safety, diversity) in unintended ways. The paper's geometric interpretation (Section 5) shows that CPO shifts the zero-loss region of the ranking loss to guarantee δ_π > 0 — but it says nothing about what happens to the absolute probabilities π_θ(y|x) for responses NOT in the preference pairs, or about the policy's behavior on prompts outside the training distribution.

The corrupted-reference experiments (Appendix A.2, Tables 3–4, Figure 2) are the paper's strongest mechanistic evidence, but the corruption procedure is artificial: fine-tuning on rejected responses from a fraction of the data creates a reference that systematically disfavors winners. Real-world reference policies may exhibit misalignment patterns that are more subtle — for instance, biases correlated with specific topics, demographic associations, or response formats — rather than a uniform preference for rejected responses across a random subset of the data. Whether CPO's margin correction, which depends on δ_ref and π_ref probabilities for each pair, behaves similarly under these more realistic misalignment patterns is untested.

The paper partially addresses the benchmark diversity concern by including IFEval (Appendix A.4), which measures instruction-following rather than preference judgments, and the gains there (+1.1% Strict, +3.3% Loose over DPO) suggest the improvement is not purely a conversational preference artifact. But the scope remains narrow relative to the breadth of capabilities that alignment methods are expected to preserve or enhance. An evaluation on reasoning benchmarks (e.g., GSM8K, MMLU), safety benchmarks (e.g., TruthfulQA, ToxiGen), or factuality metrics would significantly strengthen confidence that CPO's constraint term does not introduce capability regressions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the conversation around preference optimization from "DPO is equivalent to RLHF" to "DPO is equivalent to RLHF if and only if the reference policy already respects human preferences." The shift is conceptual rather than methodological — the paper does not introduce a new training paradigm but rather identifies a hidden premise in an existing one and provides a constructive fix. The magnitude is best understood as a corrective reframing with practical consequences: the work changes what practitioners should monitor, what theorists should prove, and what the field means when it says a method "is equivalent" to another.

The key conceptual move is separating algebraic validity from semantic validity in the reward reparameterization. The identity r*(y_w) - r*(y_l) = β(δ_π* - δ_ref) is algebraically exact. The claim that substituting this into the Bradley-Terry model yields a loss equivalent to RLHF is a separate assertion — one that assumes π* encodes human preferences in its log-probability ratios. Before this paper, the field conflated these two steps: the mathematical elegance of the reparameterization was taken as evidence of equivalence. After this paper, researchers must ask: "does my reference policy already prefer the winners for most pairs in my dataset?" If the answer is no, DPO is optimizing something else — silently, without indication in the loss curve.

The practical consequence is that alignment practitioners now have a pre-training diagnostic: compute δ_ref for all preference pairs before running DPO. Any pair with δ_ref < 0 has a negative DPO margin target (by the geometric interpretation of Section 5) — DPO can achieve low loss on that pair without the policy preferring the correct response. The fraction of such pairs is a direct measure of exposure to the pathology. On the paper's setup (Llama-3-8B-Instruct, UltraFeedback, β = 0.1), that fraction is 45.5% — nearly half the training data. This diagnostic requires no training, no reward model, and one forward pass over the dataset. It should become standard practice before deploying DPO, much as practitioners check for class imbalance before training a classifier.

This work reconciles a growing set of contradictory findings. Prior papers documented that DPO's implicit rewards overfit (Fisch et al., 2024), that implicit reward models generalize poorly (Lin et al., 2024), that DPO prioritizes statistical distinguishability over value alignment (Shi et al., 2025), and that performance gaps between DPO and RLHF depend on representational capacity (Im & Li, 2024). Each of these papers identified a symptom. This paper identifies the unifying mechanism: when δ_ref < -Δr*/β, DPO's optimization objective diverges from RLHF's. Overfitting, poor generalization, and value misalignment are downstream consequences of optimizing the wrong objective. The conflict between "DPO works well" (many empirical papers) and "DPO has theoretical problems" (recent analysis papers) is resolved by the conditional nature of the equivalence: DPO works well when the reference is already well-aligned; it fails in precisely characterized ways when the reference is misaligned. The 45.5% violation rate on a standard instruction-tuned model demonstrates that both regimes coexist in typical deployments — explaining why some practitioners report success and others report degradation, depending on their specific reference, dataset, and β.

Research directions that become more attractive:

• Verifier/loss landscape analysis for preference optimization. The undesirable solution space U is a concrete object that can be studied: its geometry, its dependence on β and reference quality, and the optimization dynamics that lead policies into and out of it. The margin ranking perspective (Section 5) connects this to decades of learning-to-rank research — techniques for margin adaptation, hard negative mining, and loss landscape shaping from that literature may transfer directly.
• Principled reference policy design for alignment. If DPO's equivalence depends on reference quality, then selecting or constructing the reference policy becomes a first-class design decision rather than an afterthought. The paper shows that SFT on rejected responses produces a misaligned reference that devastates DPO performance (Table 4: LC drops from 25.09% to 15.48% at R = 0.3). This suggests that careful reference construction — perhaps through iterative refinement, or through reference models specifically trained to be "preference-neutral" — could substantially improve DPO's reliability.
• Direct DPO-RLHF comparison under controlled assumption violation. The paper proves DPO and RLHF optimize different objectives when the assumption is violated, but never experimentally compares them. A study that trains both DPO and PPO-based RLHF from identical misaligned references, measuring policy divergence, would directly test Theorem 3.5 and provide the missing empirical validation.

Research directions that become less attractive:

• More complex DPO loss functions that do not address the assumption. The paper shows that IPO, KTO, ORPO, and other DPO variants inherit the same structural dependence on π_ref that creates the assumption violation because they all use the reward reparameterization. Modifying the loss function (e.g., squared loss in IPO, binary feedback in KTO) changes what is optimized but not whether the optimization target aligns with human preferences when δ_ref is negative. Future work on preference optimization should either address the assumption violation directly (as CPO and E-CPOC do) or operate outside the RLHF-BT framework entirely (as SimPO does, at the cost of abandoning formal equivalence to any reward-maximizing objective).
• Reward model distillation as a general solution to DPO's problems. Fisch et al. (2024) proposed distilling explicit reward models to improve DPO. This paper suggests that distillation addresses a symptom (implicit reward overfitting) but not the root cause (objective mismatch when δ_ref is negative). A distilled reward model may provide better reward estimates, but if the DPO loss still optimizes δ_π - δ_ref rather than δ_π, the fundamental pathology remains. Resources are better directed toward methods that guarantee alignment through constraints (CPO, E-CPOC) or that build verifiable equivalence into the loss derivation.

---

Follow-Up Research This Work Enables

Characterizing violation rates across model families, scales, and β values. The paper's 45.5% violation rate is measured on a single configuration: Llama-3-8B-Instruct, UltraFeedback, β = 0.1. A systematic study measuring δ_ref distributions and violation rates across (a) model families (Llama, Mistral, Qwen, Gemma), (b) model scales (1B, 8B, 70B), (c) alignment stages (base, SFT-only, instruction-tuned, RLHF-tuned), (d) β values (0.01 to 1.0), and (e) preference datasets (UltraFeedback, HH-RLHF, OpenAssistant, custom domain-specific data) would establish the expected range of this pathology in practice. The study would test the paper's prediction (Proposition 3.2) that larger β increases violation rates, and would identify which deployment scenarios are most at risk. A negative result — finding that violation rates are consistently low (<10%) across diverse realistic settings — would limit CPO's practical relevance while leaving the theoretical contribution intact. A positive result — finding that violation rates are routinely 30-50% across model families — would make the pre-training δ_ref diagnostic an essential tool for the field.

Training an explicit RLHF baseline and measuring policy divergence from DPO under controlled assumption violation. The paper proves DPO and RLHF optimize different objectives when Assumption 3.1 is violated (Theorem 3.5), but this is never experimentally verified because no RLHF model (with explicit reward model and PPO) is trained for comparison. A direct experiment would: (1) train an explicit reward model on the UltraFeedback preference data, (2) run PPO-based RLHF from a misaligned reference (e.g., the R = 0.3 corrupted reference from Appendix A.2), (3) train DPO from the same reference on the same data, (4) compare the resulting policies on (a) preference satisfaction (fraction of pairs with δ_π > 0), (b) downstream benchmark performance, and (c) KL divergence from the reference. Theorem 3.5 predicts that DPO's policy will diverge from RLHF's policy specifically on pairs where δ_ref ≤ -Δr*/β, and that DPO will achieve lower loss but worse preference satisfaction on those pairs. This experiment would directly validate the paper's core theoretical claim and provide the first empirical characterization of how the DPO-RLHF divergence varies with assumption violation severity.

Empirical evaluation of E-CPOC against CPO and DPO under varying reference quality. The paper's strongest theoretical results concern E-CPOC (Theorem L.17, the Loss-to-Delta bridge, the conservative bound strategy), but E-CPOC is never evaluated. A head-to-head comparison of CPO vs. E-CPOC under the same misaligned reference conditions (Appendix A.2, Tables 3-4) would determine whether E-CPOC's theoretical advantages — hard constraints with provable equivalence, adaptive margin Φ_cons(δ_ref), elimination of the 1/π* approximation error — translate to practical improvements. The experiment should sweep the E-CPOC smoothness parameter τ (controlling the softplus approximation to the hard max) to identify whether sharper approximations (larger τ) improve alignment at the cost of optimization stability. A negative result — finding that E-CPOC performs similarly to or worse than CPO — would be valuable: it would suggest that the conservative bound (assuming Δr = 0) over-regularizes in practice, or that the adaptive margin Φ_cons interacts unfavorably with optimization dynamics. A positive result — E-CPOC outperforming CPO, particularly at high violation rates — would establish E-CPOC as the theoretically and empirically superior method. Given E-CPOC's stronger guarantees, this experiment is the single highest-priority follow-up to this paper.

Measuring capability side effects of the constraint term on reasoning, factuality, and safety benchmarks. CPO's constraint term γE[δ_π] explicitly pushes the policy to increase log π_θ(y_w|x) - log π_θ(y_l|x) on training pairs. This targeted pressure could have unintended consequences: overfitting to preference pairs at the expense of general language modeling quality, reduced diversity of generated responses, or interference with factual accuracy and safety behaviors that are not captured by preference data. A comprehensive evaluation on (a) reasoning benchmarks (GSM8K, MATH, MMLU), (b) factuality benchmarks (TruthfulQA, FreshQA), (c) safety benchmarks (ToxiGen, RealToxicityPrompts), and (d) diversity metrics (distinct-n, self-BLEU) would characterize CPO's capability profile beyond the conversational preference benchmarks already studied. The paper's IFEval results (Table 6) provide initial evidence that instruction-following improves, but this is a narrow capability slice. A negative result — finding that CPO degrades factuality or safety relative to DPO at the same level of preference alignment — would reveal a trade-off that practitioners must navigate and would motivate research into constraint mechanisms that are more selective (pushing δ_π upward only for preference-relevant dimensions of the response distribution without affecting factual or safety-related behaviors).

Extending the constrained RLHF framework to iterative/online preference optimization. The paper studies a single round of preference optimization from a fixed reference and fixed preference dataset. Modern alignment pipelines increasingly use iterative approaches: generate responses from the current policy, collect human/AI preference labels on those responses, and retrain (as in ReST, iterative DPO, or online RLHF). In this setting, the reference policy changes each round (it becomes the previous round's optimized policy), and the preference data is on-policy for the reference. How does the assumption violation evolve across rounds? If each round's DPO optimization enters U on some fraction of pairs, the reference for the next round may be more misaligned than the previous reference — creating a compounding pathology. Applying CPO or E-CPOC with the constraint term in each round might prevent this drift. An experiment tracking the violation rate, fraction in U, and downstream performance across 3-5 rounds of iterative CPO vs. iterative DPO would characterize whether the constrained approach provides compounding benefits or whether single-round correction is sufficient.

Connecting the margin ranking perspective to hard negative mining and curriculum learning for preference data. The geometric interpretation (Section 5) shows that all preference optimization methods can be understood as margin ranking in log-probability space, with DPO's target margin m = δ_ref potentially negative. This perspective naturally suggests techniques from the learning-to-rank literature: hard negative mining (prioritize training on pairs where the margin is smallest or negative), curriculum learning (start with easy pairs where δ_ref is positive, gradually introduce harder pairs), and margin adaptation (adjust the target margin based on training progress). A study applying these techniques to CPO or E-CPOC training — for instance, sampling training batches to oversample pairs where δ_ref is negative, or scheduling γ from small to large during training — could improve sample efficiency and final alignment quality. The paper's γ sensitivity analysis (Table 5) shows a sharp threshold below which CPO provides minimal benefit; curriculum-based approaches might help the policy cross this threshold more reliably.

---

Practical Applications and Downstream Use Cases

Pre-training diagnostics for DPO deployments. The paper's most immediately actionable contribution is the δ_ref diagnostic: before training DPO on any preference dataset, compute δ_ref = log π_ref(y_w|x) - log π_ref(y_l|x) for all pairs. The fraction of pairs with δ_ref < 0 directly measures the reference policy's misalignment with the preference data. If this fraction is high (>30% in the paper's measurement), the practitioner should either (a) switch to CPO or E-CPOC, (b) adjust β downward to strengthen the reward signal relative to the KL penalty (reducing the right-hand side −Δr*/β in the violation condition), or (c) invest in improving the reference policy before alignment. This diagnostic requires one forward pass over the dataset — negligible cost relative to the full training pipeline — and provides an evidence-based decision criterion that did not exist before this work. A deployment team currently using DPO on a new domain-specific dataset can run this diagnostic in minutes and determine whether their configuration is at risk.

Alignment from imperfect or partially trained reference models. The misaligned reference experiments (Appendix A.2) demonstrate that CPO maintains strong performance (LC >27%) even when the reference is systematically corrupted, while DPO collapses (LC drops to <17%). This makes CPO attractive for scenarios where the reference policy is necessarily imperfect: (a) alignment of smaller models where capacity limits prevent the reference from already capturing preference distinctions, (b) domain-specific alignment where the reference is trained on general data and transferred to a specialized domain, (c) alignment in low-resource languages where high-quality SFT data for the reference is limited. In each case, CPO's constraint term provides robustness to reference policy quality that DPO lacks, reducing the need for expensive reference model improvement before alignment can begin.

Cost-sensitive alignment pipelines where reward model training is infeasible. CPO and E-CPOC retain DPO's key practical advantage: no separate reward model training or RL infrastructure is required. The additional computational cost over DPO is one scalar operation per training sample (subtracting the precomputed margin from the logit). This makes CPO deployable in the same resource-constrained settings where DPO is currently used — on-device fine-tuning, rapid iteration on small preference datasets, alignment of models where training a reward model would double the GPU requirements. The paper's efficiency analysis (Appendix I) shows that CPO's precomputation overhead is essentially identical to DPO's — the margin terms γ̃_ref or Φ_cons are computed during the same forward pass that DPO already requires for δ_ref precomputation. For teams currently using DPO, switching to CPO means changing ~5 lines of training code and adding one hyperparameter (γ), with no increase in per-iteration cost and a demonstrated improvement on standard benchmarks (+0.55% WR on AlpacaEval 2, +3.7% on Arena-Hard). The sensitivity analysis (Appendix A.3) shows that γ tuning is feasible within a modest sweep (5-8 values), making adoption practical even for teams without extensive hyperparameter optimization resources.

(Conditional) When to Prefer This Method

The paper explicitly positions CPO and E-CPOC as alternatives to DPO when the reference policy is not already well-aligned with preference data. It does not claim CPO should replace RLHF when computational resources permit explicit reward modeling and RL training. The decision rule implicit in the paper's results and theoretical analysis is:

• Prefer CPO or E-CPOC over DPO when: (a) the fraction of training pairs with δ_ref < 0 is non-trivial (>20%), as measured by a pre-training diagnostic; (b) the reference policy is not specifically optimized for the target preference distinctions (e.g., a generic instruction-tuned model applied to a specialized domain); (c) β is moderate to large (strong KL regularization), which amplifies the assumption violation; or (d) robustness to reference policy quality is desired without the cost of training a reward model. The misaligned reference experiments (Table 4) show CPO maintains performance when DPO degrades — this is the clearest practical differentiator.

• Prefer standard RLHF over CPO when: (a) the computational budget allows training an explicit reward model and running RL; (b) the reward model is expected to generalize well beyond the preference dataset (RLHF's explicit reward model may handle distribution shift better than implicit reward reparameterization, as suggested by Lin et al., 2024); or (c) exact control over the KL penalty is required for safety or stability reasons, since RLHF provides direct β tuning in the objective while CPO introduces the additional γ hyperparameter whose interaction with β is only partly characterized (the γ sensitivity analysis in Table 5 is empirical, not theoretical).

E-CPOC is theoretically preferable to CPO when formal alignment guarantees with verifiable conditions are needed (Theorem L.17, Corollary L.18), but this claim is not yet empirically validated since E-CPOC is untested. A practitioner choosing between CPO and E-CPOC today should prefer CPO for its simplicity and empirical validation, and should treat E-CPOC as a promising theoretical direction pending experimental confirmation.