A General Theoretical Paradigm to Understand Learning from Human Preferences

ArXiv: 2310.12036

🎯 Pitch

RLHF and DPO both collapse to a degenerate policy when human preferences are deterministic—the KL-regularization term vanishes entirely, regardless of regularization strength. This paper proves that this failure is baked into the Bradley-Terry model and proposes Identity-PO, which stays robust by expressing pairwise preferences directly without the pointwise reward approximation.

---

1. Executive Summary

This paper introduces a general theoretical framework for learning from human preferences by deriving a unified objective called Ψ-preference optimisation (ΨPO), which expresses the learning problem entirely in terms of pairwise preferences — bypassing both the pointwise reward approximation and the reward model generalisation assumptions that underpin standard RLHF and DPO. The authors analyze RLHF and DPO as special cases of ΨPO (corresponding to the logit mapping Ψ(q) = log(q/(1−q))) and identify a fundamental vulnerability: when preferences are deterministic or nearly deterministic, the Bradley-Terry modelisation forces the KL-regularisation term to become arbitrarily weak, causing both methods to converge to degenerate policies that ignore the reference policy entirely regardless of the regularisation strength τ. As a principled solution, they propose Identity-PO (IPO), the special case of ΨPO with Ψ set to the identity mapping, and derive a sampled loss function — a simple squared regression of the log-likelihood ratio gap to τ⁻¹/2 — that learns directly from pairwise preferences without reward modelling while maintaining bounded regularisation even with {0,1}-valued empirical preferences, establishing that a bounded Ψ is sufficient to prevent overfitting to the preference dataset when preferences are sampled.

2. Context and Motivation

The Core Problem: A Flaw in the Foundation of Learning from Human Preferences

The dominant paradigm for aligning generative language models with human values — Reinforcement Learning from Human Feedback (RLHF) and its derivatives like Direct Preference Optimisation (DPO) — rests on two approximations that the paper argues are not merely implementation details but potential sources of systematic failure. These approximations have been so widely adopted that they are often treated as part of the definition of the problem rather than as design choices to be scrutinized. This paper steps back and asks: what happens when we remove them, and what do we gain by doing so?

The two approximations are laid out clearly in the abstract:

"The first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy."

The first approximation is the Bradley-Terry modelisation — the assumption that human preferences can be represented as a sigmoid of the difference between two scalar "Elo scores" (pointwise rewards). The second is the reward model generalisation assumption — that a classifier trained on preference data collected under one policy (the behavior policy µ) will produce accurate reward estimates for actions drawn from a different policy (the one being optimized). DPO made progress by eliminating the second approximation: it learns a policy directly from preference data without training an explicit reward model that must generalize out-of-distribution. But DPO retains — and indeed bakes in more deeply — the first approximation.

The paper's central motivation is that this remaining Bradley-Terry assumption is not innocuous, and that its failure mode — which the paper identifies and characterizes — can cause both RLHF and DPO to produce policies that are fundamentally misaligned with the intentions behind the KL-regularisation term that is supposed to keep them safe.

Why This Matters: The KL-Regularisation Security Blanket

To understand the practical stakes, it's essential to understand why both RLHF and DPO include a KL-regularisation term that penalizes deviation from a reference policy πᵣₑf. The reference policy is typically the base pretrained model before any preference fine-tuning. The KL penalty serves as a safety constraint: it prevents the optimized policy from drifting too far from a known, reasonably safe behavior distribution while pursuing higher reward. This is not merely a theoretical nicety — model drift is a documented failure mode in language model fine-tuning (Lazaridou et al., 2020; Lu et al., 2020), where optimization can cause the model to produce degenerate outputs that technically maximize a learned reward signal but are no longer coherent, factual, or safe.

The KL-regularised objective takes the form (Equation 3):

$J(\pi) = \mathbb{E}_{\pi}[r(x, y)] - \tau D_{KL}(\pi \| \pi_{ref})$

where τ controls the strength of the regularisation. When τ is large, the policy is pulled strongly toward πᵣₑf. When τ is small, reward maximization dominates. The assumption underlying this design is that τ provides a controllable knob for practitioners to trade off between exploiting the learned preferences and staying close to safe behavior.

The paper's key insight is that this knob breaks when preferences are deterministic or near-deterministic. Section 4.2 demonstrates this with a simple, devastating example:

"Consider the simple example where we have two actions y and y′ such that p*(y ≻ y′) = 1, i.e., y is always preferred to y′. Then the Bradley-Terry model would require that (r(y) − r(y′)) → +∞ to satisfy (1). If we plug this into the optimal policy (7) then we would get that π*(y′)/π*(y) = 0 (i.e., π*(y′) = 0) irrespective of what constant τ is used for the KL-regularisation."

In plain language: when the data says "y is always better than y′", the Bradley-Terry model — which maps preference probabilities to reward differences via the logit function Ψ(q) = log(q/(1−q)) — demands an infinite reward gap. The closed-form optimal policy (Equation 7) then assigns zero probability to y′, completely ignoring the reference policy's distribution over these actions. The parameter τ becomes irrelevant — not because τ itself is small, but because the logit mapping amplifies extreme preferences so aggressively that τ cannot meaningfully counteract it.

This is not just a theoretical pathology. In practice, preference datasets are finite, and empirical preference probabilities often land on exactly 0 or 1 — especially in large action spaces where most pairs of actions are never directly compared, or where one action happens to win all observed comparisons in a small sample. The paper explicitly connects this to the scale of modern LLM deployments:

"This means that overfitting can be a substantial empirical issue, especially when the context and action spaces are extremely large as it is for large language models."

Conflicting Prior Evidence and the Unexplained Gap

The paper is motivated by a genuine tension in the literature between theoretical guarantees and practical behavior. On one side, RLHF has been deployed with remarkable success in systems like InstructGPT and GPT-4 (Ouyang et al., 2022; OpenAI, 2023). On the other side, the theoretical foundations remain thin — the paper notes that:

"Despite this practical success, little is known regarding theoretical foundations of these practical methods."

Prior theoretical work on preference-based and dueling bandits (Busa-Fekete et al., 2014, 2013; Novoseller et al., 2020; Pacchiano et al., 2023) provides regret bounds in standard bandit settings, but these analyses do not address the specific approximations (Bradley-Terry modelisation, reward model generalisation) that characterize the RLHF/DPO pipeline as deployed in language modeling. The gap between the theoretical bandit literature and the practical LLM alignment pipeline is wide, and this paper explicitly aims to bridge it by providing a framework that can analyze the actual objectives being optimized in practice.

DPO (Rafailov et al., 2023) was proposed as a simplification — eliminate the reward model, optimize the policy directly, and avoid the generalisation problem. But the paper observes that DPO's own derivation still assumes the Bradley-Terry model holds. More subtly, DPO's claim to equivalence with RLHF under the BT assumption means that DPO inherits the same vulnerability when preferences are near-deterministic, but without the implicit regularisation that the reward modelling stage can sometimes provide (discussed below).

Where Existing Approaches Fall Short

The paper identifies specific limitations in both RLHF and DPO, but importantly, it distinguishes between theoretical vulnerabilities and practical mitigations that partially mask them.

RLHF's implicit safety net: reward model underfitting. A subtle but crucial observation in Section 4.2 is that standard RLHF may be partially protected from the deterministic-preference problem because of a flaw in its own pipeline — but one that turns out to be beneficial:

"While a purported advantage of DPO is that it avoids the need to fit a reward function, we observe that in practice when empirical preference probabilities are in the set {0, 1}, the reward function ends up being underfit. The optimal rewards in the presence of {0,1} preference probabilities are infinite, but these values are avoided, and indeed regularisation of the reward function has been observed to be an important aspect of RLHF training in practice."

In other words, RLHF's two-stage design introduces an unintended regularisation through the reward model training process. The reward model, being a finite-capacity neural network trained with finite data, cannot actually achieve infinite reward differences even when the data demands them. This "underfitting" of the reward model — typically considered a limitation — actually prevents the policy optimization stage from receiving the extreme reward signals that would cause it to ignore the KL penalty. DPO, by collapsing everything into a single end-to-end loss, loses this implicit protection:

"DPO, in avoiding the training of the reward function, loses the regularisation of the policy that the underfitted reward function affords."

This is a genuinely non-obvious insight: making the pipeline simpler and more direct can actually make it more vulnerable, because it removes an accidental source of beneficial regularisation.

The Bradley-Terry model as a single point of failure. Both RLHF and DPO share the assumption that the preference function decomposes as:

$p(y \succ y' | x) = \sigma(r(x, y) - r(x, y'))$

where σ is the sigmoid. This is not a neutral modeling choice — it imposes a specific functional form that is transitive (if A is preferred to B, and B is preferred to C, then the reward differences enforce that A must be preferred to C) and additive in the reward differences. Real human preferences can violate both of these properties — they can exhibit cycles (A > B, B > C, C > A is possible in human judgment) and non-transitive structures that the Bradley-Terry model fundamentally cannot represent. The paper's ΨPO framework generalizes this by allowing any non-decreasing function Ψ, not just the logit mapping that enforces the BT structure.

No framework for analyzing the objective itself. Perhaps the deepest gap the paper identifies is conceptual: the field lacked a language for talking about the objective function of preference optimization independently of the algorithmic pipeline used to optimize it. RLHF defines the objective implicitly through the reward modelling + RL pipeline. DPO defines it through a specific loss function. Neither provides a direct answer to: what are we actually trying to maximize, and what is the role of the BT assumption in that task?

How This Paper Positions Itself

The paper's contribution is fundamentally analytical and conceptual, not algorithmic in the narrow sense. It does not propose a new architecture, a new training trick, or a new benchmark. Instead, it provides:

1. A unifying mathematical framework (ΨPO) that expresses the preference optimization objective purely in terms of pairwise preference probabilities and a non-decreasing function Ψ. This framework reveals that RLHF and DPO are instances of the same underlying objective with a specific choice of Ψ — the logit mapping Ψ(q) = log(q/(1−q)) — that corresponds to the Bradley-Terry modelisation.

2. An analysis of why that specific Ψ is problematic: because it is unbounded at 0 and 1, causing the KL regularisation to become ineffective exactly when empirical preferences are most confident.

3. A constructive demonstration that a bounded Ψ (specifically, the identity Ψ(q) = q) avoids the problem entirely. The IPO algorithm that emerges from this choice is not the paper's primary contribution — it's a proof of concept that the analysis has practical teeth. The paper is careful to derive a computationally tractable sampled loss for IPO (Equation 17) that is simple enough to implement in a few lines of code, and to demonstrate on minimal bandit examples (Section 5.3 and 5.4) that the predicted qualitative differences between DPO and IPO do manifest in practice.

The paper explicitly connects to the DPO lineage and positions itself as both an extension and a critique. It adopts DPO's key insight — that the optimal policy for the KL-regularised objective has a closed form that can be manipulated to avoid explicit reward modelling — but shows that DPO's derivation is tied to a specific, brittle choice of Ψ. By generalizing Ψ, the paper opens up a space of possible preference optimization objectives, of which IPO is just one potentially useful point.

The title itself — "A General Theoretical Paradigm" — signals the ambition: this is not a paper about a new algorithm, but about a new way of thinking about what it means to learn from preferences, one that disentangles the objective from the modeling assumptions that have been implicitly bundled with it in prior work.

3. Technical Approach

3.1 Reader Orientation

This paper does not build a new system or train a model — it develops a mathematical framework for understanding and designing objective functions for learning from human preferences. The core problem it solves is designing loss functions that stay well-behaved (properly regularised) even when training data contains deterministic or near-deterministic preferences, which the paper identifies as a fundamental vulnerability in both standard RLHF and DPO.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's contribution operates at the objective function design level, not the implementation level. Conceptually, the architecture has four layers:

1. Preference Data — a dataset of context-prompted action pairs with human preferences (e.g., $(x, y_w, y_l)$ where $y_w$ is preferred over $y_l$ given context $x$). This is the same data used by RLHF and DPO.

2. The $\Psi$PO Objective (Equation 6) — a unified optimisation target parameterized by a non-decreasing function $\Psi : [0,1] \to \mathbb{R}$. Given $\Psi$, a reference policy $\pi_{ref}$, and a regularisation strength $\tau$, the objective defines what it means to be an optimal policy. Different choices of $\Psi$ produce different concrete algorithms.

3. The Analytic Solution (Equation 9) — for any choice of $\Psi$, the optimal policy under the KL-regularised objective has a closed form: $\pi^*(y) \propto \pi_{ref}(y) \exp(\tau^{-1} g(y))$ where $g(y) = \mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))]$. This allows reasoning about the policy without actually running optimization.

4. The Sampled Loss — a practical, computationally tractable loss function derived from the analytic solution that can be optimized directly on the preference dataset without training a reward model or running RL. For the specific choice $\Psi = \text{Identity}$ (IPO), this reduces to a simple squared regression (Equation 17).

Information flows as follows: the designer chooses $\Psi$ → the $\Psi$PO objective defines what optimal means for that $\Psi$ → the analytic solution reveals the structure of the optimal policy → a sampled loss is derived for empirical optimization → gradient-based training on the preference dataset yields an approximately optimal policy.

3.3 Roadmap for the Deep Dive

• First, the general $\Psi$PO objective (Equation 6) — what it maximizes, what each symbol means, and why it generalizes prior work — since everything else builds on this foundation.
• Second, the analytic solution (Equation 9) — the closed-form expression for the optimal policy, how it reveals the role of $\Psi$, and why the boundedness of $\Psi$ (or lack thereof) controls the effectiveness of KL regularisation — because this is the paper's central analytical tool.
• Third, the RLHF/DPO analysis as $\Psi = \text{logit}$ — how the Bradley-Terry modelisation corresponds to choosing the logit function for $\Psi$, why this mapping is unbounded, and how that unboundedness causes the regularisation failure identified in Section 4.2.
• Fourth, the IPO derivation ($\Psi = \text{Identity}$) — how choosing the identity mapping leads to a bounded objective, why the optimal policy becomes $\pi^*(y) \propto \pi_{ref}(y) \exp(\tau^{-1} p^*(y \succ \mu))$, and why the KL regularisation now remains effective even with {0,1}-valued preferences.
• Fifth, the root-finding reformulation and IPO sampled loss — how the authors convert the analytic expression into a practical regression loss (Equation 17) by setting up a system of equations and proving that the unique minimizer of a squared loss recovers the optimal policy (Theorem 2).
• Sixth, the Proposition 3 equivalence proof — how the population loss (Equation 13) is shown to equal a loss expressed in terms of sampled binary preference indicators (Equation 16), enabling empirical optimization from a finite dataset.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical analysis paper whose core idea is that the choice of the non-linearity $\Psi$ in a preference-optimization objective determines whether KL regularisation remains effective under deterministic or near-deterministic empirical preferences, and that switching to a bounded $\Psi$ (specifically, the identity function) resolves a fundamental vulnerability shared by RLHF and DPO.

---

The $\Psi$PO Objective: A Unified Expression for Preference Optimization

The paper's central construction is a single objective function that encompasses both RLHF and DPO as special cases while revealing a design dimension — the choice of $\Psi$ — that prior work implicitly fixed. The objective balances two competing forces: maximizing a (potentially nonlinear) function of how much the policy's actions are preferred over a reference distribution's actions, and penalizing deviation from a known-safe reference policy.

The $\Psi$-preference optimisation objective is defined in Equation (6):

$\max_{\pi} \ \mathbb{E}_{\substack{x \sim \rho \\ y \sim \pi(\cdot|x) \\ y' \sim \mu(\cdot|x)}} [\Psi(p^*(y \succ y' | x))] - \tau D_{KL}(\pi \| \pi_{ref})$

where:

• $\pi$ is the policy being optimized — a mapping from contexts $x$ to distributions over actions $y$,
• $\rho$ is the distribution over contexts (e.g., the distribution of prompts the system receives),
• $\mu$ is the behavior policy — the policy that generated the preference data (the actions $y'$ it generates serve as comparison points),
• $p^*(y \succ y' | x)$ is the true human preference probability: the probability, over the randomness of which human rater is asked, that action $y$ is judged better than action $y'$ given context $x$,
• $\Psi : [0,1] \to \mathbb{R}$ is any non-decreasing function — this is the design choice that parameterizes the objective family,
• $\pi_{ref}$ is the reference policy (typically the base pretrained model before preference fine-tuning),
• $\tau \in \mathbb{R}^*_+$ is a strictly positive regularisation parameter controlling the strength of the KL penalty,
• $D_{KL}(\pi \| \pi_{ref}) = \mathbb{E}_{x\sim\rho}[\text{KL}(\pi(\cdot|x) \| \pi_{ref}(\cdot|x))]$ is the expected KL divergence between the new policy and the reference.

What it computes: For a given policy $\pi$, the first term draws a context $x$, samples an action $y$ from the policy being evaluated, samples a comparison action $y'$ from the behavior policy $\mu$, looks up the true probability that a human would prefer $y$ over $y'$, applies the transformation $\Psi$ to that probability, and takes the expectation over all these random variables. This expected transformed preference is the "reward" the policy receives. The second term penalizes the policy for deviating from $\pi_{ref}$, with $\tau$ controlling the penalty strength. Maximizing the objective finds a policy that produces actions humans prefer (in the $\Psi$-transformed sense) while staying close to safe behavior.

Why this form: The objective is designed to express preference optimization in terms of pairwise preference probabilities directly, without first converting them to pointwise rewards. The expectation over $y' \sim \mu$ (the behavior policy) is crucial — it means the objective evaluates a candidate action $y$ by comparing it to the distribution of actions the system has data about, not to a single scalar reward function. The KL penalty is the standard mechanism for preventing model drift, borrowed from the regularised MDP literature (Geist et al., 2019). The key innovation is the introduction of $\Psi$ as an explicit design parameter — prior work implicitly set $\Psi$ to the logit function $\Psi(q) = \log(q/(1-q))$ without recognizing this as a choice with consequences. Making $\Psi$ explicit reveals that the mapping from preference probabilities to optimization incentives is a design degree of freedom, not a fixed consequence of the problem formulation.

A crucial operational detail: in practice $p^*$ is not directly observed — only binary preference samples $I(y, y'|x) \sim \text{Bernoulli}(p^*(y \succ y' | x))$ are available, taking value 1 when $y$ is preferred and 0 otherwise. The objective in Equation (6) is stated in terms of the true expectation $p^*$, and the paper later derives sampled losses that work with the binary indicators. This two-step structure — first reason about optimal policies under true preferences, then derive empirical estimators — is the analytical backbone of the paper.

---

The Analytic Solution: What the Optimal Policy Looks Like for Any $\Psi$

A powerful property of KL-regularised objectives of this form is that they admit a closed-form expression for the optimal policy. This is a standard result in the regularised MDP literature (Appendix A.1 provides the proof), but the paper applies it to the $\Psi$PO objective to reveal how $\Psi$ shapes the optimal policy.

The general solution, stated in Equation (9), is:

$\pi^*(y) \propto \pi_{ref}(y) \exp(\tau^{-1} g(y))$

where $g(y) = \mathbb{E}_{y' \sim \mu}[\Psi(p^*(y \succ y'))]$ is the expected $\Psi$-transformed preference of action $y$ over actions drawn from the behavior policy $\mu$, and the proportionality constant ensures $\pi^*$ sums to 1 over all actions. (Context $x$ is omitted from notation for clarity but the expression holds per-context.)

What it computes: Given a particular action $y$, the exponential term $\exp(\tau^{-1} g(y))$ is a multiplicative weight that modifies the reference policy's probability $\pi_{ref}(y)$. The weight is large when $g(y)$ is large — meaning action $y$ is strongly preferred (in the $\Psi$-transformed sense) over the comparison distribution $\mu$. The weight is small when $g(y)$ is small. The parameter $\tau$ controls sensitivity: when $\tau$ is small, even modest differences in $g(y)$ produce large differences in the weight, pushing $\pi^*$ toward actions with high $g(y)$. When $\tau$ is large, the weights are all close to 1, keeping $\pi^*$ close to $\pi_{ref}$.

Why this form: The exponential form emerges from the KL divergence penalty. KL divergence penalizes probability ratios logarithmically — $D_{KL}(\pi \| \pi_{ref}) = \mathbb{E}_\pi[\log(\pi/\pi_{ref})]$. The optimal tradeoff between maximizing expected reward and minimizing this log-ratio penalty always produces a policy proportional to $\pi_{ref}$ times an exponential of the reward, regardless of the specific reward function. This is the same structure as a Boltzmann distribution in statistical mechanics, where $\tau$ plays the role of temperature. The form reveals a separation of concerns: $\pi_{ref}$ provides the base distribution, $g(y)$ provides the action-specific scores, and $\tau^{-1}$ controls how aggressively the policy pursues high-scoring actions.

This analytic solution is the paper's primary analytical tool. It allows the authors to answer: for a given choice of $\Psi$, what is the optimal policy, and does it behave as intended? In particular, it reveals exactly when KL regularisation fails: when $g(y)$ can take on values that cause $\exp(\tau^{-1} g(y))$ to be zero or infinite — meaning $\pi^*(y)$ becomes 0 or 1 — regardless of $\tau$.

---

The RLHF/DPO Analysis: $\Psi$ as the Logit Function and Why It Fails

The paper's central critical analysis comes from examining what happens when $\Psi$ is set to the logit function $\Psi(q) = \log(q/(1-q))$, which is the implicit choice in both RLHF and DPO.

Proposition 1 establishes the connection formally: when $\Psi(q) = \log(q/(1-q))$ and the Bradley-Terry model holds — meaning there exists a reward function $r$ such that $p^*(y \succ y') = \sigma(r(y) - r(y'))$ — then the optimal policy for $\Psi$PO, the optimal policy for the RLHF objective (Equation 3), and the optimal policy for the DPO objective (Equation 5) are all identical.

The proof works by showing that under the Bradley-Terry assumption:

$\mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))] = \mathbb{E}_{y'\sim\mu}\left[\log\left(\frac{e^{r(y)}}{e^{r(y)} + e^{r(y')}}\right)\right] = \mathbb{E}_{y'\sim\mu}[r(y) - r(y')] = r(y) - \mathbb{E}_{y'\sim\mu}[r(y')]$

The expectation $\mathbb{E}_{y'\sim\mu}[r(y')]$ is a constant with respect to $y$, so maximizing $\mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))]$ is equivalent to maximizing $r(y)$. This means the $\Psi$PO objective reduces to the standard RLHF objective (maximize expected reward minus KL penalty), and the DPO equivalence follows from prior work (Rafailov et al., 2023; and the paper's Proposition 4 in Appendix B, which shows this equivalence holds even when $p^*$ does not correspond to a Bradley-Terry model, as long as a minimizer of the BT loss exists).

The analytic solution for this choice of $\Psi$ becomes (Equation 7, specialized from Equation 9):

$\pi^*(y) \propto \pi_{ref}(y) \exp\left(\tau^{-1} \mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))]\right)$

What $\Psi = \text{logit}$ does to preference probabilities: The logit function $\Psi(q) = \log(q/(1-q))$ maps the interval $(0,1)$ onto the entire real line $(-\infty, \infty)$. When a preference probability is $q = 0.5$ (complete indifference), $\Psi(0.5) = 0$ — no incentive. When $q = 0.9$, $\Psi(0.9) = \log(9) \approx 2.2$ — a moderate positive incentive. When $q \to 1^-$, $\Psi(q) \to +\infty$ — the incentive becomes infinitely large. When $q \to 0^+$, $\Psi(q) \to -\infty$ — the disincentive becomes infinitely negative.

Why this causes the regularisation failure: Plug a deterministic preference $p^*(y \succ y') = 1$ into the expression for $g(y)$. The logit function produces $g(y) \to +\infty$ (because $\Psi$ is unbounded above at 1). The exponential $\exp(\tau^{-1} g(y))$ then goes to infinity regardless of $\tau$ — no finite $\tau$ can dampen an infinite input. The proportionality in Equation (9) means that the relative probability of action $y$ compared to all other actions dominates completely:

$\frac{\pi^*(y)}{\pi^*(y')} = \frac{\pi_{ref}(y)}{\pi_{ref}(y')} \exp(\tau^{-1}(g(y) - g(y')))$

If $g(y) - g(y') \to \infty$ for some $y'$, then $\pi^*(y')/\pi^*(y) \to 0$, so $\pi^*(y') \to 0$ for any $y'$ that is strictly dominated. The reference policy's probabilities become irrelevant — the optimal policy is fully determined by which actions have won all comparisons, with $\tau$ playing no role at all.

The paper identifies this in Section 4.2 with a concrete example of two actions with $p^*(y \succ y') = 1$, concluding that "$\pi^*(y') = 0$ irrespective of what constant $\tau$ is used for the KL-regularisation." The asymptotic analysis in Section 5.3.1 confirms this: with two actions and deterministic preference $p^*(y_1 \succ y_2) = 1$, DPO converges to $\pi^*(y_1) = 1, \pi^*(y_2) = 0$ "regardless of the value of $\tau$."

This analysis reveals a deeper structural issue: the Bradley-Terry modelisation is not merely an assumption about the functional form of preferences — it is an assumption that forces the reward gap to be infinite for deterministic preferences, which in turn forces the optimal policy to be degenerate regardless of the regularisation strength. The paper's key theoretical contribution is identifying that this is not an inherent property of preference optimization, but a consequence of the specific choice $\Psi = \text{logit}$.

---

The Overfitting in the Finite-Data Regime

The problem compounds when preferences are estimated from finite data. Section 4.2 explains:

"Even if the true preference is, e.g., $p^*(y \succ y') = 0.8$, empirically it can be very possible when we only have a few data points to estimate $\hat{p}(y \succ y') = 1$, in which case the empirical optimal policy would make $\pi(y') = 0$ for any $\tau$."

The mechanism is: the empirical preference frequency $\hat{p}$ can land on exactly 0 or 1 due to sampling noise — especially in large action spaces where most pairs have few or zero comparisons (labeled data is expensive). The logit function amplifies these extrema into $\pm\infty$, and the optimal empirical policy eliminates any action that happens to have lost all its observed comparisons, even if the true preference is ambiguous. This is overfitting in the strict sense: the policy fits noise in the preference dataset rather than the underlying signal, and the KL penalty — designed precisely to prevent this — is powerless to stop it because the $\Psi$ mapping has already converted finite sampling noise into infinite reward gaps.

This analysis explains an otherwise puzzling practical phenomenon: why practitioners often observe that DPO can produce policies that diverge too far from the reference model even with apparently reasonable $\tau$ settings. The unboundedness of the logit function means that $\tau$ and the preference data interact in a way that is not monotonic — increasing $\tau$ provides more protection against moderate preference signals but is completely bypassed by extreme ones.

---

The RLHF Safety Net: Why Two-Stage Training Sometimes Shields Against This

Section 4.2 contains a subtle and important observation about why standard RLHF may be partially insulated from this failure mode in practice, despite sharing the same theoretical vulnerability:

"While a purported advantage of DPO is that it avoids the need to fit a reward function, we observe that in practice when empirical preference probabilities are in the set $\{0, 1\}$, the reward function ends up being underfit. The optimal rewards in the presence of $\{0,1\}$ preference probabilities are infinite, but these values are avoided, and indeed regularisation of the reward function has been observed to be an important aspect of RLHF training in practice."

The mechanism is: in RLHF, the reward model is a neural network trained with finite capacity and finite data to minimize the Bradley-Terry logistic regression loss. When preferences are deterministic, the theoretically optimal reward assigns infinite values — but a neural network trained with gradient descent for finite steps cannot and does not achieve this. The optimization plateaus at some finite (and therefore bounded) reward values. When these underfit rewards are passed to the policy optimization stage (PPO), the KL penalty in Equation (3) operates on bounded reward differences and therefore remains effective.

DPO, by eliminating the reward model, eliminates this beneficial underfitting. The DPO loss directly optimizes the policy, and the logit function's unboundedness operates directly on the policy's log-ratios without any intermediate model to absorb the extreme values. The paper summarizes this insight precisely:

"DPO, in avoiding the training of the reward function, loses the regularisation of the policy that the underfitted reward function affords."

This is an elegant instance of emergence through indirection: a perceived limitation of the two-stage RLHF pipeline (that the reward model cannot perfectly fit the data) turns out to be a crucial source of robustness that the more direct DPO approach unwittingly sacrifices. It also suggests that DPO practitioners may need additional forms of regularisation — such as early stopping, weight decay on the policy parameters, or explicit constraints on the log-ratio gap — to compensate for the regularisation that RLHF gets "for free" through reward model underfitting.

---

IPO: Setting $\Psi$ to the Identity Function

Having diagnosed the problem — $\Psi = \text{logit}$ is unbounded, causing KL regularisation to fail on extreme preferences — the paper proposes a constructive solution: choose $\Psi(q) = q$, the identity mapping. This produces the IPO objective (Equation 8):

$\max_{\pi} \ p^*_\rho(\pi \succ \mu) - \tau D_{KL}(\pi \| \pi_{ref})$

where $p^*_\rho(\pi \succ \mu) = \mathbb{E}_{x\sim\rho, y\sim\pi(\cdot|x)}[p^*(y \succ \mu | x)]$ is the total preference of the policy $\pi$ over the behavior distribution $\mu$ — the expected probability (not log-odds) that an action drawn from $\pi$ is preferred to an action drawn from $\mu$.

What it computes: The objective simply maximizes the expected probability that the policy's actions beat actions from $\mu$, minus the KL penalty. There is no nonlinear transformation of preference probabilities — $p^*(y \succ y') = 0.8$ contributes exactly 0.8 to the objective, $p^*(y \succ y') = 1.0$ contributes exactly 1.0. The scale is bounded: preference probabilities live in $[0,1]$, so $g(y) = \mathbb{E}_{y'\sim\mu}[p^*(y \succ y')] \in [0,1]$ for every action.

Why this form: The identity function is bounded on $[0,1]$. Unlike the logit function which maps the entire real line from a bounded interval, the identity leaves the values bounded. This means that no matter how extreme the preference data, the incentive $g(y)$ is always a finite number between 0 and 1. The exponential $\exp(\tau^{-1} g(y))$ is then also bounded — between $\exp(0) = 1$ and $\exp(\tau^{-1})$ — for any finite $\tau$. The relative probability of two actions is bounded by $\exp(\tau^{-1})$, and this bound tightens as $\tau$ increases. The KL regularisation remains effective: if $\tau$ is large, $\exp(\tau^{-1}) \approx 1$, and the optimal policy stays close to $\pi_{ref}$. If $\tau$ is small, $\exp(\tau^{-1})$ is large, and the policy can diverge further. This recovers the intended behavior — $\tau$ as a genuine control knob — that the logit function destroyed.

The asymptotic analysis in Section 5.3.1 demonstrates this concretely. For the two-action deterministic preference case ($p^*(y_1 \succ y_2) = 1$), the authors compute $p^*(y_1 \succ \mu) = 3/4$ and $p^*(y_2 \succ \mu) = 1/4$ (with uniform $\mu$). The IPO optimal policy becomes:

$\pi^*(y_1) = \frac{\exp(0.75 \tau^{-1})}{\exp(0.75 \tau^{-1}) + \exp(0.25 \tau^{-1})} = \sigma(0.5 \tau^{-1})$

$\pi^*(y_2) = \sigma(-0.5 \tau^{-1})$

where $\sigma$ is the sigmoid function. Now:

• As $\tau \to +\infty$ (strong regularisation), $\sigma(0) = 0.5$, so $\pi^* \to (0.5, 0.5)$ — back to the uniform reference policy.
• As $\tau \to 0^+$ (weak regularisation), $\sigma(+\infty) = 1$, so $\pi^* \to (1, 0)$ — the deterministic optimal policy.

This is the intended behavior: $\tau$ smoothly interpolates between the reference policy and the preference-maximizing policy. The contrast with DPO — which jumps directly to $(1,0)$ for any $\tau$ — is stark and captures the paper's central argument.

---

Root-Finding Reformulation: From Analytic Solution to Optimisable Loss

The analytic solution in Equation (9) gives the optimal policy, but it requires knowing $p^*$ — the true preference probabilities — which are unavailable in practice. The paper needs to derive an empirical loss function that, when minimized on a finite dataset of pairwise preferences, recovers (approximately) the optimal policy. The technique used is root-finding reformulation, following the general approach introduced by Rafailov et al. (2023) for DPO.

The derivation proceeds as follows. Starting from the analytic solution (Equation 9), for any two actions $y, y'$ in the support of $\pi_{ref}$, the ratio of their optimal probabilities is:

$\frac{\pi^*(y)}{\pi^*(y')} = \frac{\pi_{ref}(y)}{\pi_{ref}(y')} \exp(\tau^{-1}(g(y) - g(y')))$

Taking logs and rearranging:

$\log\left(\frac{\pi^*(y)}{\pi^*(y')}\right) - \log\left(\frac{\pi_{ref}(y)}{\pi_{ref}(y')}\right) = \tau^{-1}(g(y) - g(y'))$

Define the log-ratio gap for a candidate policy $\pi$:

$h_\pi(y, y') = \log\left(\frac{\pi(y)}{\pi(y')}\right) - \log\left(\frac{\pi_{ref}(y)}{\pi_{ref}(y')}\right) = \log\left(\frac{\pi(y) \pi_{ref}(y')}{\pi(y') \pi_{ref}(y)}\right)$

$h_\pi(y, y')$ measures how much the log-probability ratio of $(y, y')$ under $\pi$ differs from that under $\pi_{ref}$. When $h_\pi(y, y') > 0$, $\pi$ assigns relatively more probability to $y$ than $\pi_{ref}$ does. When $h_\pi(y, y') < 0$, $\pi$ assigns relatively less.

The optimal policy $\pi^*$ satisfies, for all $y, y'$:

$h_{\pi^*}(y, y') = \tau^{-1}(g(y) - g(y'))$

For $\Psi = \text{Identity}$, $g(y) = p^*(y \succ \mu)$, so:

$h_{\pi^*}(y, y') = \tau^{-1}(p^*(y \succ \mu) - p^*(y' \succ \mu))$

This is a system of equations: for every pair of actions, the log-ratio gap under the optimal policy should equal $\tau^{-1}$ times the difference in expected preference probabilities. The insight is that solving all these equations simultaneously is equivalent to finding the optimal policy — and this can be cast as a minimization problem.

The IPO population loss (Equation 13):

$L(\pi) = \mathbb{E}_{y,y' \sim \mu}\left[\left(h_\pi(y, y') - \frac{p^*(y \succ \mu) - p^*(y' \succ \mu)}{\tau}\right)^2\right]$

What it computes: For each pair of actions $(y, y')$ drawn independently from the behavior policy $\mu$, compute the squared difference between (1) the log-ratio gap $h_\pi(y, y')$ and (2) the target value $\tau^{-1}(p^*(y \succ \mu) - p^*(y' \succ \mu))$. Average this squared error over the distribution of action pairs. The loss is zero exactly when $\pi = \pi^*$ (when the equations are satisfied for all pairs), and positive otherwise.

Why this form: Squared loss is the natural choice for solving a system of equations — it is differentiable, convex in the appropriate parameterization, and its unique minimizer (when it exists) is the solution to the equations. The expectation over $\mu$ ensures that pairs are weighted by how often they appear in the data, which is the distribution under which empirical estimates are available.

Theorem 2 establishes that this loss has a unique local/global minimum in the set $\Pi$ of policies whose support equals $\text{Supp}(\mu) = \text{Supp}(\pi_{ref})$, and that minimizer is exactly $\pi^*$. The proof (which we walk through below) is constructive and reveals why the squared loss is particularly well-suited to this problem.

---

Theorem 2: Uniqueness of the IPO Minimizer

Theorem 2 is the paper's main technical guarantee for the IPO loss. It states:

"Assume that $\text{Supp}(\mu) = \text{Supp}(\pi_{ref})$ and define $\Pi$ to be the set of policies $\pi$ such that $\text{Supp}(\pi) = \text{Supp}(\mu)$. Then $\pi \mapsto L(\pi)$ has a unique local/global minimum in $\Pi$, which is $\pi^*$."

The proof proceeds in three conceptual steps, which the paper executes formally in Appendix A.2.

Step 1: Parameterize policies by logits. Any policy $\pi \in \Pi$ can be represented by a vector of logits $s \in \mathbb{R}^{|J|}$ where $J = \text{Supp}(\mu)$, via the softmax:

$\pi_s(y) = \frac{\exp(s(y))}{\sum_{y' \in J} \exp(s(y'))}$

for $y \in J$, and $\pi_s(y) = 0$ otherwise. The log-ratio gap $h_\pi(y, y')$ simplifies under this parameterization to:

$h_\pi(y, y') = (s(y) - s(y')) - \log\left(\frac{\pi_{ref}(y')}{\pi_{ref}(y)}\right)$

Step 2: Show the loss is quadratic and convex. Substituting this expression into $L(\pi)$ (Equation 14 in the proof) yields an objective $L(s)$ that is quadratic in the logits $s$. Expanding the square produces three types of terms:

• A quadratic form: $\mathbb{E}_{y,y'\sim\mu}[(s(y) - s(y'))^2]$ — this is a quadratic function of $s$,
• Linear terms: terms where $s(y)$ is multiplied by constants involving $p^*(y \succ \mu)$ and $\log \pi_{ref}(y)$,
• Constant terms: independent of $s$.

The quadratic form can be rewritten (Equation 15) as:

$\sum_{y,y' \in J} \mu(y)\mu(y')(s(y) - s(y'))^2$

This is a positive-semidefinite quadratic form — it is never negative — which means $L(s)$ is a convex function. A fundamental property of convex functions is that every local minimum is also a global minimum (Boyd and Vandenberghe, 2004, Chap. 4). Since $L(\pi^*) = 0$ and $L(\pi) \geq 0$ for all $\pi$ (it is an expectation of squares), $\pi^*$ achieves the global minimum.

Step 3: Establish strict convexity up to an irrelevant direction. The quadratic form in Equation (15) penalizes differences between logits. The only direction in logit space that does not increase the quadratic form is the all-ones direction $e = (1, 1, \ldots, 1)$: adding any constant to all logits simultaneously leaves all differences $(s(y) - s(y'))$ unchanged. However, adding a constant to all logits also does not change the resulting softmax policy:

$\pi_{s + \lambda e}(y) = \frac{\exp(s(y) + \lambda)}{\sum_{y'} \exp(s(y') + \lambda)} = \frac{\exp(s(y))}{\sum_{y'} \exp(s(y'))} = \pi_s(y)$

Therefore, in the space of actual policies (not logits), the loss is strictly convex: any two distinct policies differ by more than just a constant shift, and the loss increases when moving between them. This strict convexity, combined with $\pi^*$ being a global minimizer, proves that $\pi^*$ is the unique global and local minimum in $\Pi$.

The support condition is necessary. Appendix A.2 provides a counterexample showing that when $\text{Supp}(\mu) \subsetneq \text{Supp}(\pi_{ref})$ — that is, when the behavior policy does not cover the entire action space that the reference policy considers — there can be infinitely many policies achieving zero loss. The intuition: if an action $y_3$ never appears in the preference data ($\mu(y_3) = 0$), then the loss places no constraints on $\pi(y_3)$ relative to other actions, and any value consistent with the constraints on the observed actions is equally optimal. The assumption $\text{Supp}(\mu) = \text{Supp}(\pi_{ref})$ ensures that the preference data covers all actions the reference policy considers, providing enough constraints to uniquely identify the optimal policy.

---

From Population Loss to Sampled Loss: Proposition 3

The population loss in Equation (13) is expressed in terms of $p^*(y \succ \mu)$, the expected preference probability. In practice, we have a dataset of binary preference indicators $I(y, y') \in \{0, 1\}$, not probabilities. Proposition 3 bridges this gap by proving:

$\mathbb{E}_{y,y' \sim \mu}\left[\left(h_\pi(y, y') - \frac{p^*(y \succ \mu) - p^*(y' \succ \mu)}{\tau}\right)^2\right] = \mathbb{E}_{y,y' \sim \mu}\left[\left(h_\pi(y, y') - \tau^{-1} I(y, y')\right)^2\right] + C$

where $I(y, y') \sim \text{Bernoulli}(p^*(y \succ y'))$ is the binary preference indicator (1 if $y$ is preferred to $y'$, 0 otherwise), and $C$ is a constant independent of $\pi$.

What this says: Replacing the expected preference difference $p^*(y \succ \mu) - p^*(y' \succ \mu)$ with the binary indicator $I(y, y')$ (scaled by $\tau^{-1}$) in the squared loss produces an objective that differs from the original only by an additive constant. Therefore, minimizing one is equivalent to minimizing the other — they share the same optimal policy.

Why this is non-trivial: At first glance, the two losses look different because $I(y, y')$ is a binary indicator of whether $y$ beats $y'$ in a single comparison, while $p^*(y \succ \mu) - p^*(y' \succ \mu)$ is the difference in expected preferences against the entire distribution $\mu$. The substitution works because of a symmetry property combined with the additive structure of $h_\pi$.

The proof (provided in full in the paper) works by expanding both quadratics and comparing cross-terms. The critical manipulation concerns:

$\mathbb{E}_{y,y' \sim \mu}[h_\pi(y, y') I(y, y')]$

Since $h_\pi(y, y') = \log(\pi(y)/\pi_{ref}(y)) - \log(\pi(y')/\pi_{ref}(y'))$ is anti-symmetric ($h_\pi(y, y') = -h_\pi(y', y)$) and $I(y, y')$ has the property that $\mathbb{E}_{y'}[I(y, y') | y] = p^*(y \succ \mu)$ and $\mathbb{E}_y[I(y, y') | y'] = 1 - p^*(y' \succ \mu)$, the cross-term simplifies to exactly the same expression as when $I(y, y')$ is replaced by $p^*(y \succ \mu) - p^*(y' \succ \mu)$. The constant $C$ absorbs the variance of $I$ (the squared terms $\mathbb{E}[I^2] - \mathbb{E}[p^2]$), which depends only on the data distribution, not on $\pi$.

The squaring operation is essential to this equivalence. A linear loss — e.g., $\mathbb{E}[\tau^{-1} I - h_\pi]$ — would not share this property because the expectation of $I$ conditional on $(y, y')$ depends on the specific pair, not just the marginal expectations.

---

The Empirical IPO Loss

The paper now derives the practical loss function that can be optimized from a dataset $D = (y_{w,i}, y_{l,i})_{i=1}^N$, where $y_{w,i}$ is the preferred action and $y_{l,i}$ is the dispreferred action for each pair.

Each datapoint $(y_{w,i}, y_{l,i})$ provides two terms to the expectation in Equation (16):

• $(y, y', I) = (y_{w,i}, y_{l,i}, 1)$: the preferred action beats the dispreferred one,
• $(y, y', I) = (y_{l,i}, y_{w,i}, 0)$: the dispreferred action loses to the preferred one (this is the symmetric complement, with $I = 0$).

Exploiting this symmetry and averaging over the dataset yields the empirical IPO loss (Equation 17):

$\mathbb{E}_{(y_w, y_l) \sim D}\left[\left(h_\pi(y_w, y_l) - \frac{\tau^{-1}}{2}\right)^2\right]$

where $h_\pi(y_w, y_l) = \log\left(\frac{\pi(y_w) \pi_{ref}(y_l)}{\pi(y_l) \pi_{ref}(y_w)}\right)$.

What it computes: For each preference pair in the dataset, compute the log-ratio gap $h_\pi(y_w, y_l)$ between the preferred and dispreferred actions, subtract the constant target $\tau^{-1}/2$, square the result, and average over the dataset. If $h_\pi(y_w, y_l) = \tau^{-1}/2$ for all pairs, the loss is zero. If the log-ratio gap is larger or smaller than the target, the loss is positive and quadratic in the deviation.

Why this form: The constant $\tau^{-1}/2$ emerges from the algebra of symmetrizing the two contributions $(h_\pi - \tau^{-1})^2$ (from $(y_w, y_l, 1)$) and $h_\pi^2$ (from $(y_l, y_w, 0)$):

$\frac{1}{2}\left[(h_\pi - \tau^{-1})^2 + (-h_\pi - 0)^2\right] = h_\pi^2 - \tau^{-1}h_\pi + \frac{\tau^{-2}}{2} = \left(h_\pi - \frac{\tau^{-1}}{2}\right)^2 + \text{constant}$

The loss is remarkably simple: IPO learns by regressing the gap between the policy's log-ratio and the reference policy's log-ratio to $\tau^{-1}/2$. This has an intuitive interpretation:

• The target $\tau^{-1}/2$ is positive: the policy should assign a higher log-probability to $y_w$ relative to $y_l$ than the reference policy does, and the magnitude of this increase is controlled by $\tau^{-1}$.
• When $\tau$ is large (strong regularisation), $\tau^{-1}/2$ is small, meaning the policy should deviate only slightly from $\pi_{ref}$'s relative probabilities.
• When $\tau$ is small (weak regularisation), $\tau^{-1}/2$ is large, allowing the policy to strongly favor $y_w$ over $y_l$.

Crucially, the target is bounded and finite regardless of how deterministic the preference data is — there is no mechanism for $\tau^{-1}/2$ to become infinite, and the squared loss penalizes large deviations quadratically rather than exponentially. This is the mathematical realization of the paper's claim that IPO "always regularizes its solution towards $\pi_{ref}$ by controlling the gap between the log-likelihood ratios."

Algorithm 1 in the paper summarizes the entire procedure:

1. Define $h_\pi(y, y', x) = \log(\frac{\pi(y|x) \pi_{ref}(y'|x)}{\pi(y'|x) \pi_{ref}(y|x)})$ (restoring context $x$).
2. Starting from $\pi = \pi_{ref}$, minimize the empirical loss $\mathbb{E}_{(y_w, y_l, x) \sim D}[(h_\pi(y_w, y_l, x) - \tau^{-1}/2)^2]$ using gradient-based optimization.

---

Design Choices and Their Justifications

Choice of $\Psi = \text{Identity}$ over $\Psi = \text{logit}$: Boundedness on $[0,1]$. The identity maps the preference probability interval $[0,1]$ onto itself, while the logit maps it onto $(-\infty, \infty)$. The boundedness means that $g(y) = \mathbb{E}_{y'\sim\mu}[p^*(y \succ y')]$ is always in $[0,1]$, and the exponential weight $\exp(\tau^{-1} g(y))$ is bounded between $1$ and $\exp(\tau^{-1})$. The KL penalty therefore cannot be overwhelmed by infinite incentives, regardless of how extreme the preferences are.

Choice of squared loss over DPO's logistic loss: The squared loss in Equation (13) is the natural objective for solving the root-finding equations $h_\pi(y, y') = \tau^{-1}(g(y) - g(y'))$. It produces a convex optimization problem with a unique minimizer (Theorem 2). In contrast, DPO's loss — $-\log \sigma(\tau h_\pi)$ — can be driven arbitrarily close to zero by making $h_\pi$ arbitrarily large for winning pairs, which is exactly the unboundedness that causes overfitting.

Symmetrization in the empirical loss: Using both $(y_w, y_l, 1)$ and $(y_l, y_w, 0)$ contributions from each preference pair reduces variance in the gradient estimate and ensures the loss depends only on the relative ordering of the pair, not on arbitrary labeling conventions. The algebra that collapses this to a single squared term centered at $\tau^{-1}/2$ is an elegant simplification.

Assumption $\text{Supp}(\mu) = \text{Supp}(\pi_{ref})$: This is required for uniqueness of the solution (Theorem 2, with the counterexample in Appendix A.2). In practice, the data distribution $\mu$ is determined by the data collection process, and $\pi_{ref}$ is the base model. The assumption says that the data must cover all actions that the reference model assigns non-zero probability to — otherwise, the loss cannot uniquely determine the optimal probabilities for unobserved actions.

Derivation from population to sampled loss via Proposition 3: The paper could have directly proposed the empirical loss $(h_\pi - \tau^{-1}/2)^2$ as a heuristic, but Proposition 3 provides the theoretical justification by proving it is an unbiased estimator (up to a constant) of the population IPO objective. This connects the practical algorithm back to the principled $\Psi$PO framework.

4. Key Insights and Innovations

Innovation 1: The Choice of Non-linearity Ψ is a Design Degree of Freedom, Not a Fixed Consequence of the Problem

Prior to this paper, the field treated the Bradley-Terry modelisation — the assumption that pairwise preferences decompose into a sigmoid of pointwise reward differences — as essentially synonymous with the problem of learning from preferences. RLHF adopts it explicitly through the logistic regression loss for reward model training (Equation 2). DPO bakes it in through the logistic loss on policy log-ratios (Equation 4). The theoretical equivalence between RLHF and DPO (Proposition 1 and Rafailov et al., 2023) is predicated on it. The field had converged on a single answer to the question "what does it mean to optimize for human preferences?" — maximize the log-odds of being preferred.

The paper's most fundamental conceptual move is to extract the Bradley-Terry assumption from the definition of the problem and expose it as one point in a design space. The ΨPO objective (Equation 6) parameterizes the optimization target by an arbitrary non-decreasing function Ψ, revealing that what prior work treated as a fixed modeling choice is actually a hyperparameter of the objective function itself. The logit mapping Ψ(q) = log(q/(1−q)) is not the way to convert preference probabilities into optimization incentives — it is one way, with specific and potentially pathological properties.

This is a reframing of the first order. It changes the question from "how do we maximize reward under the Bradley-Terry model?" to "what function Ψ should we use to transform preference probabilities into optimization incentives, and what properties should it have?" The paper does not claim to have solved the problem of choosing the optimal Ψ — it merely opens the space and demonstrates that the standard choice has a specific failure mode. But opening that space is the enabling move for everything that follows. It is the difference between asking "how can we regularize DPO better?" (an incremental improvement question) and asking "what objective are we even optimizing, and is it the right one?" (a foundational question).

The evidence for this reframing is Proposition 1 itself: RLHF and DPO are proved to be the same ΨPO instantiation, which means any property they share stems from Ψ = logit. If they share a flaw, the flaw is in Ψ, not in the specific algorithmic realization. This diagnostic move — tracing empirical behavior to a mathematical property of Ψ rather than to implementation details — is what allows the paper to analyze both methods in a unified way and to propose a fix (IPO) that operates at the objective-design level rather than the algorithm-design level.

The significance is amplified by Proposition 4 (Appendix B), which shows that DPO and RLHF are equivalent even when the Bradley-Terry model does not hold — as long as a minimizer of the BT loss exists. This means the equivalence is more robust than the assumption that justifies it. But it also means that if Ψ = logit is problematic, both methods are problematic simultaneously — there is no diversity of objective between RLHF and DPO, only a diversity of implementation.

Why this is fundamental rather than incremental: prior work explored different algorithms to optimize the same underlying objective (RLHF via PPO vs. DPO via direct policy optimization). This paper steps outside that objective entirely and asks whether we should be optimizing something else. That is a category shift — it's not a better optimizer, it's a different target.

---

Innovation 2: The Unboundedness of Ψ as a Diagnostic for KL-Regularisation Failure

The paper's second contribution is identifying a precise, generalizable diagnostic for when a preference optimization objective will fail to properly regularize: the unboundedness of Ψ at 0 and 1. This is not merely an observation about the logit function — it is a criterion that can be applied to any candidate Ψ to predict whether KL regularisation will remain effective under extreme empirical preferences.

The diagnostic works as follows. The analytic solution to ΨPO (Equation 9) gives the optimal policy as:

$\pi^*(y) \propto \pi_{ref}(y) \exp(\tau^{-1} \mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))])$

If Ψ is unbounded above at 1 — meaning Ψ(q) → +∞ as q → 1⁻ — then any action y that has a deterministic preference over some comparison action (p*(y ≻ y′) = 1) receives an infinite incentive g(y). The exponential exp(τ⁻¹g(y)) then goes to infinity regardless of τ. The proportionality normalization drives all other action probabilities to zero. The KL penalty, which is designed to prevent this, cannot counteract an infinite input — no finite τ can dampen an infinite g(y). The regularisation knob breaks.

The paper demonstrates this concretely in Section 4.2 with the two-action deterministic preference example, but the diagnostic is general: any Ψ that diverges at the boundaries will cause KL regularisation to fail for actions involved in deterministic or near-deterministic comparisons. This is the paper's core explanatory insight — it answers why DPO and RLHF can overfit to the preference dataset, and why that overfitting is not fixable by simply increasing τ.

What makes this intellectually distinctive is that it identifies the failure mode in the objective function's mathematical structure, not in the optimization procedure. Prior work might have attributed DPO's tendency to diverge from π_ref to insufficient tuning of τ, to gradient dynamics, or to the lack of an explicit reward model. The paper argues that none of these are the root cause — the root cause is that Ψ = logit maps a bounded domain [0,1] onto an unbounded range (−∞, ∞), and this topological mismatch is what defeats the KL penalty. This is a structural diagnosis: it tells you that no amount of hyperparameter tuning can fix the problem at its source, because the problem is in the definition of the objective, not in its parameters.

The diagnostic also explains a puzzle that practitioners might have observed empirically: why DPO sometimes seems to "ignore" τ. If the dataset contains even a small number of deterministic preferences (and in large action spaces with finite data, this is essentially guaranteed — some action will win all its observed comparisons by chance), the logit mapping converts those into infinite incentives that dominate the objective regardless of τ. The paper doesn't just observe this behavior; it explains the mechanism by which it occurs, and in doing so provides a tool for reasoning about any future Ψ one might propose.

The significance of this diagnostic is that it is actionable and falsifiable. It predicts that any Ψ bounded on [0,1] — like the identity (IPO) or a sigmoid — will not suffer from this failure mode. The paper validates this prediction with the IPO analysis (Section 5.3.1) and the illustrative experiments (Section 5.4, Figures 1 and 2), where DPO collapses to deterministic policies for any τ while IPO smoothly interpolates between π_ref and the preference-maximizing policy as τ varies. The diagnostic is therefore not just a theoretical curiosity — it has empirical consequences that the paper demonstrates.

Why this is fundamental rather than incremental: before this paper, the field lacked a vocabulary for talking about why KL regularisation might fail. The papers that observed DPO producing degenerate policies would have been forced to speculate about optimization dynamics, data quality, or hyperparameter settings. This paper provides a clean, formal criterion — unboundedness of Ψ — that explains the phenomenon and points to a solution. This is the kind of diagnostic that can guide future algorithm design: if someone proposes a new preference optimization method, one can immediately check whether its implicit Ψ is bounded, and predict whether it will suffer from regularisation collapse.

---

Innovation 3: The Accidental Regularisation of RLHF as a Cautionary Tale About Pipeline Simplicity

Section 4.2 contains a genuinely counterintuitive observation: eliminating an intermediate model from a machine learning pipeline can make the overall system less robust, not more. DPO was motivated in part by the desire to simplify RLHF — remove the reward model, optimize the policy directly, avoid the generalisation error that comes from training a proxy. This is a natural and appealing design principle: fewer components means fewer places for things to go wrong. The paper argues that, in this particular case, precisely the opposite is true.

The argument, detailed in Section 4.2, is that the reward model in RLHF acts as an accidental bottleneck that prevents the unboundedness of Ψ = logit from reaching the policy optimization stage. When the Bradley-Terry logistic regression loss is minimized on empirical data with {0,1}-valued preferences, the theoretically optimal reward values are ±∞. But a neural network trained with finite capacity, finite data, and gradient-based optimization cannot achieve these values — it underfits, producing finite (and therefore bounded) reward estimates. When these underfit rewards are passed to the policy optimization stage (PPO), they are bounded, and the KL penalty in Equation (3) operates as intended. The reward model, by failing to perfectly fit the data, provides a form of regularisation that protects the downstream policy.

DPO eliminates this bottleneck. The logit function operates directly on the policy's own log-ratios without any intermediate model to absorb the extremes. As a result, deterministic or near-deterministic preferences in the data map directly to unbounded incentives in the loss, bypassing any opportunity for the system to "underfit" them. The paper states this precisely:

"DPO, in avoiding the training of the reward function, loses the regularisation of the policy that the underfitted reward function affords."

This is a significant insight because it inverts the usual narrative about DPO. DPO is often presented as a more elegant, more direct alternative to RLHF — it achieves the same theoretical optimum (under the BT assumption) with a simpler pipeline. The paper does not dispute the theoretical equivalence, but argues that the empirical behavior can be worse precisely because the pipeline is simpler. The complexity of RLHF — the fact that it has two stages with different optimization dynamics — turns out to be a source of robustness that was not intentionally designed but is nonetheless important.

The implication reaches beyond preference optimization. It is an instance of a more general phenomenon: intermediate representations can provide implicit regularisation by constraining the information that flows between stages of a pipeline. When a system is made more end-to-end, information that was previously compressed or distorted by intermediate representations now flows unimpeded, and the system can exploit that information in ways that were previously impossible — including overfitting to noise. This is a cautionary tale for the broader trend toward end-to-end learning: removing components can remove not just sources of error, but also sources of beneficial constraint.

The insight is backed by the paper's own empirical demonstrations (Figures 1 and 2), where DPO consistently converges to degenerate policies on datasets with deterministic or near-deterministic preferences while IPO does not. The RLHF comparison is not directly tested in these experiments (the paper does not implement RLHF on the bandit examples), but the theoretical argument — that RLHF's reward model underfitting provides protection — is logically self-contained: it follows from the fact that finite-capacity models cannot achieve infinite outputs, combined with the fact that DPO has no intermediate model to underfit.

Why this is fundamental rather than incremental: this is not a claim about how to tune hyperparameters or which architecture to use. It is a claim about the relationship between pipeline structure and robustness, and it suggests that the field's intuition that "simpler is better" may need to be qualified — sometimes the complexity is doing unseen work. This is the kind of insight that can influence how researchers think about system design across domains, not just in preference optimization.

---

Innovation 4: IPO's Sampled Loss Derivation as a Template for Deriving Practical Algorithms from ΨPO

The fourth contribution is methodological rather than diagnostic: the paper demonstrates a general derivation pipeline for converting any chosen Ψ in the ΨPO family into a practical, sampled loss function that can be optimized directly from a preference dataset without reward modelling or RL. This pipeline has three stages, and while the paper only executes it fully for Ψ = Identity (producing IPO), it is general enough to be applied to other choices of Ψ.

The three-stage pipeline works as follows:

Stage 1 (Analytic): Write the optimal policy for the chosen Ψ via Equation (9): π*(y) ∝ π_ref(y) exp(τ⁻¹ 𝔼_{y'∼μ}[Ψ(p*(y ≻ y′))]).

Stage 2 (Root-finding reformulation): Take ratios of optimal probabilities for pairs of actions, take logs, and rearrange to express the problem as a system of equations h_π(y, y′) = τ⁻¹(g(y) − g(y′)), where h_π is the log-ratio gap and g(y) = 𝔼_{y'∼μ}[Ψ(p*(y ≻ y′))]. A policy satisfies these equations iff it is optimal.

Stage 3 (Squared loss + sampling): Formulate a squared loss whose unique minimizer solves the system of equations from Stage 2. Prove that replacing the expected preference terms with sampled binary indicators I(y, y′) does not change the minimizer (the Proposition 3 step). This yields a loss expressible entirely in terms of (y_w, y_l, I) triples from the preference dataset.

For Ψ = Identity, this pipeline produces the elegantly simple IPO loss: 𝔼_{(y_w, y_l)∼D}[(h_π(y_w, y_l) − τ⁻¹/2)²]. For other choices of Ψ, Stage 3 would produce different target values — the squared loss would regress h_π(y, y′) to a different constant (or function) depending on Ψ.

What makes this contribution distinctive is that it is not just the IPO algorithm — it is the recipe for generating a family of algorithms. Someone who wants to use Ψ(q) = q², or Ψ(q) = √q, or any other non-decreasing bounded function, can follow the same three stages to derive a sampled loss. The paper provides the template; IPO is the proof of concept that the template works and produces a practically useful algorithm.

The Theorem 2 uniqueness proof is the linchpin of Stage 3. It shows that the squared loss formulation has exactly one minimizer in the relevant policy class, establishing that gradient-based optimization of the empirical loss will converge to a well-defined target (modulo optimization error). The convexity argument — that the loss is quadratic in the logits and strictly convex up to the irrelevant constant-shift direction — is a clean piece of analysis that would generalize to any Ψ that produces a g(y) expressible as an expectation over the data distribution.

The Proposition 3 equivalence is the bridge that makes the whole pipeline practical. Without it, the squared loss in Stage 3 would require knowing p*(y ≻ μ) for every action, which is not available from a finite preference dataset. Proposition 3 shows that substituting the binary indicator I(y, y′) — which is available from the dataset — produces a loss with the same minimizer. This is a non-trivial cancellation that depends on the anti-symmetry of h_π and the structure of the binary preference indicators; it is not generally true for arbitrary loss functions.

The significance of this contribution is that it decouples objective design from algorithm design in a principled way. Prior to this paper, if you wanted to change the preference optimization objective, you had to either modify the reward model training procedure (for RLHF) or derive a new closed-form loss from scratch (as DPO did for the BT assumption). The ΨPO framework, combined with the three-stage derivation pipeline, provides a systematic way to explore the space of possible objectives — choose a Ψ, derive the loss, test it. This is a significant methodological advance for the field, even if the paper only demonstrates it for one alternative Ψ.

Why this is fundamental rather than incremental: the derivation pipeline is not a small tweak to an existing algorithm. It is a new way of thinking about how to construct preference optimization algorithms — one that starts from the desired mathematical properties of the objective (encoded in Ψ) and systematically derives the corresponding practical loss. This is a design philosophy, not just a recipe.

---

Innovation 5: The Cyclic Preference Example as a Stress Test for Transitivity Assumptions

The paper's illustrative experiments in Section 5.4 include a subtle but powerful demonstration that is easy to overlook. The second dataset configuration, D₂ = {(y_a, y_b), (y_b, y_c), (y_c, y_a)}, represents a cyclic preference structure — A beats B, B beats C, C beats A. This is a preference pattern that the Bradley-Terry model cannot represent, because it assumes transitivity (reward differences are additive and necessarily produce transitive overall preferences). Human preferences, however, can be intransitive — individuals and groups can exhibit cyclic preferences, especially when different attributes are being traded off.

The paper does not analyze D₂ in detail (it focuses on D₁, the total ordering), but the inclusion of this dataset configuration in the experimental setup signals an important point: ΨPO with Ψ = Identity does not assume transitivity. The IPO objective maximizes expected preference probability directly, without decomposing it into pointwise rewards. A policy that beats B 60% of the time, B beats C 60% of the time, and C beats A 60% of the time can be perfectly accommodated — the objective simply tries to pick the action with the highest expected win rate against the distribution μ, which is well-defined even under intransitivity.

This is a fundamental distinction from Ψ = logit, which enforces transitivity through the additivity of the reward function. Under the Bradley-Terry model, if r(A) > r(B) and r(B) > r(C), then r(A) > r(C) by transitivity of real numbers. A cyclic preference pattern cannot be represented — the BT model would be forced to assign rewards that misrepresent at least one of the pairwise comparisons. This is not a pathological edge case: intransitivity in human preferences is well-documented in the judgment and decision-making literature, and can arise naturally when preferences involve multiple incommensurable attributes.

The paper's treatment of this is understated — it merely mentions D₂ as one of "two outcomes (up to permutations of the actions)" and focuses its analysis on D₁ — but the presence of D₂ in the experimental design implicitly stakes a claim: IPO does not need the transitivity assumption that RLHF and DPO rely on. This is significant because it means IPO applies to a strictly larger class of preference structures. In practice, whether real human preference data exhibits meaningful intransitivity is an empirical question that would depend on the domain and the annotation protocol, but the theoretical point is that IPO does not rule it out a priori.

Why this is incremental rather than fundamental (in the context of this paper's contributions): the paper does not develop this point theoretically or empirically — there is no Proposition about IPO under intransitive preferences, and no experiment measuring performance on cyclic data. It is a latent insight that the framework enables, not a fully realized contribution. It is included here because it illustrates the kind of thinking that the ΨPO framework opens up: by abstracting away from the Bradley-Terry model, one can ask questions about what preference structures the objective can and cannot represent, and choose Ψ accordingly. The logit function imposes one set of representational constraints (transitivity, additivity); the identity function imposes a different set (boundedness, no enforced transitivity). This is a design choice that the field was not previously aware it was making.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use synthetic bandit preference datasets constructed by the authors, not a standard benchmark. Three configurations are studied: D₁ = {(yₐ, y_b), (y_b, y_c), (yₐ, y_c)} with 3 observed preferences representing a total ordering; D₂ = {(yₐ, y_b), (y_b, y_c), (y_c, yₐ)} representing a preference cycle; and D₃ = {(yₐ, y_b), (y_b, yₐ)} leaving a third action y_c completely unobserved. The action space Y = {yₐ, y_b, y_c} has 3 actions. There is no context x — this is the pure bandit setting.

• Base model(s). There is no pretrained language model. Policies are parameterized as π_θ(y) = softmax(θ) using a vector θ ∈ ℝ³, initialized to produce the uniform reference policy π_ref (all actions equally likely). The parametrization is deliberately minimal to isolate the behavior of the loss functions themselves, uncontaminated by model capacity, representation learning, or optimization artifacts from large-scale training.

• Metrics. The primary metric is the learned action probabilities π_θ(y_i) for each action, reported as learning curves over 18,000 optimization steps. There is no held-out accuracy metric or downstream task evaluation — the analysis is purely behavioral, examining what distribution the optimization converges to under different τ values and preference dataset configurations. Convergence to degenerate policies (π(y) = 1 or 0) is the key failure mode being measured.

• Baselines. The sole comparison method is Direct Preference Optimisation (DPO) (Rafailov et al., 2023), optimized via the empirical loss in Equation (4). No RLHF baseline (e.g., PPO on a trained reward model) is implemented in the bandit experiments. The paper argued theoretically in Section 4.2 that RLHF's reward model underfitting provides implicit protection against degenerate policies, but this claim is not empirically tested in the experimental section.

• Generation budget / compute accounting. There is no generation budget in the usual sense — no sampling from the policy during training. The "compute" is measured in optimization steps (18,000 steps of Adam). The mini-batch size is 9, constructed by uniform sampling with replacement from the preference dataset D. Since the datasets contain only 2–3 preference pairs, the effective data seen during training is determined by the number of steps and the resampling procedure.

• Cross-validation / statistical protocol. Each configuration (combination of dataset, τ value, and algorithm) is repeated 10 times with different random seeds, and results are reported as mean and 95% confidence intervals (Figures 1 and 2). The policies are optimized using Adam (Kingma and Ba, 2014) with learning rate 0.01. All experiments run on "a modern cloud virtual machine with 4 cores and 32GB of RAM." The flax framework (Bradbury et al., 2018; Heek et al., 2023) implements both policies and losses, with the Adam optimizer from optax (Babuschkin et al., 2020).

---

Main Quantitative Results

Asymptotic Analysis: Two-Action Deterministic Preference (Section 5.3.1)

The paper first analyzes the closed-form optimal policies analytically, before any optimization experiments. For the two-action deterministic preference case p*(y₁ ≻ y₂) = 1 with uniform π_ref and μ:

• DPO: The optimal policy converges to π*(y₁) = 1, π*(y₂) = 0 "regardless of the value of τ." The paper states this follows directly from plugging the Bradley-Terry implication (r(y₁) − r(y₂) → +∞) into Equation (7).

• IPO: The paper computes p*(y₁ ≻ μ) = 3/4 and p*(y₂ ≻ μ) = 1/4 (details in Section 5.3.1), yielding π*(y₁) = σ(0.5τ⁻¹) and π*(y₂) = σ(−0.5τ⁻¹), where σ is the sigmoid. As τ → +∞, both probabilities converge to 0.5 (the uniform reference policy). As τ → 0⁺, they converge to (1, 0). The regularisation parameter τ "can now actually be used to control how close to π_ref we are."

This is the paper's central analytical demonstration: DPO's optimal policy is degenerate (independent of τ) while IPO's optimal policy is τ-controlled. No figure is needed — this follows from the closed-form expressions.

Sampled Preferences: Total Ordering D₁ (Section 5.4, Figure 1)

The first empirical experiment uses D₁ = {(yₐ, y_b), (y_b, y_c), (yₐ, y_c)}, a dataset of 3 pairwise preferences forming a total ordering (yₐ beats y_b, y_b beats y_c, yₐ beats y_c). This is the "one action wins against all others" scenario.

Headline result: DPO always converges to a deterministic policy assigning probability 1 to yₐ and 0 to y_b and y_c, for all tested values of τ. IPO converges to a distribution that depends on τ, staying closer to the uniform reference policy when τ is large.

Figure 1 specifics: The figure shows learning curves (action probabilities vs. optimization steps) for both DPO and IPO across multiple τ values (though exact τ values are not specified in the text — the figure itself must be consulted). The paper states:

"We observe that DPO always converges to the deterministic policy for all values of τ. In other word DPO completely ignores the reference policy, no matter how strong is the regularisation term, and converges to the action which is preferred in the dataset. On the other hand, IPO prevent the policy from becoming greedy when the regularisation is strong."

The key behavioral difference: DPO's probability for the dominant action yₐ rises monotonically toward 1.0 for all τ curves, with no visible dependence on τ. IPO's probability for yₐ stabilizes at different levels depending on τ (presumably higher for smaller τ, lower for larger τ, converging toward 1/3 for very large τ).

Interpretation: This experiment validates the theoretical prediction from Section 4.2: when empirical preferences contain a strictly dominant action (one that wins all its observed comparisons), Ψ = logit (DPO) converts this into unbounded incentive that overwhelms the KL penalty. Ψ = Identity (IPO) treats "winning 2 out of 2 comparisons" as a finite signal bounded within [0,1], and the squared loss regresses the log-ratio gap to τ⁻¹/2 rather than to infinity. The result that DPO ignores τ entirely — not just that it converges to a deterministic policy for small τ, but that it does so for all τ — is the starkest demonstration of the regularisation failure.

Sampled Preferences: Unobserved Action D₃ (Section 5.4, Figure 2)

The second empirical experiment uses D₃ = {(yₐ, y_b), (y_b, yₐ)}, a dataset where yₐ and y_b each win one comparison (symmetric), and the third action y_c is completely unobserved. This tests the "one action never wins" scenario: y_c has zero victories in the dataset, not because it is genuinely dispreferred, but because it was never sampled.

Headline result: DPO sets π(y_c) → 0 regardless of τ, excluding the unobserved action entirely. IPO gradually decreases π(y_c) as τ decreases but keeps it non-zero, with the probability depending on τ.

Figure 2 specifics: The paper states:

"We observe again here that DPO ignores the prior π_ref completely, no matter how strong we regularize the objective, whereas IPO gradually decreases the probability of unobserved action with τ."

The learning curves show π(y_c) dropping to zero under DPO for all τ values. Under IPO, π(y_c) presumably decreases from the uniform 1/3 starting point to lower values as optimization proceeds, with larger τ values maintaining higher π(y_c) and smaller τ values producing lower π(y_c).

Interpretation: This is a critical practical scenario that the paper argues is "much more common in real-world data":

"whenever the action space is large but the dataset small, some actions will necessarily be sampled rarely or only once, making it likely to never observe a victory. Especially because we do not have data on their performance π should stick close to π_ref for safety, but DPO's objective does not promote this."

DPO's behavior — zeroing out any action that has no observed wins — is particularly dangerous in LLM fine-tuning because it means the policy will completely suppress any generation that happened not to appear as a winner in the preference dataset, even if the true human preference for that generation is moderate or unknown. The reference policy, which encodes the base model's distribution acquired through pretraining on vast data, is discarded as irrelevant precisely when it's most needed (for actions with sparse preference coverage).

Magnitude note: The paper does not report exact numerical values for the converged probabilities at specific τ values. The results are presented as qualitative patterns in learning curves. This is consistent with the paper's framing of these as "illustrative examples" (Section 5.3) meant to demonstrate the qualitative difference between the methods, not to benchmark performance.

---

Ablation Studies and Robustness Checks

Multiple τ values tested: Both DPO and IPO are run with varying τ values (the exact set is not specified in the text but is visible in Figures 1 and 2 as multiple curves per method). This functions as an implicit ablation: if DPO were merely "too aggressive" at a particular τ, increasing τ should bring it closer to π_ref. The fact that DPO's convergence is τ-independent across all tested values is the key negative result — it demonstrates that the problem is structural (Ψ = logit is unbounded) rather than a matter of hyperparameter choice.

Dataset structure as a robustness test (D₁ vs. D₂ vs. D₃): The paper tests three qualitatively different preference structures (total order, cycle, and partial observation). D₂ (the cycle) is mentioned in the setup but not analyzed in the results — the paper focuses on D₁ and D₃. This is a missed opportunity to demonstrate IPO's behavior under intransitive preferences, which (as discussed in Innovation 5 of the Key Insights section) is a theoretical advantage of Ψ = Identity over Ψ = logit that the paper does not empirically validate.

Deterministic preferences as a stress test: Both D₁ and D₃ are constructed so that empirical preference frequencies are exactly 0 or 1 for all observed pairs (in D₁, no action ever loses to a lower-ranked action; in D₃, yₐ and y_b each have exactly one win and one loss, but y_c has 0 wins and 0 losses — it is not in the dataset at all). This is the hardest case for any method relying on Ψ = logit, and the experiments confirm the theoretical prediction. What is not tested is the intermediate regime where empirical frequencies are in (0,1) for all pairs — e.g., (0.7, 0.3) — where the Bradley-Terry model is well-specified and DPO should perform as intended. The paper does not assess where the transition from "DPO works fine" to "DPO collapses" occurs as preferences become increasingly deterministic.

No experiment on the effect of dataset size: All experiments use tiny datasets (2–3 preference pairs for a 3-action space). The paper's theoretical argument (Section 4.2) is that overfitting risk increases as the dataset becomes sparser relative to the action space, but no experiment varies the number of observed preferences per action to test this prediction. An experiment showing that DPO's τ-independence persists even with moderate amounts of data (e.g., 5–10 comparisons per action pair) would strengthen the claim that this is a practical concern for LLMs; conversely, showing that DPO recovers τ-sensitivity with enough data would narrow the scope of the concern. Neither experiment is performed.

No comparison with an explicit RLHF baseline: The paper argues in Section 4.2 that RLHF's two-stage pipeline provides implicit protection through reward model underfitting. This is a testable claim: one could implement a minimal RLHF pipeline (train a reward model on the bandit data, then run KL-regularised policy optimization) and compare its τ-sensitivity to DPO's. The bandit experiments do not include this comparison, leaving the RLHF advantage as a theoretical argument rather than an empirically validated fact.

---

Critical Assessment

The experiments in Section 5.4 are minimal, qualitative demonstrations of the paper's central theoretical predictions, not comprehensive benchmarks. They should be assessed in this context — the paper's contribution is primarily theoretical, and the experiments serve to validate that the predicted behaviors actually manifest when the loss functions are optimized with gradient descent.

Claim from the executive summary: "IPO maintains bounded regularisation even with {0,1}-valued empirical preferences." This is the paper's primary empirical claim, and Figures 1 and 2 directly support it: for both D₁ (total ordering with a dominant action) and D₃ (unobserved action with zero wins), IPO's converged policy depends on τ (visible as separation between curves for different τ values), while DPO's converged policy does not (all τ curves collapse to the same degenerate policy). The evidence is clear and consistent across two qualitatively different dataset configurations.

However, the evidence establishes a narrower claim than "IPO solves the problem." It shows that IPO is less degenerate than DPO — it converges to distributions that are not fully deterministic and that respond to τ. Whether those distributions are correct or useful in any downstream sense is not evaluated. The paper does not measure whether IPO's converged probabilities match the true preference structure, only that they are not collapsed to 0/1.

Claim: "DPO ignores the reference policy regardless of τ when preferences are deterministic." Figures 1 and 2 provide strong evidence for this in the specific setting tested (3 actions, 2–3 preference pairs, Adam optimization, 18,000 steps). The learning curves show DPO converging to deterministic policies for all τ values. The evidence is limited in several ways:

• τ range not specified. The exact τ values tested are not listed in the text. If the tested τ values were all relatively small, the experiment might miss a regime where even DPO would show τ-sensitivity. The paper argues theoretically that the τ-independence holds for any finite τ (because the logit function diverges to infinity, overpowering any finite τ⁻¹), so the exact values should not matter — but this argument relies on the optimization actually reaching the theoretical optimum, which is not guaranteed in finite steps with a specific optimizer.

• Single optimizer, single learning rate. The experiments use Adam with learning rate 0.01 for all configurations. DPO's collapse to deterministic policies could be influenced by optimization dynamics — different learning rates or optimizers might converge to different points in the loss landscape. The paper's theoretical analysis (Section 4.2) suggests the collapse is a property of the loss function's global optimum, not an optimization artifact, but the experiments do not rule out the possibility that early stopping or different optimization hyperparameters could mitigate DPO's behavior.

• No experiment on non-deterministic preferences. The theoretical analysis predicts that the collapse is specific to {0,1}-valued empirical preferences. An experiment with, say, 10 preference samples per pair producing empirical frequencies like 0.6–0.4 would test whether DPO's τ-sensitivity recovers when preferences are not at the boundary. The absence of this experiment means we do not know whether the problem is fundamentally about deterministic preferences or about any skewed empirical frequencies.

Claim: "IPO is better suited to learn from sampled preferences than DPO." This claim (from the Conclusion) is supported only in the narrow sense that IPO avoids degenerate policies. "Better suited" implies a normative judgment that the paper's experiments cannot fully justify: there is no evaluation of which method recovers a policy closer to the true underlying preferences (in cases where the true preferences are known and not deterministic), no comparison on standard preference-based RL benchmarks, and no scaling to language model fine-tuning tasks. The experiments demonstrate that DPO has a specific failure mode that IPO does not share — this is a necessary condition for being "better suited" but not a sufficient one.

What is missing:

1. A ground-truth preference evaluation. The experiments observe what policies the methods converge to, but do not measure whether those policies are good — e.g., by computing the expected true preference win rate of the learned policy against the behavior policy. This would distinguish between "IPO avoids degeneracy" and "IPO learns a meaningfully better policy." It is possible that DPO's deterministic policy, while degenerate, actually achieves higher expected preference on the true distribution (if the true preferences are indeed close to deterministic), in which case the "overfitting" might be benign or even beneficial.

2. Scaling to larger action spaces and realistic dataset sizes. The bandit experiments use 3 actions and 2–3 preference pairs. LLM action spaces are enormous (vocabulary-level generation or sequence-level generation). The paper argues that the problem becomes more severe with larger action spaces (because some actions will have zero observed wins by chance), but does not test this scaling claim. An experiment with 100+ actions and varying dataset sizes would test whether DPO's collapse is an edge case or the expected behavior.

3. The D₂ (cyclic) preference experiment. The paper mentions D₂ as a possible dataset outcome but does not report results for it. Testing IPO on cyclic preferences would validate one of its theoretical advantages over the Bradley-Terry model (no enforced transitivity). The absence of this experiment is notable given its theoretical relevance.

4. An explicit RLHF baseline in the bandit setting. The paper's argument about RLHF's accidental regularisation is compelling theoretically but untested empirically. A bandit experiment with a small reward model (e.g., a linear function of one-hot action features) trained via the BT loss and then used for KL-regularised policy optimization would provide evidence for or against the claim that RLHF is more robust than DPO in this setting.

5. Sensitivity to the parametrization. The policies are parameterized as softmax(θ). This is the natural choice, but it interacts with the DPO loss in specific ways. An alternative parameterization (e.g., directly parameterizing the logits with a constraint that they sum to zero) might produce different optimization dynamics. The paper does not discuss whether the collapse behavior is parametrization-dependent.

Summary of evidence quality: The experiments succeed at their stated purpose — "illustrative examples" (Section 5.3 title) that "prove that IPO is better suited to learn from sampled preferences than DPO" (Conclusion) in the specific, narrow sense of avoiding degenerate policies on bandit problems with deterministic empirical preferences. They do not establish that IPO produces better policies in any task-oriented sense, that the problem manifests at scale in language model fine-tuning, or that IPO's advantage persists when preferences are not at the boundary. The experimental section is best understood as a proof of concept that the theoretical analysis has empirical teeth, not as a comprehensive empirical evaluation of IPO as a practical algorithm. This is consistent with the paper's self-positioning as a theoretical contribution, but readers expecting experimental validation at the scale where these methods are actually deployed (LLM fine-tuning) will find the evidence preliminary.

6. Limitations and Trade-offs

The Bandit Experiments Do Not Validate the Claims at Realistic Scale

The assumption or constraint. The paper's empirical validation consists entirely of synthetic bandit experiments with 3 actions and 2–3 preference pairs, optimized via Adam for 18,000 steps (Section 5.4). The entire argument for IPO's practical superiority rests on these "illustrative examples" (Section 5.3 title). The paper explicitly frames itself as a theoretical contribution and acknowledges the need for scaling in its conclusion:

"Future works should scale those experiments to more complex settings such as training language models on human preferences data."

The consequence. The central claim — that IPO avoids the regularisation failure that DPO suffers from — is demonstrated only in a setting that abstracts away nearly every aspect of real LLM preference fine-tuning: there is no context $x$, no high-dimensional continuous action space (vocabulary-size generation), no neural network policy beyond a 3-parameter softmax, no stochastic mini-batching beyond resampling from a 3-pair dataset, and no interaction between the preference data and the policy's own generation distribution (the behavior policy $\mu$ is fixed and uniform rather than being the model's own outputs). The leap from "a 3-way softmax avoids degeneracy on 2–3 preference pairs" to "LLM fine-tuning with IPO will be well-behaved" is entirely unvalidated by the evidence in the paper. A practitioner deciding whether to replace DPO with IPO in a production LLM alignment pipeline has no empirical basis in this paper for the decision.

What evidence exists in the paper. The experiments (Figures 1 and 2) demonstrate the predicted qualitative difference between DPO (collapses to deterministic policies for all $\tau$) and IPO ($\tau$-sensitive convergence) under the specific conditions tested. These are internally valid and support the theoretical analysis. But they are not even argued to be representative of LLM-scale behavior — the paper makes no attempt to characterize how the loss functions interact with neural network parametrizations, large action spaces, or realistic preference datasets where empirical frequencies are rarely exactly 0 or 1.

Mitigation status. The paper acknowledges the limitation explicitly in the conclusion but does nothing to address it. The framing of the work as theoretical is legitimate, but the paper's title ("A General Theoretical Paradigm to Understand Learning from Human Preferences") and abstract ("demonstrate its empirical superiority to DPO on some illustrative examples") set an expectation that the theoretical insights carry practical force. The gap between "illustrative examples" and "demonstrate superiority" is unusually wide — the experiments prove that IPO is less degenerate than DPO in a toy setting, which is a necessary condition for being better, not a sufficient one. The explicit call for future work on scaling is appropriate but does not reduce the uncertainty for a practitioner reading the paper today.

---

No Measurement of Whether IPO Actually Learns Better Policies

The assumption or constraint. The experiments in Section 5.4 report learning curves of action probabilities — what policy the optimization converges to — but never evaluate the quality of those policies against the true underlying preference distribution. There is no metric analogous to accuracy, expected win rate, or regret. The paper implicitly assumes that "avoids collapse to a degenerate policy" is sufficient evidence of improvement.

The consequence. A reader cannot determine whether IPO's non-degenerate policies are actually better than DPO's degenerate ones in any task-relevant sense. Consider a scenario where the true human preferences are genuinely near-deterministic — say, $p^*(y_a \succ y_b) = 0.98$, $p^*(y_a \succ y_c) = 0.97$, $p^*(y_b \succ y_c) = 0.95$. In this case, DPO's convergence to $\pi(y_a) = 1$ may be closer to optimal than IPO's $\tau$-regulated distribution that assigns non-zero probability to clearly inferior actions. The paper's theoretical analysis (Section 4.2) shows that DPO ignores $\pi_{ref}$ when preferences are deterministic, but does not establish that ignoring $\pi_{ref}$ is bad when $\pi_{ref}$ is genuinely wrong about relative action quality. The entire critique of DPO's overfitting assumes that the deterministic empirical preferences are noise rather than signal, but the experiments provide no way to distinguish these cases because they use synthetic data where the true $p^*$ is implicit in the dataset construction (it is defined by the dataset itself rather than a separate ground truth).

What evidence exists in the paper. None. Section 5.3.1 does provide a closed-form comparison: under deterministic preferences $p^*(y_1 \succ y_2) = 1$, DPO converges to $(1, 0)$ while IPO converges to $(\sigma(0.5\tau^{-1}), \sigma(-0.5\tau^{-1}))$. The paper treats the IPO result as clearly superior because it respects the KL penalty. But whether a policy of $(0.7, 0.3)$ (for moderate $\tau$) is objectively better than $(1, 0)$ when $y_1$ genuinely always beats $y_2$ depends on what the policy is used for — if the goal is to always pick the best action (greedy decoding), they are equivalent; if the goal is to sample diverse high-quality actions, IPO may be preferable; if the goal is to maximize expected preference, the deterministic policy may be strictly better. The paper provides no framework for making this normative judgment.

Mitigation status. Not addressed. The paper's theoretical framework (the $\Psi$PO objective in Equation 6) defines optimality relative to a chosen $\Psi$, so in principle one could evaluate which $\Psi$ recovers a policy closer to the true preference-maximizing policy under some external metric. But the paper never does this — it argues that $\Psi = \text{Identity}$ is better because it avoids degeneracy, without measuring whether the resulting policies actually achieve higher expected preference or any other downstream metric. This is a gap between the paper's diagnostic ("DPO has a failure mode") and its prescriptive claim ("IPO is the solution"). The diagnostic is well-supported; the prescription is not evaluated.

---

The Cost of Difficulty Estimation Is Not Accounted For (This Limitation Does Not Apply — No Difficulty Estimation)

Note: the above heading is included to preserve the structure, but this specific limitation from the reference example does not apply to the current paper. The paper does not use difficulty estimation. Proceeding to the next actual limitation.

---

The Assumption $\text{Supp}(\mu) = \text{Supp}(\pi_{ref})$ Is Strong and Potentially Violated in Practice

The assumption or constraint. Theorem 2 (the uniqueness guarantee for the IPO minimizer) requires that the behavior policy $\mu$ and the reference policy $\pi_{ref}$ have identical support — that is, every action that $\pi_{ref}$ assigns non-zero probability to must also appear in the preference dataset with non-zero probability, and vice versa. Appendix A.2 provides a counterexample demonstrating that when this condition fails — specifically, when there exists an action in the support of $\pi_{ref}$ that is never sampled by $\mu$ — the IPO loss has infinitely many minimizers, not a unique one.

"Assuming that the supports of $\pi_{ref}$ and $\mu$ coincide enables us to recover uniqueness of the solution, as proven in Theorem 2."

The consequence. In LLM fine-tuning, $\pi_{ref}$ is the base pretrained model, which assigns non-zero probability to an enormous number of token sequences (effectively the entire vocabulary). The behavior policy $\mu$ is the policy that generated the preference data — typically a model fine-tuned for a specific task, or the base model itself with some sampling strategy, operating on a finite set of prompts. For any realistic preference dataset size, the vast majority of possible generations will have zero empirical probability under $\mu$ (they were never sampled). This means the condition $\text{Supp}(\mu) = \text{Supp}(\pi_{ref})$ is almost certainly violated in any practical LLM application of IPO. The Theorem 2 uniqueness guarantee does not apply, and the loss may have many equally optimal policies that differ in their treatment of actions not covered by the preference data.

The practical manifestation: for any generation that the reference model could produce but that never appeared in the preference data (which describes essentially all possible generations for most prompts), the IPO loss provides no constraint on the learned policy's probability relative to other actions. The policy is free to assign arbitrary probabilities to these unobserved actions, constrained only by the softmax normalization. This is precisely the regime where the KL penalty to $\pi_{ref}$ is supposed to provide a default — the policy should fall back to the reference model's distribution for actions without preference signal. But Theorem 2 no longer guarantees this behavior when the support condition is violated; the optimizer may find a solution that satisfies all the constraints on observed actions while arbitrarily deviating from $\pi_{ref}$ on unobserved ones.

What evidence exists in the paper. The counterexample in Appendix A.2 demonstrates the non-uniqueness explicitly for a 3-action case where $\mu$ only covers 2 of the 3 actions. The paper does not test whether this non-uniqueness leads to degenerate behavior in practice — the bandit experiments in Section 5.4 all use uniform $\mu$ over all actions, satisfying the support condition by construction. The D₁ and D₃ experiments therefore operate in a regime where Theorem 2 applies, and the results (IPO converges to well-behaved policies) cannot speak to what happens when the condition is violated, as it would be in any realistic LLM scenario.

Mitigation status. The paper acknowledges the condition is required for uniqueness but does not propose any mitigation for the case where it fails — no regularisation term beyond the KL penalty, no technique for imputing preferences for unobserved actions, no analysis of whether gradient-based optimization with finite steps tends to find a "reasonable" minimizer (e.g., one close to $\pi_{ref}$ by virtue of initialization) even when the minimizer is not unique. The assumption is stated as a technical condition for the theorem, and its violation in practice is not discussed. This is a significant gap because a practitioner cannot determine from the paper whether IPO would still behave reasonably when the condition does not hold, which is the expected case.

---

The Analysis Is Restricted to the Bandit Setting and Does Not Address Sequential Generation

The assumption or constraint. The entire paper operates in the contextual bandit setting (Section 3): a context $x$ is drawn, a single action $y$ is chosen, a preference is expressed between pairs of actions. The derivation of the $\Psi$PO objective, the analytic solution, the root-finding reformulation, and the IPO sampled loss all assume that the policy produces a single action per context, and preferences compare individual actions.

The consequence. LLM alignment involves sequential generation: the action is not a single token but a sequence of tokens (a "generation" or "continuation"), and human preferences are expressed over complete sequences, not individual tokens. The bandit formalization collapses this sequential process into a single action — essentially treating each possible complete generation as one action in an enormous discrete space. This abstraction is standard in the RLHF literature (Rafailov et al., 2023 uses it as well), but it hides a crucial complication: the policy must assign probabilities to sequences autoregressively, token by token, and the IPO loss as derived operates on the probability of the complete sequence $\pi(y|x)$, not on the per-token probabilities that are actually optimized during training.

In practice, DPO and IPO are applied to autoregressive language models by treating the likelihood ratio $\log(\pi(y|x)/\pi_{ref}(y|x))$ as the sum of per-token log-probability ratios along the sequence. This is a natural extension, but the theoretical analysis in this paper does not address whether the properties derived for the bandit setting (uniqueness of the minimizer, boundedness of regularisation, the $\tau^{-1}/2$ target) carry over to the autoregressive case. The loss in Algorithm 1 includes context $x$ and defines $h_\pi(y_w, y_l, x)$, treating $y_w$ and $y_l$ as complete sequences, but the optimization modifies per-token probabilities, which interact in potentially complex ways (changing the probability of an early token affects the probability of subsequent tokens through the autoregressive factorization).

What evidence exists in the paper. None. The paper never discusses the transition from bandit actions to autoregressive sequence generation. All experiments use the bandit setting with a 3-action softmax. The DPO paper (Rafailov et al., 2023) demonstrated empirical success on language tasks with the bandit-derived loss applied auto-regressively, which provides some precedent, but this paper provides no evidence — theoretical or empirical — that IPO's theoretical properties survive the transition to sequential generation.

Mitigation status. Not addressed. The paper inherits the bandit formalization from the RLHF/DPO literature and applies it without discussion of the autoregressive complication. This is a defensible abstraction for a theoretical paper — the bandit setting captures the core preference optimization problem without the confound of sequential credit assignment — but it means that none of the paper's theoretical guarantees apply to the setting where IPO would actually be deployed. A practitioner implementing IPO for LLM fine-tuning is operating in an extrapolated regime where the theoretical analysis is suggestive but not probative.

---

No Comparison with RLHF Under Any Circumstance, Despite Theoretical Claims About RLHF's Advantages

The assumption or constraint. Section 4.2 contains a significant theoretical claim about RLHF: that its two-stage pipeline provides implicit regularisation through reward model underfitting, making it more robust to deterministic preferences than DPO:

"The optimal rewards in the presence of {0,1} preference probabilities are infinite, but these values are avoided, and indeed regularisation of the reward function has been observed to be an important aspect of RLHF training in practice."

This argument is central to the paper's narrative about why DPO is vulnerable in a way that the older, more complex RLHF pipeline is not — and by extension, why IPO (which also avoids the vulnerability, but through a different mechanism) is a principled alternative that recovers the robustness of RLHF without the complexity.

The consequence. The claim about RLHF's robustness is entirely theoretical and untested in the paper. The bandit experiments in Section 5.4 compare IPO only against DPO. There is no implementation of RLHF — no reward model trained via the Bradley-Terry logistic regression loss, no KL-regularised policy optimization (e.g., PPO or analytic optimization from the learned reward). The paper cannot distinguish between two possibilities: (a) RLHF does indeed underfit the reward model and produce policies that remain regularised toward $\pi_{ref}$ even with deterministic preferences, or (b) RLHF suffers from the same regularisation failure as DPO when the reward model is sufficiently expressive, or when the policy optimization stage is run to convergence. The theoretical argument in Section 4.2 is plausible but incomplete — it rests on the empirical claim that reward model training "avoids" infinite values, but does not characterize how much underfitting occurs, whether it is sufficient to prevent degeneracy, or whether alternative reward model architectures or training procedures could eliminate this accidental protection.

What evidence exists in the paper. The paper cites Christiano et al. (2017) for the observation that "regularisation of the reward function has been observed to be an important aspect of RLHF training in practice," but provides no empirical evidence of its own. Appendix B's Proposition 4 proves that DPO and RLHF share the same optimal policy (under the BT model, for any preference distribution $p^*$), which actually suggests they should behave identically at the optimum — the claim that they differ in practice is precisely a claim about optimization dynamics and finite model capacity, which the proposition does not address.

Mitigation status. The paper makes no attempt to validate the RLHF robustness claim empirically. The future work section does not mention this as an open question. For a paper whose central narrative is "DPO loses the implicit regularisation that RLHF had, and IPO restores it in a principled way," the absence of any empirical comparison with RLHF is a significant evidentiary gap. Without it, the reader cannot assess whether IPO is (a) better than DPO and comparable to RLHF, (b) better than both, or (c) different from both but not clearly superior to either in terms of final policy quality. All three are consistent with the evidence presented.

---

The $\Psi$PO Framework Provides No Guidance for Choosing $\Psi$ Beyond the Boundedness Criterion

The assumption or constraint. The paper's central conceptual contribution is the $\Psi$PO framework, which parameterizes the preference optimization objective by an arbitrary non-decreasing function $\Psi : [0,1] \to \mathbb{R}$. The paper analyzes two points in this space in detail: $\Psi = \text{logit}$ (which recovers RLHF/DPO) and $\Psi = \text{Identity}$ (IPO). It identifies one property — boundedness — that distinguishes them and explains their different behavior under deterministic preferences. But it provides no further guidance on how to select among the infinitely many other bounded, non-decreasing functions.

The consequence. The $\Psi$PO framework is descriptive (it shows that RLHF/DPO and IPO are instances of the same family) but not prescriptive (it does not tell you that $\Psi = \text{Identity}$ is the best choice, or how to navigate the tradeoffs among bounded $\Psi$ functions). Consider alternative choices: $\Psi(q) = q^2$ (quadratic), $\Psi(q) = \sqrt{q}$ (square root), $\Psi(q) = 2q - 1$ (centered identity), or any sigmoid function with finite asymptotes. All of these are bounded on $[0,1]$ and would therefore avoid the particular failure mode analyzed in the paper. But they would produce different optimal policies, different target values in the root-finding reformulation (the $\tau^{-1}/2$ constant is specific to the identity), and potentially different empirical behavior. The paper provides no analysis — theoretical or empirical — of how these choices differ or why identity is preferable.

This matters because the choice of $\Psi$ encodes assumptions about how preference probabilities should be weighted. The identity function treats a shift from $0.5 \to 0.6$ (indifference to slight preference) identically to a shift from $0.9 \to 1.0$ (strong preference to certainty). The quadratic function would weight extreme preferences more heavily. The square root would weight them less heavily. These are substantively different optimization targets, and a practitioner choosing among them needs guidance that the paper does not provide.

What evidence exists in the paper. The paper demonstrates that $\Psi = \text{Identity}$ works (in the limited sense of avoiding degeneracy on the bandit experiments) and that $\Psi = \text{logit}$ fails (collapses to deterministic policies). This establishes the existence of at least one $\Psi$ that is better than the standard choice along one dimension (degeneracy), but does not establish that identity is optimal, or even that the degeneracy problem is the only relevant criterion for selecting $\Psi$. The paper acknowledges the generality of the framework but delegates the question of optimal $\Psi$ selection entirely to future work.

Mitigation status. The paper positions IPO as "a particularly natural form of objective" (Section 5) and justifies the choice of identity by its boundedness, simplicity, and the interpretability of the resulting loss (regressing the log-ratio gap to $\tau^{-1}/2$). This is reasonable for an initial exploration but does not constitute a systematic analysis of the $\Psi$ design space. The framework's generality is both its strength (it opens up new design possibilities) and its weakness (it provides no guidance for navigating those possibilities). A practitioner who is convinced that $\Psi = \text{logit}$ is problematic but unsure whether $\Psi = \text{Identity}$ is the right replacement will find no principled basis in the paper for choosing among the alternatives.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing rather than a paradigm shift: it does not displace the RLHF/DPO paradigm but reveals a previously invisible design dimension — the choice of the non-linearity $\Psi$ — that controls a fundamental vulnerability in the entire family of preference-optimization objectives. The magnitude of the reframing is significant because it operates at the objective-design level, not the algorithm-design level. Prior work asked "how should we optimize the Bradley-Terry–derived objective?" This paper asks "should the Bradley-Terry mapping be part of the objective at all, and what happens when we replace it?" This is the difference between tuning hyperparameters and questioning the optimization target.

The paper's central diagnostic — that the unboundedness of $\Psi$ at 0 and 1 causes KL-regularisation collapse — provides a clear, falsifiable criterion for evaluating any future preference-optimization objective. Before this work, a researcher proposing a new loss function for learning from preferences had no principled vocabulary for analyzing whether the KL penalty would remain effective under extreme empirical preferences. After this work, the boundedness of the implicit $\Psi$ becomes a first-order check: if $\Psi$ diverges at the boundaries, the objective is vulnerable to the same degeneracy that DPO exhibits, regardless of $\tau$. This diagnostic is general enough to survive beyond the specific IPO proposal — it applies to any preference-optimization method expressible in the $\Psi$PO framework, and plausibly to methods outside it that share the structure of mapping preference probabilities to optimization incentives.

The paper also reconciles a latent contradiction in the literature that was not widely recognized as a contradiction. On one hand, DPO was presented as a simpler, more direct alternative to RLHF that achieves the same theoretical optimum (Rafailov et al., 2023). On the other hand, practitioners have observed that DPO can produce policies that diverge too far from the reference model — a phenomenon typically attributed to insufficient hyperparameter tuning or data quality issues. The paper provides a unified explanation: both RLHF and DPO share the same structural vulnerability ($\Psi = \text{logit}$ is unbounded), but RLHF is accidentally protected by reward-model underfitting — a bug in the reward modeling stage that acts as a beneficial regulariser — while DPO, by eliminating that stage, exposes the vulnerability directly. This resolution is intellectually satisfying because it explains why a simpler pipeline can be less robust: the complexity was doing unseen work.

The reframing redirects research attention in two ways:

• Toward objective design rather than optimizer design. The paper demonstrates that the failure mode it identifies cannot be fixed by tuning $\tau$, changing the optimizer, or adjusting the learning rate — it is structural, embedded in the definition of what the loss is trying to achieve. This suggests that the more productive research direction is not "how can we make DPO more stable?" but "what $\Psi$ should we use?" — a fundamentally different question that the field was not previously asking.

• Away from the assumption that eliminating intermediate models is always beneficial. The observation that RLHF's reward model provides implicit regularisation by underfitting is a concrete counterexample to the appealing intuition that end-to-end pipelines are always preferable. It suggests that some intermediate representations act as information bottlenecks that improve robustness, and that removing them can expose the system to failure modes that the bottleneck was suppressing. This has implications beyond preference optimization — it is relevant to any setting where a pipeline is being simplified by removing a learned component.

The paper also shifts the burden of proof for future preference-optimization methods. Before this work, a method could be justified by showing that it converged to the same optimum as RLHF under the BT assumption. After this work, that justification is insufficient — one must also demonstrate that the method's implicit $\Psi$ is bounded, or that it includes an alternative mechanism for preventing regularisation collapse. IPO sets the standard: a method should remain $\tau$-sensitive even when empirical preferences are deterministic.

What becomes less attractive: Research on increasingly sophisticated search or optimization strategies for the DPO/RLHF objective (e.g., better PPO variants, better reward model architectures) loses relative priority if the objective itself is structurally flawed. The paper suggests that no amount of optimization improvement can fix the unboundedness of $\Psi = \text{logit}$ — the problem is in the target, not the path to reaching it.

Follow-Up Research This Work Enables

IPO at scale: fine-tuning a language model with IPO on a standard human-preference dataset. The paper's experiments are deliberately minimal — 3-action bandits with 2–3 preference pairs. The single most important follow-up is a direct, head-to-head comparison of IPO and DPO on a realistic LLM fine-tuning task, using standard human-preference datasets (e.g., Anthropic's helpfulness/harmlessness data, or the Summarize from Feedback dataset). This experiment would measure: (a) whether the degeneracy that DPO exhibits in the bandit setting manifests at scale — specifically, whether DPO-trained policies show reduced $\tau$-sensitivity (measured by KL divergence from the reference model) compared to IPO-trained policies at matched $\tau$; (b) whether IPO-trained policies achieve comparable or better downstream performance (win rate against a baseline, human evaluation scores) while maintaining closer KL proximity to the reference; and (c) whether the $\tau^{-1}/2$ target in the IPO loss produces well-calibrated log-ratio gaps in practice, or whether the target needs adjustment for the high-dimensional autoregressive setting. A strong negative result — e.g., DPO showing normal $\tau$-sensitivity at scale despite the bandit pathology — would narrow the practical significance of the paper's diagnostic. A strong positive result — DPO collapsing to high-KL policies for all $\tau$ while IPO smoothly interpolates — would make a compelling case for switching preference-optimization objectives in production LLM pipelines.

Characterizing the transition region: how deterministic must preferences be before DPO's regularisation breaks? The paper's theoretical analysis identifies the failure at the boundary ($p^* \in \{0,1\}$), but provides no analysis of the intermediate regime where empirical preferences are in $(0,1)$ but skewed. A natural follow-up would systematically vary the skew of empirical preferences — e.g., by controlling the number of preference samples per pair in a synthetic dataset to produce empirical frequencies of $0.6, 0.7, 0.8, 0.9, 0.95, 0.99$ — and measure the effective $\tau$-sensitivity (the KL divergence from $\pi_{ref}$ as a function of $\tau$) for both DPO and IPO. The key question: is the collapse a hard threshold at exactly $\{0,1\}$ (in which case it may be rare in practice), or does $\tau$-sensitivity degrade gradually as preferences become more extreme (in which case it is a continuous practical concern)? The squared loss structure of IPO suggests its $\tau$-sensitivity should be continuous and smooth across all preference strengths; DPO's logistic loss may show a sharp nonlinearity as preferences approach the boundary. This experiment would quantify the "danger zone" for DPO and help practitioners determine whether their specific data distribution is vulnerable.

Solving the $\text{Supp}(\mu) \neq \text{Supp}(\pi_{ref})$ problem: extending IPO to handle unobserved actions. Theorem 2's uniqueness guarantee requires the behavior policy's support to cover the reference policy's support — a condition almost certainly violated in LLM-scale action spaces. The counterexample in Appendix A.2 shows that when this condition fails, the IPO loss has infinitely many minimizers, with no constraint on the probabilities of unobserved actions. A productive follow-up would propose and evaluate regularisation schemes that restore uniqueness or at least bias optimization toward $\pi_{ref}$ for unobserved actions. Options include: (a) adding an explicit penalty on the KL divergence between $\pi$ and $\pi_{ref}$ for unobserved actions only (beyond the global KL penalty already in the objective); (b) using the reference policy to impute pseudo-preferences for unobserved actions (e.g., treating $\pi_{ref}$ as the "preference" target when data is absent); (c) initializing the policy at $\pi_{ref}$ and limiting the number of optimization steps to prevent divergence on unobserved actions (a form of early stopping). Each scheme would be evaluated on a synthetic bandit with unobserved actions (a scaled-up version of D₃) by measuring whether the learned policy's probabilities for unobserved actions remain close to $\pi_{ref}$ as a function of $\tau$.

Stress-testing IPO on cyclic and intransitive preference structures. The paper notes that $\Psi = \text{Identity}$ does not enforce transitivity (unlike $\Psi = \text{logit}$, which imposes it through the additivity of pointwise rewards), but provides no empirical demonstration of IPO's behavior under cyclic preferences. A targeted experiment would construct preference datasets with known cyclic structures — e.g., $p^*(A \succ B) = 0.7, p^*(B \succ C) = 0.7, p^*(C \succ A) = 0.7$ for 3 actions, or more complex intransitive structures with larger action spaces — and compare IPO and DPO on: (a) whether they can represent the intransitive structure at all (DPO, by construction, cannot — it will force a transitive ordering), and (b) the expected preference win rate of the learned policy against the behavior distribution. The hypothesis is that IPO should achieve higher expected win rate on genuinely intransitive preference structures because it does not waste capacity trying to fit a transitive reward model to data that violates transitivity. This experiment would also connect the paper to the literature on social choice and voting theory, where intransitive collective preferences are a well-documented phenomenon (Condorcet cycles), and would establish whether IPO's release from the transitivity assumption has practical benefits or is merely a theoretical curiosity.

The $\Psi$ design space: a systematic empirical comparison of bounded $\Psi$ functions. The paper opens the $\Psi$PO framework but only explores two points in it: $\Psi = \text{logit}$ (RLHF/DPO) and $\Psi = \text{Identity}$ (IPO). A natural follow-up systematically evaluates a spectrum of bounded $\Psi$ functions — e.g., $\Psi(q) = q^p$ for various $p$ (controlling how much extreme preferences are upweighted), sigmoid functions with tunable asymptotes, or piecewise linear functions — measuring: (a) $\tau$-sensitivity (does the learned policy smoothly interpolate between $\pi_{ref}$ and the preference-maximizing policy?), (b) downstream policy quality (expected win rate against a held-out preference distribution), and (c) robustness to dataset size and noise. This would transform the $\Psi$PO framework from a descriptive tool into a prescriptive one, identifying which properties of $\Psi$ (beyond boundedness) matter for practical performance, and providing guidance for practitioners choosing a $\Psi$ for their specific application.

IPO in the context of iterative self-improvement and on-policy data collection. The paper's analysis of RLHF's accidental regularisation (Section 4.2) raises a question about what happens when preference data is collected on-policy — i.e., using the policy currently being optimized to generate the actions that are then compared. In standard RLHF and DPO, the preference data is typically collected once (or periodically) from a relatively fixed behavior policy $\mu$, creating a mismatch with the optimized policy $\pi$. IPO's $\tau$-sensitivity means that as the policy evolves, the effective strength of the preference signal changes (the log-ratio gap is regressed to a fixed $\tau^{-1}/2$ target regardless of how $\pi$ has shifted). A follow-up would simulate an iterative data collection loop: (1) collect preferences using the current policy as $\mu$, (2) train a new policy via IPO on the updated data, (3) repeat. The key measurement is whether IPO's boundedness provides stability across iterations — preventing the "runaway" effect where the policy becomes increasingly extreme because it generates increasingly deterministic comparisons that DPO would amplify unboundedly.

Practical Applications and Downstream Use Cases

LLM alignment fine-tuning with controlled KL budgets. The primary application is a direct replacement for DPO in standard LLM alignment pipelines (e.g., training a chat model to be helpful and harmless). IPO's practical advantage is that the KL budget becomes controllable: the $\tau$ parameter actually governs how far the fine-tuned model can diverge from the base model, because the squared-loss target $\tau^{-1}/2$ is finite for any finite $\tau$, regardless of how confident the preference data is. For teams deploying aligned LLMs where staying close to a known-safe base model is a hard requirement (e.g., in regulated environments or customer-facing applications where off-policy outputs carry legal or reputational risk), IPO provides a guarantee that DPO cannot: even if the preference dataset contains deterministic or near-deterministic preferences, the policy's KL divergence from the reference model is bounded by a function of $\tau$, not potentially unbounded.

Preference-based fine-tuning with small, sparse datasets. The paper's D₃ experiment (Figure 2) demonstrates that DPO collapses the probability of actions with zero observed wins to zero regardless of $\tau$, while IPO maintains non-zero probabilities that depend on $\tau$. This behavior is directly relevant to fine-tuning scenarios where preference data is expensive to collect and therefore sparse — e.g., domain-specific fine-tuning for medical, legal, or scientific applications where expert annotators provide pairwise judgments. In these settings, many high-quality generations may never appear as winners in the preference data simply because they were compared against even better generations or were never sampled. DPO will aggressively suppress these generations; IPO will fall back to the reference model's distribution (the expensive pretrained model that encodes substantial domain knowledge) for actions with sparse preference coverage. The practical benefit is that the fine-tuned model retains the base model's knowledge in areas where preference data is absent, rather than overfitting to the narrow set of generations that happened to appear in the annotated pairs.

Offline preference optimization where running RL is infeasible. A practical motivation for DPO was the elimination of the RL stage, which requires online sampling from the policy during training — a computationally expensive and engineering-intensive requirement. IPO inherits this advantage: it is a pure supervised loss that can be optimized from a static preference dataset with standard gradient descent, requiring no online sampling, no reward model training, and no PPO-style policy optimization. Unlike DPO, however, it does not sacrifice regularisation control for this simplicity. This makes IPO particularly attractive for teams with limited infrastructure (academic labs, smaller companies) who want to fine-tune open-source models on preference data but cannot run the full RLHF pipeline. The implementation complexity is comparable to DPO — the loss in Algorithm 1 is a few lines of code — but the robustness properties are better.

When to Prefer This Method

The paper does not articulate an explicit decision rule for choosing between IPO, DPO, and RLHF across different operational conditions. It argues that IPO avoids a specific failure mode (KL-regularisation collapse under deterministic preferences) that affects both DPO and (in theory) RLHF's policy optimization stage, while noting that RLHF is partially protected by reward-model underfitting in practice. However, it does not provide empirical comparisons against RLHF, nor does it characterize the conditions under which DPO's failure mode is empirically consequential versus benign. The paper's prescriptive stance is qualitative — "IPO is better suited to learn from sampled preferences than DPO" (Conclusion) — without specifying boundary conditions for when the advantage matters. The following is therefore constructed from the paper's theoretical analysis, not from an explicit tradeoff framework stated by the authors.

The theoretical analysis suggests IPO is preferable when:

• The preference dataset is small relative to the action space, making it likely that some actions have zero or near-zero observed win rates due to sampling noise rather than genuine dispreference. This is the D₃ scenario (Figure 2), and it is the expected regime for LLM fine-tuning where the generation space is combinatorially large.
• Controlling the KL divergence from the reference model is a hard requirement — e.g., for safety, regulatory compliance, or because the reference model encodes expensive domain knowledge that must not be discarded. IPO's bounded $\Psi = \text{Identity}$ ensures that $\tau$ remains a genuine control knob even under extreme empirical preferences, which the paper proves is not true for DPO (Section 4.2, Section 5.3.1).
• The preference data may contain intransitive or cyclic patterns that the Bradley-Terry model cannot represent. IPO does not enforce transitivity (unlike DPO/RLHF, which impose it through the additivity of pointwise rewards), so it can faithfully optimize for expected win rate even when pairwise preferences violate transitivity.
• The deployment setting cannot run RL (no online sampling during training) but the team is concerned about DPO's documented tendency to overfit — IPO provides DPO-level simplicity (pure supervised loss, no reward model) with RLHF-level regularisation control (if the paper's claims about RLHF's implicit robustness are correct).

The paper does not establish conditions where DPO is preferable to IPO — it identifies a vulnerability in DPO and demonstrates that IPO avoids it, but does not measure whether DPO outperforms IPO on any metric when preferences are non-deterministic (the regime where the Bradley-Terry model is well-specified and DPO's unboundedness is not triggered). A practitioner with a large, well-distributed preference dataset where empirical frequencies are far from {0,1} has no evidence from this paper for choosing between the methods.