Contrastive Prefence Learning: Learning from Human Feedback without RL

ArXiv: 2310.13639

🎯 Pitch

Human preference judgments are based on regret—how suboptimal an action is—not on raw rewards, yet standard RLHF still tries to learn rewards and then apply RL. This paper shows you can skip both steps entirely: by plugging the regret-based preference model into a contrastive loss that only uses the policy’s own log-probabilities, you get a supervised algorithm that matches or beats PPO-based RLHF while being 1.6× faster and up to 4× more parameter-efficient.

---

1. Executive Summary

This paper introduces Contrastive Preference Learning (CPL), a new family of algorithms for learning optimal policies directly from human feedback without reinforcement learning. CPL operates on the regret-based model of human preferences—where a user's choice between behavior segments reflects which has lower regret under the optimal policy (operationalized as a Boltzmann distribution over the discounted sum of optimal advantages)—and combines it with the maximum entropy RL framework to substitute the optimal advantage function with the policy's log-probability, yielding a purely supervised contrastive objective. On the MetaWorld robotics benchmark using sub-optimal offline data and synthetic regret-based preferences, CPL matches or exceeds PPO and Preference IQL baselines while being 1.6× faster and using less than a quarter of the parameters, and achieves up to 4× parameter efficiency (2.1M vs. 9.6M parameters for image-based tasks). On real human preferences from the D4RL benchmark, CPL outperforms prior methods on three of four tasks, establishing that a fully off-policy supervised objective can recover the expert's optimal policy without value functions or policy gradients—but only when preference data is sufficiently dense to provide informative contrastive signal.

2. Context and Motivation

The Core Problem: RLHF Is Built on a Flawed Assumption and an Unnecessary Algorithm

The paper targets a fundamental mismatch at the heart of contemporary Reinforcement Learning from Human Feedback (RLHF). The standard RLHF pipeline—widely used to align large language models (Ouyang et al., 2022), image generation models (Lee et al., 2023), and robot policies (Christiano et al., 2017)—operates in two phases: first, learn a reward function from human preference data, and second, optimize that learned reward using reinforcement learning. This paradigm assumes that human preferences are distributed according to the discounted sum of rewards (partial returns) of each behavior segment. Under this model, the probability that a user prefers segment $\sigma^+$ to $\sigma^-$ is:

$P[\sigma^+ \succ \sigma^-] = \frac{\exp \sum_{\sigma^+} \gamma^t r_E(s_t, a_t)}{\exp \sum_{\sigma^+} \gamma^t r_E(s_t, a_t) + \exp \sum_{\sigma^-} \gamma^t r_E(s_t, a_t)}$

The paper argues that this assumption is wrong. Recent work by Knox et al. (2022) provides evidence that humans instead evaluate behaviors based on regret—how much worse a given action is compared to what the optimal policy would have done—rather than on the raw sum of rewards. The paper crystallizes this with a concrete counterexample: consider a sparse reward $r_E(s, a) = \mathbf{1}\{s = g\}$ for reaching a goal. Two trajectory segments that both fail to reach the goal would have identical partial returns (both zero), even if one segment moved toward the goal while the other moved away from it. A human observer would naturally prefer the segment that made progress toward the goal, but the partial return model cannot distinguish between them. The regret model, however, assigns higher preference probability to the segment moving toward the goal because its actions have higher optimal advantage—they reduce the expected regret relative to the expert's optimal policy.

This is not merely a philosophical quibble. The paper contends that the partial return model is inconsistent with how humans actually provide feedback, and building algorithms on this mistaken assumption cascades into practical problems throughout the RLHF pipeline.

Why This Problem Matters

The flawed preference model creates two intertwined pathologies that the paper sees as urgent to resolve:

1. The two-phase approach forces practitioners to use RL algorithms that are themselves problematic. Reinforcement learning—whether in the form of policy gradients (PPO) or approximate dynamic programming (IQL)—suffers from well-documented optimization challenges: high-variance gradient estimates (Marbach & Tsitsiklis, 2003), instability from bootstrapping with function approximation (Van Hasselt et al., 2018), and sensitivity to hyperparameters. These challenges are severe enough that the RLHF community has resorted to restricting the problem formulation to avoid them:

• LLM fine-tuning treats text generation as a contextual bandit (Ouyang et al., 2022; Ziegler et al., 2019), where the policy receives a single reward value for a complete response to a query. This eliminates temporal credit assignment—and with it, the need for value functions or policy gradients over long horizons—but it fundamentally ignores that real user interactions with language models are multi-step and sequential. A multi-turn dialogue is a general MDP, not a bandit.
• Robotics RLHF has been limited to low-dimensional state-based control (Christiano et al., 2017; Sikchi et al., 2023a), where approximate dynamic programming methods like Q-learning perform well. Scaling to high-dimensional image inputs—as is standard in modern computer vision-based robotics—has not been demonstrated for these approaches, partly because RL algorithms struggle with the representation learning and optimization challenges that come with large neural networks and visual observations (Ota et al., 2021).

The paper thus identifies a capability ceiling: because RL is hard to scale, RLHF practitioners are forced to constrain the sequential nature or dimensionality of their problems rather than solving them in their full generality. This limits the deployment of preference-based learning to relatively simple settings compared to what supervised learning can handle.

2. Learning a reward function is both unnecessary and introduces additional failure modes. Even setting aside the philosophical objection that human preferences reflect regret rather than reward, the paper points out that reward learning in the Boltzmann rational preference model has a built-in pathology: the model is shift-invariant. Adding a constant to each exponent in the preference model does not change the probability $P[\sigma^+ \succ \sigma^-]$, meaning that reward functions differing by an additive constant are indistinguishable from preference data. This identifiability issue complicates learning and can harm downstream RL optimization because the learned reward function may be poorly scaled or shifted relative to what the RL algorithm expects. Prior work by Hejna & Sadigh (2023) documents that this invariance can degrade reward learning performance.

The paper's framing, then, is that the standard RLHF pipeline is doubly flawed: it learns the wrong quantity (reward instead of advantage/regret) using the wrong optimization tool (RL instead of supervised learning). These are not independent problems—the choice of what to learn determines what optimization machinery is necessary. The paper's key insight, developed fully in Section 3, is that switching to the regret preference model naturally eliminates the need for RL because the optimal advantage function is directly related to the optimal policy's log-likelihood under the maximum entropy framework. This means that learning from preferences becomes a purely supervised learning problem that inherits the scalability and simplicity of supervised methods.

Where Prior Approaches Fall Short

The paper situates itself against several strands of prior work, each of which addresses part of the problem but introduces its own limitations:

Standard two-phase RLHF (reward learning + RL). This is the dominant paradigm, instantiated in methods like PPO with a learned reward (Christiano et al., 2017; Ouyang et al., 2022) and Preference IQL (Hejna & Sadigh, 2023). These methods: (a) assume the partial return preference model, (b) learn a reward function via maximum likelihood on preferences, and (c) optimize it with RL. Their limitations have been discussed above: they learn the wrong quantity (reward instead of regret), they require RL which struggles with high-dimensional or sequential settings, and they are complex—P-IQL, for example, must simultaneously learn a reward function, a Q-function, a value function, and a policy, making it parameter-heavy and sensitive to hyperparameters.

Prior regret-based methods (Knox et al., 2022; 2023). The authors acknowledge that modeling preferences with regret is not their invention. Knox et al. (2022) first proposed the regret preference model, and Knox et al. (2023) further developed the theory. However, the paper argues that existing regret-based algorithms are brittle and unscalable. Specifically, they require estimating gradients with respect to a moving reward function, which in practice has only been approximated through successor features under the assumption of a correct linear or tabular representation of the expert's reward function $r_E$. These constraints confine regret-based methods to simplistic grid-world environments and make them unsuitable for the complex continuous control problems that CPL targets. The paper positions CPL as the first method to operationalize the regret preference model in a way that scales to high-dimensional observations and neural network policies.

Direct Preference Optimization (DPO) (Rafailov et al., 2023). DPO is the closest precursor to CPL in spirit. It also eliminates reward learning and RL in favor of a supervised objective, operating directly on the policy. The paper shows in Appendix A.6 that DPO is a special case of CPL when (a) the MDP terminates after a single step (contextual bandit), (b) all preferences share the same starting state, and (c) the expert's reward $r_E$ is used directly rather than the optimal advantage $A^*$. In the single-step bandit setting, $A^*(s, a) = r_E(s, a) - V^*(s)$, and since all preferences share the same state, the $V^*(s)$ term cancels in the preference model, reducing the regret model to the standard partial-return Boltzmann model. CPL thus generalizes DPO to the full MDP setting with multi-step segments, which is a significant theoretical expansion—but also a practical one, since it enables RLHF on temporally extended tasks like robotic manipulation where DPO's bandit assumption would be violated.

Contextual-bandit RLHF for LLMs. Methods like the one in Ouyang et al. (2022) and Ziegler et al. (2019) have demonstrated impressive results on text generation by treating the problem as a single-step bandit. The paper does not dispute their empirical success but argues that this formulation is a simplifying approximation, not a fundamental solution. Real conversations with LLMs span multiple turns, and the paper contends that CPL's ability to handle sequential preferences could unlock improvements in these settings—though no multi-turn preference dataset currently exists for LLMs, and the paper does not itself conduct language experiments.

Preference-based RL (PbRL). The paper also contextualizes itself within the broader PbRL literature (Furnkranz et al., 2012), which has historically focused on learning from comparisons and rankings. Earlier works (Akrour et al., 2011, 2012; Wilson et al., 2012) established the foundations, and more recent deep learning variants (Christiano et al., 2017; Lee et al., 2021; Ibarz et al., 2018; Brown et al., 2019, 2020) have shown that with thousands of queries, neural network policies can be trained for control. However, the paper notes that these methods have been demonstrated almost exclusively on low-dimensional state-based control, because the RL phase they rely on struggles with the representation learning and optimization demands of high-dimensional inputs and larger networks. CPL's contribution is to show that removing RL from the pipeline—and operating directly on the policy with a contrastive loss—enables scaling to image-based observations where prior PbRL methods have not succeeded.

How This Paper Positions Itself

The paper's positioning can be understood as a unification and transcendence argument:

1. Unification. By adopting the regret preference model and the maximum entropy RL framework, CPL provides a single theoretical umbrella under which prior methods can be understood. DPO emerges as the $k=1$, same-starting-state special case. The standard two-phase RLHF approach corresponds to learning an (implicit) advantage function via reward modeling and then extracting the policy. CPL collapses these two phases into one by exploiting the bijection between advantages and policies in MaxEnt RL.

2. Transcendence. CPL is claimed to be the first method that simultaneously achieves three desiderata: (a) fully off-policy learning (can use any sub-optimal dataset), (b) applicability to arbitrary MDPs (not just contextual bandits), and (c) reliance on only supervised objectives (no policy gradients, no dynamic programming, no value function learning). The paper argues that no prior RLHF method satisfies all three. This triple combination is what enables CPL to operate on sequential decision-making problems with high-dimensional observations and large neural networks—regimes where prior methods either fail or have not been attempted.

The paper's title—"Learning from Human Feedback Without RL"—encapsulates this positioning concisely. The contribution is not a new preference model (that belongs to Knox et al., 2022), nor the idea of directly optimizing a policy from preferences (Rafailov et al., 2023; An et al., 2023), but rather the synthesis of these ideas into a framework that is simultaneously more faithful to the psychology of human preferences (regret-based), simpler to implement and scale (supervised contrastive learning), and more general in its applicability (arbitrary MDPs with sequential feedback). The paper's empirical agenda is to demonstrate that this synthesis works in practice on problems—sequential manipulation from images—that push beyond the complexity envelope of prior RLHF methods.

3. Technical Approach

3.1 Reader Orientation

This paper presents a policy learning algorithm—a procedure that takes a dataset of pairwise human preferences between behavior segments and produces a neural network policy that approximates the human's ideal behavior. The core idea is that by adopting a more psychologically accurate model of how humans generate preferences (one based on regret rather than reward) and combining it with the maximum entropy reinforcement learning framework, the optimal policy can be learned directly through a simple contrastive supervised objective without ever training a reward function, Q-function, or value function, and without running any reinforcement learning algorithm.

The problem CPL solves is: given only a dataset of preferences $\mathcal{D}_{\text{pref}} = \{(\sigma^+_i, \sigma^-_i)\}_{i=1}^n$ where the human prefers segment $\sigma^+_i$ over segment $\sigma^-_i$, find the optimal policy $\pi^*$ that maximizes the human's hidden reward function $r_E$ without ever knowing what $r_E$ is. The "shape" of the solution is a contrastive objective: the learned policy's log-probabilities on the preferred segment are pulled up relative to the unpreferred segment, and through the maximum entropy bijection between optimal advantage functions and optimal policies, this contrastive signal directly shapes the policy toward $\pi^*$.

3.2 Big-Picture Architecture (Diagram in Words)

The CPL system has three major components:

1. Preference Data Generator — produces a dataset $\mathcal{D}_{\text{pref}}$ of pairwise segment preferences. Each comparison says "segment $\sigma^+$ is better than segment $\sigma^-$ under the human's hidden reward $r_E$." This is the only input signal; no reward function, value labels, or optimality demonstrations are given.

2. The Regret Preference Model — a mathematical model of how the human generates preferences, describing the probability that $\sigma^+$ is preferred to $\sigma^-$ as a Boltzmann distribution over the discounted sum of the optimal advantage function $A^*(s, a)$ along each segment. This model encodes the assumption that humans prefer behaviors with lower regret under their optimal policy.

3. The CPL Objective and Policy Network — a feed-forward process that takes the preference data, applies the maximum entropy identity $A^*(s, a) = \alpha \log \pi^*(a|s)$ to replace the unknown advantage function with the policy's own log-probabilities, and then trains the policy via maximum likelihood on the induced preference distribution. The result is a single neural network representing $\pi_\theta(a|s)$; there is no separate reward model, Q-network, or value network.

Information flows as follows: preferences between segments enter the system → the regret preference model interprets each preference as a signal about the relative sums of optimal advantages on the two segments → the max-entropy bijection converts that signal into relative sums of policy log-probabilities → the CPL loss function (a contrastive logistic loss) updates the policy's parameters to increase the likelihood of actions on preferred segments relative to unpreferred ones → at convergence, the policy recovers $\pi^*$.

3.3 Roadmap for the Deep Dive

• First, the formal derivation of CPL's core equation (Eq. 5), which transforms the regret preference model into a policy-only objective. This is the intellectual centerpiece and must be understood step-by-step.
• Second, the proof sketch for Theorem 1 (convergence to the optimal policy), since it establishes that the learned implicit advantage function is both Bellman-consistent and recovers $\pi^*$ with unbounded data.
• Third, the non-convexity analysis and the conservative bias regularizer (Eq. 6), which addresses the practical problem that in finite data regimes, CPL's objective admits multiple solutions, some of which place probability on out-of-distribution actions.
• Fourth, the practical training recipes: BC pre-training, the bias regularizer parameter $\lambda$, the temperature parameter $\alpha$, and the CPL variants for different data types (rankings, dense preferences, KL-constrained settings).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical derivation paper whose core idea is that substituting the max-entropy identity $A^*(s, a) = \alpha \log \pi^*(a|s)$ into the regret-based preference model yields a supervised contrastive objective whose optimum is the expert's optimal policy.

---

The Regret Preference Model Formalized

The foundation CPL builds on is the regret-based model of human preferences from Knox et al. (2022). Under this model, when a human is shown two behavior segments $\sigma^+$ and $\sigma^-$ (each a length-$k$ sequence of state-action pairs), the probability they prefer $\sigma^+$ is:

$P_{A^*}[\sigma^+ \succ \sigma^-] = \frac{\exp \sum_{\sigma^+} \gamma^t A^*(s^+_t, a^+_t)}{\exp \sum_{\sigma^+} \gamma^t A^*(s^+_t, a^+_t) + \exp \sum_{\sigma^-} \gamma^t A^*(s^-_t, a^-_t)}$

where:

• $\sigma^+ = (s^+_1, a^+_1, s^+_2, a^+_2, \ldots, s^+_k, a^+_k)$ is the preferred behavior segment,
• $\sigma^-$ is the unpreferred segment,
• $\gamma \in (0, 1]$ is the discount factor (assumed known and shared between human and algorithm),
• $A^*(s, a) = Q^*(s, a) - V^*(s)$ is the optimal advantage function under the human's hidden reward $r_E$,
• $Q^*(s, a)$ is the optimal state-action value (expected future return from taking action $a$ in state $s$ and then acting optimally),
• $V^*(s) = \mathbb{E}_{a \sim \pi^*(s)}[Q^*(s, a)]$ is the optimal state value (expected future return from state $s$ under the optimal policy),
• and $A^*(s_t, a_t)$ measures how much better or worse action $a_t$ is compared to what the optimal policy would have chosen at state $s_t$. Formally, it is the negated regret—actions with high optimal advantage are ones the human's ideal policy would be likely to take, and actions with low or negative optimal advantage represent mistakes relative to that ideal.

What this equation computes: it is a logistic (sigmoid-activated) comparison between the total discounted optimal advantage accumulated on $\sigma^+$ versus $\sigma^-$. If $\sigma^+$'s actions are closer to what the optimal policy would do—yielding higher summed advantage—the exponent in its numerator is larger, and the preference probability increases toward 1. The denominator normalizes the two segments' scores into a valid probability. This is operationally identical to a Bradley-Terry model of paired comparisons (Bradley & Terry, 1952), but with the score of each segment being its discounted sum of optimal advantages rather than its discounted sum of rewards.

Why this form over the partial-return model: the standard RLHF approach assumes the score is $\sum_{\sigma} \gamma^t r(s_t, a_t)$. That form cannot distinguish between two segments that both fail to reach a goal in sparse-reward tasks but differ in their approach behavior, because the sum of rewards is zero for both. The optimal advantage $A^*(s, a)$ captures progress toward optimality, not just reward accumulation—if an action moved toward a goal state even without reaching it, $Q^*(s, a)$ would be higher than for an action that moves away, and $A^*(s, a)$ would reflect that the action is "closer" to optimal. The regret model thus provides a continuous preference signal even in sparse-reward settings where the partial-return model would see no difference.

---

The Maximum Entropy Bijection: From Advantage to Policy

The paper's central technical maneuver is to replace the optimal advantage function $A^*$—which is unknown and difficult to learn directly—with the optimal policy's log-probability $\log \pi^*$. This substitution is possible because of a fundamental relationship in maximum entropy reinforcement learning.

In standard RL, the objective is to maximize the expected sum of rewards $\mathbb{E}_{\pi}[\sum_t \gamma^t r(s_t, a_t)]$. In maximum entropy RL (MaxEnt RL), an entropy bonus is added to encourage exploration and robustness:

$\pi^* = \arg\max_\pi \mathbb{E}_\pi\left[\sum_{t=0}^\infty \gamma^t (r_E(s_t, a_t) - \alpha \log \pi(a_t|s_t))\right]$

where $\alpha > 0$ is a temperature parameter controlling the strength of the entropy regularization relative to the reward. The term $-\alpha \log \pi(a_t|s_t)$ penalizes deterministic behavior by subtracting the policy's negative log-likelihood (which is exactly the policy's entropy at that state-action pair), encouraging the policy to remain stochastic.

Ziebart (2010) proved that under this MaxEnt objective, the optimal policy and optimal advantage function satisfy an exact bijection:

$\pi^*(a|s) = e^{A^*(s, a) / \alpha}$

and equivalently by taking the logarithm:

$A^*(s, a) = \alpha \log \pi^*(a|s)$

where $A^*(s, a)$ is the optimal advantage under reward $r_E$ and temperature $\alpha$, and $\pi^*(a|s)$ is the corresponding optimal MaxEnt policy.

What this equality means operationally: for the optimal MaxEnt policy, the advantage of action $a$ at state $s$ is simply $\alpha$ times the log-probability of that action under the optimal policy. An action that is twice as likely under $\pi^*$ as another action has an advantage difference of $\alpha \log 2$ nats. An action the optimal policy would never take (probability zero) has an advantage of negative infinity. This is not an approximation—it is an exact equality that holds for the optimal solution of the MaxEnt RL objective with reward $r_E$.

Why this form matters for CPL: this bijection means that preferences, which provide information about $A^*$, actually directly provide information about $\pi^*$. We do not need to learn $A^*$ explicitly and then derive $\pi^*$ from it (which would require solving an RL problem). We can learn $\pi^*$ directly by substituting $\alpha \log \pi^*$ for $A^*$ wherever it appears. This eliminates the entire RL phase from the pipeline.

Why alternatives fail: if one attempted to learn the advantage function $A_\theta$ directly from preferences—maximizing the log-likelihood $\log P_{A_\theta}[\sigma^+ \succ \sigma^-]$—the learned $A_\theta$ must satisfy $\int_\mathcal{A} e^{A_\theta(s, a)/\alpha} \, da = 1$ for all states $s$ to correspond to a valid optimal advantage (since $\pi^*$ must integrate to 1). Enforcing this normalization constraint is intractable with continuous actions and large neural networks. By learning the policy directly, the constraint $\int_\mathcal{A} \pi_\theta(a|s) \, da = 1$ is automatically satisfied by construction of the policy network (which outputs a normalized probability distribution, e.g., a Gaussian with unit variance). CPL thus exchanges an intractable constrained optimization over advantage functions for a natural unconstrained optimization over policies.

---

Deriving the CPL Objective (Eq. 5)

The CPL objective is obtained by substituting the max-entropy bijection $A^*(s, a) = \alpha \log \pi^*(a|s)$ into the regret preference model, then maximizing the log-likelihood of the preference data under that model.

Starting from the regret preference model:

$P_{A^*}[\sigma^+ \succ \sigma^-] = \frac{\exp \sum_{\sigma^+} \gamma^t A^*(s^+_t, a^+_t)}{\exp \sum_{\sigma^+} \gamma^t A^*(s^+_t, a^+_t) + \exp \sum_{\sigma^-} \gamma^t A^*(s^-_t, a^-_t)}$

Substitute $A^*(s_t, a_t) = \alpha \log \pi^*(a_t|s_t)$ for each state-action pair in both segments:

$P_{\pi^*}[\sigma^+ \succ \sigma^-] = \frac{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi^*(a^+_t | s^+_t)}{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi^*(a^+_t | s^+_t) + \exp \sum_{\sigma^-} \gamma^t \alpha \log \pi^*(a^-_t | s^-_t)}$

This now expresses the preference probability solely in terms of the unknown optimal policy $\pi^*$. To learn an approximation $\pi_\theta$, we maximize the log-likelihood of the dataset $\mathcal{D}_{\text{pref}}$ under this model:

$\mathcal{L}_{\text{CPL}}(\pi_\theta, \mathcal{D}_{\text{pref}}) = \mathbb{E}_{(\sigma^+, \sigma^-) \sim \mathcal{D}_{\text{pref}}}\left[-\log \frac{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t)}{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t) + \exp \sum_{\sigma^-} \gamma^t \alpha \log \pi_\theta(a^-_t|s^-_t)}\right]$

where:

• $\pi_\theta$ is the learned policy with parameters $\theta$,
• $\mathcal{D}_{\text{pref}} = \{(\sigma^+_i, \sigma^-_i)\}_{i=1}^n$ is the dataset of $n$ pairwise preferences,
• the expectation is an empirical average over the $n$ comparisons,
• $\gamma^t \alpha \log \pi_\theta(a_t|s_t)$ is the contribution of step $t$ to the total segment score, discounted by $\gamma^t$ and scaled by $\alpha$,
• and the negative log is the standard cross-entropy loss for a binary classification problem where the positive class is $\sigma^+$.

What this loss computes operationally: for each comparison in the dataset, the loss function (1) computes the discounted sum of log-probabilities of the taken actions under the current policy for both segments, (2) exponentiates these sums to get unnormalized scores, (3) computes the logistic probability that $\sigma^+$ is preferred by dividing its score by the sum of both scores, and (4) penalizes the negative log of this probability. If the policy assigns much higher log-probability to actions in $\sigma^+$ than $\sigma^-$, the score ratio is large, the predicted preference probability is close to 1, and the loss is near zero. If the two scores are similar, the predicted probability is near 0.5 and the loss is $-\log(0.5) = \log 2$. If the policy incorrectly assigns higher probability to $\sigma^-$, the predicted probability is below 0.5 and the loss exceeds $\log 2$, driving a gradient signal to increase $\pi_\theta(a^+_t|s^+_t)$ and decrease $\pi_\theta(a^-_t|s^-_t)$.

Why this form enables policy learning without RL: the gradient of this loss flows directly through the policy's output log-probabilities. There is no reward function to learn, no value function to bootstrap, no Q-function to approximate, and no policy gradient to estimate from trajectory returns. The only learnable component is $\pi_\theta$, and its training signal comes entirely from relative comparisons between preference-labeled segments. This is a standard supervised classification objective—specifically, the Noise Contrastive Estimation (NCE) objective (Gutmann & Hyvärinen, 2010) where the positive example is $\sigma^+$, the negative example is $\sigma^-$, and the score of a segment is its discounted sum of log-probabilities.

Why the shift-invariance problem is automatically resolved: in the standard reward-based preference model, adding a constant $c$ to the reward function $r_E$ changes $\sum \gamma^t r_E$ to $\sum \gamma^t (r_E + c)$, and the constant appears in both the numerator and denominator of the logistic, canceling out. This makes reward functions unidentifiable up to an additive constant. In CPL, the policy's distributional constraint $\int_\mathcal{A} \pi_\theta(a|s) \, da = 1$ prevents any analogous degeneracy: if one adds a constant to $\log \pi_\theta$, the exponentials scale proportionally, but the policy constraint would be violated unless the action space is uniformly re-weighted. Since the policy network inherently normalizes its output (e.g., a Gaussian density integrates to 1), there is no free parameter shift that preserves both the preference probabilities and the valid-policy constraint. The learned implicit advantage function is thus always consistent (see Proposition 1 below).

---

Theoretical Guarantees: Convergence (Theorem 1) and Consistency (Proposition 1)

The paper provides two formal results that establish CPL's correctness, proven in Appendix A.

Theorem 1 (Convergence). The statement is: "Assume an unbounded number of preferences generated from a noisy rational regret-preference model with expert advantage function $A^*$. CPL recovers the optimal policy $\pi^*$ corresponding to reward $r_E$."

The proof strategy, elaborated in Appendix A.2, works as follows. Let $\hat{A}(s, a) = \log \hat{\pi}(a|s)$ be the implicit advantage function learned by CPL (setting $\alpha=1$ without loss of generality). The CPL loss is equivalent to minimizing the KL divergence between the true preference distribution $P_{A^*}[\sigma^+ \succ \sigma^-]$ and the predicted distribution $P_{\hat{A}}[\sigma^+ \succ \sigma^-]$ for all possible segment pairs. With unbounded data and sufficient representation power, this KL divergence can be driven to zero pointwise, meaning:

$\frac{e^{\hat{A}(\sigma^+_k)}}{e^{\hat{A}(\sigma^+_k)} + e^{\hat{A}(\sigma^-_k)}} = \frac{e^{A^*(\sigma^+_k)}}{e^{A^*(\sigma^+_k)} + e^{A^*(\sigma^-_k)}}$

for all segment pairs $\sigma^+_k, \sigma^-_k$ of arbitrary length $k$, where $A(\sigma_k) = \sum_{t=1}^k \gamma^t A(s_t, a_t)$ denotes the discounted sum of advantages along segment $\sigma_k$. Rearranging gives $e^{\hat{A}(\sigma^+_k)} e^{A^*(\sigma^-_k)} = e^{A^*(\sigma^+_k)} e^{\hat{A}(\sigma^-_k)}$. Using the consistency property that $\int_\mathcal{A} e^{\hat{A}(s,a)} \, da = 1$ and $\int_\mathcal{A} e^{A^*(s,a)} \, da = 1$ for all states (since both are derived from valid MaxEnt policies), one can inductively strip off the last state-action pair from each segment, showing that the equality holds for segments of length $k-1$, then $k-2$, and ultimately for individual state-action pairs, yielding $A^*(s, a) = \hat{A}(s, a)$ for all $(s, a)$ and consequently $\hat{\pi} = \pi^*$.

What this theorem establishes: CPL is not merely a heuristic that improves performance in practice—it is a consistent estimator of the optimal policy under the regret preference model. With infinite data, it recovers $\pi^*$ exactly. This distinguishes CPL from methods that use RL to optimize a learned reward function, where convergence to the optimal policy is not guaranteed even with a perfect reward model due to optimization error in the RL phase.

Proposition 1 (Consistency). The statement is: "CPL learns a consistent advantage function." Consistency is defined in Definition 1: an advantage function $A(s, a)$ is consistent if there exists some reward function $r(s, a)$ for which $A$ is the optimal MaxEnt advantage function—that is, $A(s, a) = A^*_r(s, a)$ for some $r$.

The proof (Appendix A.1) is straightforward from the policy constraint. Since CPL optimizes a valid policy $\pi_\theta$ satisfying $\int_\mathcal{A} \pi_\theta(a|s) \, da = 1$ and defines the implicit advantage as $\hat{A}(s, a) = \alpha \log \pi_\theta(a|s)$, it follows that $\int_\mathcal{A} e^{\hat{A}(s,a)/\alpha} \, da = \int_\mathcal{A} \pi_\theta(a|s) \, da = 1$. By Lemma 1, any function satisfying this normalization constraint is the optimal MaxEnt advantage for the reward function defined by $\hat{r}(s, a) = \hat{A}(s, a) = \alpha \log \pi_\theta(a|s)$. Therefore, $\hat{A}$ is consistent.

What this means practically: no matter how much or how little preference data CPL receives, nor how sub-optimal the learned policy is, the implicit advantage function $\hat{A} = \alpha \log \pi_\theta$ is always the optimal advantage for some reward function. Adding more data refines the implicit reward $\hat{r}$ toward the true $r_E$, but the consistency property holds at all times. This stands in contrast to learning an arbitrary advantage network $A_\phi$ by maximum likelihood on preferences: without the normalization constraint, $A_\phi$ may not correspond to any valid optimal MaxEnt advantage, potentially leading to a policy extraction step (e.g., $e^{A_\phi/\alpha}$) that does not produce a consistent improvement direction. CPL sidesteps this by baking the normalization into the policy architecture itself.

Corollary 1 (Reward-Advantage equivalence). The optimal MaxEnt policy for reward $r_E$ is the same as the optimal MaxEnt policy for the reward function $r'_E(s, a) = A^*_{r_E}(s, a)$. This means that if CPL's implicit advantage $\hat{A}$ approximates $A^*_{r_E}$, the policy $\pi_\theta$ is optimizing a reward function whose optimal policy is exactly $\pi^*_{r_E}$. This corollary also justifies the paper's P-IQL baseline: P-IQL learns a reward function from regret-based preferences (which effectively estimates $A^*_{r_E}$) and then uses RL to optimize it; since $A^*_{r_E}$ as a reward function preserves the optimal policy of $r_E$, P-IQL with a perfect reward model would recover $\pi^*$. The corollary extends the insight of Knox et al. (2023) to the MaxEnt RL setting and formally establishes that regret-based preferences and advantage-as-reward are a compatible pair.

---

Finite Data, Non-Convexity, and the Need for Regularization

While Theorem 1 guarantees convergence with infinite data, practical deployments operate with finite datasets. The CPL objective in Eq. (5) is convex but not strictly convex in the log-probabilities $\log \pi_\theta$, meaning multiple different policies can achieve the same optimal loss on a finite dataset.

To see why, the paper reformulates CPL as logistic regression. Let the policy be represented by a one-dimensional vector $\pi \in \mathbb{R}^{|S \times A|}$ over the discrete state-action space (the continuous case is conceptually analogous but technically more involved). For each preference comparison $\sigma^+ \succ \sigma^-$, define a "comparison vector" $x \in \mathbb{R}^{|S \times A|}$ where:

$x[s, a] = \begin{cases} \gamma^t & \text{if } (s, a) = (s^+_t, a^+_t) \text{ for some } t \\ -\gamma^t & \text{if } (s, a) = (s^-_t, a^-_t) \text{ for some } t \\ 0 & \text{otherwise} \end{cases}$

Then the discounted sum difference can be written as a dot product:

$\sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t) - \sum_{\sigma^-} \gamma^t \alpha \log \pi_\theta(a^-_t|s^-_t) = \alpha \, x^\top \log \pi$

And the CPL loss becomes the standard logistic regression loss:

$\mathcal{L}_{\text{CPL}}(\pi_\theta, \mathcal{D}_{\text{pref}}) = -\sum_{i=1}^{|\mathcal{D}_{\text{pref}}|} \log \operatorname{logistic}(\alpha \, x_i^\top \log \pi(a|s))$

where $\operatorname{logistic}(z) = 1/(1 + e^{-z})$.

The null-space problem. Assemble all $n$ comparison vectors into a matrix $X \in \mathbb{R}^{n \times |S \times A|}$ where row $i$ is $x_i$. Any change to $\log \pi$ that lies in the null space of $X$—that is, any vector $u$ such that $X u = 0$—leaves all the dot products $x_i^\top \log \pi$ unchanged, and therefore leaves the loss unchanged. Since typically $|S \times A| \gg n$ (there are far more possible state-action pairs than preference comparisons), the null space is non-trivial: there exist many different policies that produce exactly the same CPL loss on the given data.

Why this is dangerous for offline learning. Among the policies that minimize the CPL loss, some may place high probability on state-action pairs that never appear in the dataset—so-called out-of-distribution (OOD) actions. In offline RL, executing such a policy would lead to states outside the training distribution, where the policy's performance is unpredictable and typically poor. The principle of pessimism in offline RL dictates that one should prefer solutions that stay close to the data distribution when multiple explanations are consistent with the observed data (Levine et al., 2020; Jin et al., 2021). CPL's unregularized objective does not encode this preference.

The paper provides an explicit construction in Appendix A.3: for a single-state MDP with three actions and preferences only between two of them, the condition $\alpha \log \pi(a_1|s) - \alpha \log \pi(a_2|s) = \log(c_1/c_2)$ (matching the empirical preference ratio) determines only the relative probability of $a_1$ and $a_2$, leaving the probability of the unseen action $a_3$ completely free. A policy with $\pi = [0.5, 0.5, 0.0]$ and one with $\pi = [0.1, 0.1, 0.8]$ both satisfy the condition and achieve identical CPL loss, but the latter is entirely OOD on action $a_3$.

---

The Conservative Bias Regularizer (Eq. 6)

To resolve the null-space problem, the paper introduces a conservative bias regularizer adapted from An et al. (2023). The regularizer modifies the CPL objective by down-weighting the negative (unpreferred) segment's log-probability sum by a factor $\lambda \in (0, 1)$:

$\mathcal{L}_{\text{CPL}(\lambda)}(\pi_\theta, \mathcal{D}_{\text{pref}}) = \mathbb{E}_{(\sigma^+, \sigma^-) \sim \mathcal{D}_{\text{pref}}}\left[-\log \frac{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t)}{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t) + \exp \lambda \sum_{\sigma^-} \gamma^t \alpha \log \pi_\theta(a^-_t|s^-_t)}\right]$

where:

• $\lambda \in (0, 1)$ is the bias hyperparameter controlling how much the negative segment is discounted,
• all other symbols retain their earlier definitions,
• and the only change from Eq. (5) is the $\lambda$ multiplier in the denominator's second exponential term.

What this loss computes differently from standard CPL: the logistic now compares the positive segment's score $\sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta$ against a shrunk version of the negative segment's score $\lambda \sum_{\sigma^-} \gamma^t \alpha \log \pi_\theta$. Since $\lambda < 1$, the negative segment's score is effectively reduced, making the positive segment's relative advantage larger in the logistic. The optimization now has an additional incentive to increase $\log \pi_\theta$ on the positive segment beyond what is needed just to beat the negative segment in the comparison.

Why this favors in-distribution actions (Proposition 2). The key insight is that the bias regularizer breaks ties between policies that have the same standard CPL loss. Consider two different preference comparisons $(\sigma^+, \sigma^-)$ and $(\sigma'^+, \sigma'^-)$ that produce the same logistic logit—i.e., they differ only by an additive constant in the log-probabilities that cancels in the difference. If the first comparison involves actions with higher overall log-probability under $\pi_\theta$ (meaning they are more likely to be in the dataset), then the regularized loss $\mathcal{L}_{\text{CPL}(\lambda)}$ is lower for the first comparison than the second. This is because, when the positive segment's log-probabilities are larger, the effective penalty from discounting the negative segment's score by $\lambda$ is amplified relative to the logit-difference, reducing the loss. The regularizer thus creates a gradient that pushes the policy toward assigning higher likelihood to state-action pairs that appear in the preference data, keeping it closer to the offline data distribution.

Proof sketch of Proposition 2 (from Appendix A.4). Assume two comparisons produce the same standard CPL loss, meaning their logit differences are equal:

$\sum_{\sigma^+} \gamma^t \log \pi(a_t|s_t) - \sum_{\sigma^-} \gamma^t \log \pi(a_t|s_t) = \sum_{\sigma'^+} \gamma^t \log \pi(a_t|s_t) - \sum_{\sigma'^-} \gamma^t \log \pi(a_t|s_t)$

Further assume the first comparison has higher overall log-probability: $\sum_{\sigma^+} \gamma^t \log \pi(a_t|s_t) > \sum_{\sigma'^+} \gamma^t \log \pi(a_t|s_t)$. This implies the same difference for the negative segments. Plugging into the regularized loss (simplified to an algebraic form) shows:

$\mathcal{L}_{\text{CPL}(\lambda)}(\pi, \sigma^+ \succ \sigma^-) = \log\left(1 + \exp(\delta(1-\lambda)) \exp\left(\lambda \sum_{\sigma^-} \gamma^t \log \pi(a_t|s_t) - \sum_{\sigma^+} \gamma^t \log \pi(a_t|s_t)\right)\right) < \mathcal{L}_{\text{CPL}(\lambda)}(\pi, \sigma'^+ \succ \sigma'^-)$

where $\delta > 0$ is the log-probability surplus of the first comparison over the second. Since $\delta > 0$, $\lambda < 1$, the term $\exp(\delta(1-\lambda)) > 1$, making the regularized loss strictly larger for the comparison with lower overall probability. The regularizer thus creates a monotonic preference: among policies with equal standard CPL loss, those with higher likelihood on the dataset's state-action pairs achieve lower regularized loss.

Hyperparameter selection. The paper uses $\lambda = 0.5$ for all MetaWorld experiments and D4RL experiments (Table 6). Other values were ablated: Figure 2 (right panel) shows results for $\lambda \in \{0.25, 0.5, 0.75\}$ on Drawer Open, finding that all values perform well, with $\lambda = 0.5$ performing best. The temperature $\alpha$ was set to $0.1$ for MetaWorld and $0.2$ for D4RL (Table 6). The ablation in Figure 2 (right) tested $\alpha \in \{0.02, 0.1, 0.5\}$, finding that $\alpha = 0.1$ performed best, and the authors note that "higher performance could have been attained with further hyper-parameter tuning, particularly for $\lambda$."

---

Pretraining with Behavior Cloning

Following the common practice in RLHF (Ouyang et al., 2022; Ziegler et al., 2019), the paper pretrains the policy $\pi_\theta$ using behavior cloning (BC) on the preference dataset's action data before fine-tuning with the CPL loss:

$\min_\theta \mathbb{E}_{(s, a) \sim \mathcal{D}} [-\log \pi_\theta(a|s)]$

where $\mathcal{D}$ is the set of all state-action pairs from the sub-optimal rollout data used for preferences (in the paper's experiments, this is the same data from which preference segments are sampled).

Pretraining hyperparameters. For MetaWorld state-based experiments, the BC pretraining phase runs for 200,000 steps out of a total of 500,000 training steps. For image-based sparse experiments: 80,000 pretraining steps out of 200,000 total. For image-based dense experiments: 40,000 pretraining steps out of 120,000 total. A dotted vertical line marks the transition from pre-training to CPL fine-tuning in all learning curve figures (Figures 3–6 in the Appendix).

Mixed effectiveness. The paper reports that pretraining "helped performance in some cases, but hurt it others" (Section 3.2, "Pretraining"). Figure 10 in Appendix C.5 shows that for the 2.5K dense state-based datasets, CPL without pretraining actually outperforms CPL with pretraining in some environments (e.g., Bin Picking, Button Press). The authors posit that "pretraining helps when a policy closer to the data distribution is desirable, particularly when out-of-distribution actions are detrimental." Since the conservative bias regularizer already encourages in-distribution actions, pretraining may be partially redundant or may bias the policy too strongly toward the sub-optimal data distribution, slowing the contrastive fine-tuning phase. The D4RL real-human preference experiments explicitly do not use pretraining (Section 4.3).

---

Variants for Different Data Types

The paper describes several instantiations of the CPL framework tailored to different preference data formats. While the main paper experiments use only the pairwise comparison version (Eq. 6), the appendix details three variants:

1. CPL for rankings (Plackett-Luce model). When the data consists of a total ordering $\sigma_1 \succ \sigma_2 \succ \cdots \succ \sigma_K$ over $K$ segments (rather than independent pairs), the preference probability under the regret model follows a Plackett-Luce distribution (Plackett, 1975):

$P(\tau | \sigma_1, \ldots, \sigma_K) = \prod_{k=1}^K \frac{\exp \sum_{\sigma_{\tau(k)}} \gamma^t A^*(s_t, a_t)}{\sum_{j=k}^K \exp \sum_{\sigma_{\tau(j)}} \gamma^t A^*(s_t, a_t)}$

Substituting the max-entropy bijection and taking the negative log yields the CPL objective for rankings:

$\mathcal{L}_{\text{CPL}}(\pi_\theta, \mathcal{D}_{\text{rank}}) = \mathbb{E}_{(\sigma_1, \ldots, \sigma_K) \sim \mathcal{D}_{\text{rank}}}\left[-\sum_{k=1}^K \log \frac{\exp \sum_{\sigma_k} \gamma^t \alpha \log \pi_\theta(a^k_t|s^k_t)}{\sum_{j=k}^K \exp \sum_{\sigma_j} \gamma^t \alpha \log \pi_\theta(a^j_t|s^j_t)}\right]$

What this form means: for each rank position $k$, the score of the $k$-th ranked segment is compared against the scores of all segments ranked at or below $k$, normalized by their sum. This is exactly the InfoNCE objective (Oord et al., 2018) where the positive example at each step is the $k$-th ranked segment and the negatives are all lower-ranked segments. The paper notes this connection but does not evaluate ranking-based CPL experimentally.

2. BC-Regularized CPL. Instead of the bias regularizer, one can add an explicit behavior cloning penalty. Starting from the constrained optimization problem:

$\min_\pi \mathcal{L}_{\text{CPL}}(\pi_\theta, \mathcal{D}_{\text{pref}}) \quad \text{s.t.} \quad \mathbb{E}_{s \sim \rho_\mu}[D_{\text{KL}}(\mu(\cdot|s) \| \pi(\cdot|s))] < \epsilon$

where $\mu$ is the data distribution (estimated by BC) and $\rho_\mu$ is the state distribution under $\mu$, the Lagrangian relaxation with multiplier $\beta$ yields:

$\min_\pi \mathcal{L}_{\text{CPL}}(\pi_\theta, \mathcal{D}_{\text{pref}}) - \beta \, \mathbb{E}_{(a,s) \sim \rho_\mu}[\log \pi(a|s)]$

This is referred to as CPL (BC) in Appendix B. The paper tested it (Figures 3–6 in Appendix C) and found it generally performed worse than the bias-regularized version, though learning curves are provided in the appendix.

3. KL-Constrained CPL. This variant assumes preferences follow the KL-constrained advantage function rather than the maximum entropy advantage. Under a reference distribution $\mu(a|s)$ (trained by BC), the constrained optimal policy satisfies $\pi^*(a|s) = \mu(a|s) e^{A^*(s,a)/\alpha}$, or equivalently $A^*(s,a) = \alpha \log \frac{\pi^*(a|s)}{\mu(a|s)}$. Substituting this into the regret preference model and adding the bias regularizer yields:

$\mathcal{L}_{\text{CPL-KL}(\lambda)}(\pi_\theta, \mathcal{D}_{\text{pref}}) = \mathbb{E}_{(\sigma^+, \sigma^-)}\left[-\log \frac{\exp \sum_{\sigma^+} \gamma^t \alpha \log \frac{\pi_\theta(a^+_t|s^+_t)}{\mu(a^+_t|s^+_t)}}{\exp \sum_{\sigma^+} \gamma^t \alpha \log \frac{\pi_\theta(a^+_t|s^+_t)}{\mu(a^+_t|s^+_t)} + \exp \lambda \sum_{\sigma^-} \gamma^t \alpha \log \frac{\pi_\theta(a^-_t|s^-_t)}{\mu(a^-_t|s^-_t)}}\right]$

What this form represents: it is a multi-step generalization of Direct Preference Optimization (DPO) (Rafailov et al., 2023). In the special case where (a) segments are length 1, (b) all preferences start from the same state, and (c) $\lambda = 1$, this reduces exactly to the DPO objective. The paper shows in Appendix A.6 that DPO is a special case of CPL in the contextual bandit setting, because with length-1 segments and same starting state, the regret preference model reduces to the partial-return model (since $A^*(s,a) = r_E(s,a) - V^*(s)$ and $V^*(s)$ cancels in the comparison). CPL (KL) was tested in some MetaWorld experiments (Figures 4 and 9 in Appendix C) and performed comparably to standard CPL on some tasks.

4. Dense CPL with Transitive Augmentation. When preference labels are dense (e.g., all pairwise comparisons among a batch of segments are known), the CPL loss can be augmented to exploit transitivity. Given a batch of $b$ segments with a known total ordering, all $b(b-1)/2$ possible pairwise comparisons are used in the loss:

$\mathcal{L}_{\text{CPL}(\lambda)\text{-D}} = -\sum_{i=1}^b \sum_{j=1}^b \mathbf{1}\{\sigma_i \succ \sigma_j\} \log \frac{\exp \sum_{\sigma_i} \gamma^t \alpha \log \pi_\theta(a^i_t|s^i_t)}{\exp \sum_{\sigma_i} \gamma^t \alpha \log \pi_\theta(a^i_t|s^i_t) + \exp \lambda \sum_{\sigma_j} \gamma^t \alpha \log \pi_\theta(a^j_t|s^j_t)}$

The paper applied this technique to image-based CPL experiments and "found that it lead to a slight increase in performance for some tasks" (Appendix B).

---

Practical Implementation Details

Policy architecture and log-probability computation. For MetaWorld experiments, the policy network outputs a Gaussian distribution with a fixed variance, so the log-probability is computed as $\log \pi_\theta(a|s) = -\| \pi(s) - a \|^2_2$ where $\pi(s)$ is the deterministic mean output of the network. For D4RL experiments, a diagonal Gaussian distribution with a learned variance per action dimension is used (Appendix D.4).

Network architecture and hyperparameters. For state-based MetaWorld experiments, the policy is a 2-layer MLP with hidden size 512 and dropout 0.25. For image-based experiments, the DrQv2 architecture (Yarats et al., 2022) is used with dropout 0.5. Training uses a learning rate of 0.0001 for all CPL variants (Table 6). The discount factor $\gamma$ is set to 1 (undiscounted accumulation) for all CPL experiments, while P-IQL uses $\gamma = 0.99$ (Table 7).

Segment sizes and batching. Segments of length 64 are sampled uniformly from the rollout data. For state-based experiments, the batch size is 96 comparisons (each comparison spans $2 \times 64 = 128$ state-action pairs, for $96 \times 128 = 12,288$ total states per batch). For image-based sparse experiments, batch size is 48; for image-based dense, batch size is 32. The reduction in batch size for image experiments is attributed to GPU memory constraints from processing high-dimensional visual observations through the DrQv2 encoder.

Preference label generation. For synthetic experiments, regret-based preference labels are computed using the Q-function and policy of an oracle Soft Actor-Critic (SAC) model trained to 100% success rate on a combination of the sub-optimal rollout data and online data. The optimal advantage is estimated via the "sum" variant described in Appendix D.2, where:

$-\text{regret}(\sigma) = \gamma^k V^*(s_k) - V(s_0) + \sum_{t=0}^{k-1} \gamma^t r(s_t, a_t)$

This formulation reduces variance compared to directly computing $Q^*(s_t, a_t) - V^*(s_t)$ at each step (since the Q and V estimates have correlated errors that partially cancel). $V^*(s)$ is estimated by evaluating the SAC Q-function on 64 Markov Chain Monte Carlo samples from the oracle policy $\pi^*$. Figure 12 in the appendix compares agreement rates between different advantage estimation methods, showing that the "sum" variant has the highest average agreement across tasks.

Training procedure. For all experiments, preference comparisons are generated by first collecting 2,500 sub-optimal rollout episodes from a policy checkpoint achieving approximately 50% success rate (exact rates per environment given in Table 4), then sampling segments uniformly. For "dense" datasets (2.5K segments), every possible pairwise comparison between sampled segments is labeled, producing roughly $\binom{2500}{2} \approx 3.1$ million comparisons. For "sparse" datasets (20K segments), only one comparison is labeled for every two segments, producing 10K comparisons. The PPO baseline receives 3.84 million additional online state-action pairs collected during RL training—the paper explicitly notes this is "not a fair comparison" and delineates PPO results with a dashed line in Table 1.

---

Summary of Design Choices and Their Justifications

Regret preference model over partial-return model: the regret model distinguishes between behaviors that make progress toward a goal versus those that do not, even when both fail to achieve reward. This aligns with psychological evidence (Knox et al., 2022) that humans evaluate optimality, not just reward accumulation.

Max-entropy bijection instead of direct advantage learning: enforcing the normalization constraint $\int_\mathcal{A} e^{A(s,a)/\alpha} \, da = 1$ over continuous action spaces is intractable with neural networks. Learning a policy directly guarantees normalization by construction, trading an intractable constrained optimization for a natural unconstrained one.

Contrastive logistic loss over regression or RL: the logistic loss is the maximum-likelihood objective for the Bernoulli preference distribution. Unlike reward regression, it does not require absolute reward values (only relative comparisons). Unlike RL, it does not require bootstrapping, temporal credit assignment, or policy gradients—the gradient flows directly from the preference comparison through the log-probability to the policy's output.

Bias regularizer ($\lambda = 0.5$) over BC regularization (CPL-BC): the bias regularizer breaks ties in the logistic loss in favor of higher-probability (in-distribution) actions without requiring a separately trained reference policy. The paper's experiments suggest it performs more robustly than explicit BC regularization (CPL-BC) in the offline setting.

Fixed $\gamma = 1$ for CPL: the paper uses undiscounted accumulation for CPL's loss function, unlike P-IQL which uses $\gamma = 0.99$. This choice means CPL treats all steps in a segment equally rather than discounting later steps. The rationale is not explicitly justified, but it may reflect that discounting in the regret model already encodes temporal preference structure, and additional discounting in the loss would double-count this effect.

Gaussian policy with fixed variance for MetaWorld: this simplifies the log-probability computation to a negative squared error, making the CPL loss particularly simple to implement. The paper does not ablate this choice versus learned variance, but for continuous control tasks with continuous actions, the Gaussian assumption is standard (Haarnoja et al., 2018).

4. Key Insights and Innovations

Innovation 1: The Regret Preference Model Eliminates Reinforcement Learning by Providing Direct Information About the Optimal Policy

The paper's signature conceptual move is recognizing that switching from the standard partial-return preference model to the regret-based preference model changes what the learning problem requires. Under the partial-return model, preferences provide information about the reward function $r_E$, and recovering the optimal policy requires solving an RL problem: maximize the learned reward. Under the regret model, preferences provide information about the optimal advantage function $A^*$, which—under the max-entropy framework—is precisely the log-probability of the optimal policy. The consequence is that learning from preferences becomes a representation learning problem (learn a policy whose log-probabilities match the relative advantage structure implied by preferences) rather than a control problem (learn a reward and then optimize it).

This is a fundamental reframing, not an incremental improvement. Prior RLHF methods, whether using PPO (Ouyang et al., 2022; Christiano et al., 2017), IQL (Hejna & Sadigh, 2023), or the successor-feature regret methods of Knox et al. (2022, 2023), all retain a separation between what is learned from preferences and how a policy is extracted from that learned quantity. The former always learns a scalar function (reward or advantage), and the latter always requires some form of optimization or amortized inference—RL, policy distillation, or successor feature decomposition. CPL collapses this separation by learning the policy directly as the primary representational object. The learned policy is simultaneously the model of preferences and the behavioral output; there is no intermediate function to estimate and no downstream optimization to perform.

This reframing matters beyond computational convenience. It resolves a conceptual tension in prior work: the partial-return model says "humans judge total reward," but the downstream RL phase says "maximize total reward"—these are consistent, but only by inheriting all the difficulty of RL. The regret model says "humans judge optimality," and CPL says "then learn optimality directly from their judgments." The learning objective and the behavioral objective become the same thing. This intellectual move—use the preference model to determine what representation is needed, then design a loss function whose optimum is that representation—is the paper's deepest contribution and is likely to influence how future work thinks about aligning models from comparative feedback more broadly than the specific CPL algorithm.

The evidence that this reframing produces a qualitatively different learning algorithm is in the architecture: CPL requires only a policy network (2.1M parameters for image-based tasks) versus P-IQL's policy network, Q-network, value network, and reward network (9.6M parameters total; Table 2). The 4× parameter reduction is not just an engineering detail—it reflects the conceptual compression that happens when the intermediate reward representation is eliminated.

Innovation 2: Conservative Regularization as a Solution to the Finite-Data Non-Uniqueness Problem Specific to Contrastive Policy Objectives

The paper identifies and addresses a subtle pathology that arises specifically when trying to learn policies from pairwise preference data using contrastive objectives. Because the CPL loss depends only on the difference in log-probability sums between preferred and unpreferred segments, policies that differ by a constant shift in log-probability on all actions—or more generally, by any vector in the null space of the comparison matrix—produce identical losses on a finite dataset (Section 3.4, Appendix A.3). In the infinite-data limit, Theorem 1 guarantees uniqueness, but in practice, $|S \times A| \gg n$, and the null space is large. This means the CPL loss alone does not distinguish between a policy that concentrates probability on in-distribution actions versus one that places high probability on out-of-distribution actions that never appeared in any comparison.

This problem is specific to contrastive policy objectives. Standard two-phase RLHF avoids it because the reward learning phase estimates a single scalar function from preferences (which does not have the same null-space issue—the reward model sees individual state-action pairs, not whole-segment contrasts), and the downstream RL phase adds its own implicit regularization through the value function's Bellman consistency constraints and the offline RL algorithm's conservatism mechanisms (e.g., IQL's expectile regression). DPO (Rafailov et al., 2023) avoids it in the bandit setting because all preferences share the same starting state, so the $V^*(s)$ term cancels and the comparison reduces to a per-action reward difference. CPL, by operating on segments of arbitrary length from different starting states, cannot rely on this cancellation and must explicitly address the non-uniqueness.

The paper's solution—the bias regularizer with $\lambda \in (0, 1)$ (Eq. 6)—is not a generic regularization technique but a principled response to this specific structural issue. Proposition 2 proves that among policies with equal standard CPL loss, the regularizer prefers those with higher absolute log-probability on the data distribution, effectively encoding the pessimism principle (prefer in-distribution solutions when the data underdetermines the optimal policy) directly into the loss function rather than through a separate mechanism. The ablation in Figure 2 (right) shows that CPL is relatively robust to the choice of $\lambda$, but that different values do affect final performance—confirming that regularization matters but is not brittle.

This contribution is incremental in the sense that it adapts the conservative regularizer from An et al. (2023), but fundamental in that it diagnoses a failure mode that is specific to contrastive policy learning from segments and would not arise in either reward-model RLHF or bandit DPO. The formalization of the null-space problem through the comparison matrix $X$ (Appendix A.3) provides a diagnostic tool for understanding when and why contrastive policy objectives might fail with insufficient data, which is likely to be useful as these methods are applied to increasingly complex domains with sparse preference coverage.

Innovation 3: Dense Comparative Feedback as a Key Scaling Axis for Contrastive Policy Learning

The paper's experiments reveal a pattern that is both empirically striking and conceptually important: CPL benefits substantially more from increased comparison density than the RL-based baseline P-IQL. The data scaling experiments in Figure 7 (Appendix C.3) show that as the number of paired comparisons per segment increases (keeping the number of segments fixed at 5,000), CPL's performance improves monotonically across five of six tasks, while P-IQL's performance sometimes degrades. The paper attributes P-IQL's degradation to reward model underfitting—with more comparisons, the reward model's training loss no longer approaches zero, suggesting it cannot fully absorb the additional preference signal (Appendix C.3, Figure 8).

This finding reorients how one should think about data efficiency in preference-based learning. The conventional wisdom in RLHF—implicit in methods that sample a single comparison per pair of segments (Christiano et al., 2017; Ouyang et al., 2022)—is that more segments (state-action coverage) are the primary driver of performance. CPL's results suggest that more comparisons (preference signal density) matter at least as much, and that contrastive objectives are particularly well-suited to exploit dense comparative structure. This makes intuitive sense: contrastive learning methods in computer vision (Chen et al., 2020; He et al., 2020) are known to benefit from larger numbers of negative examples, and CPL's objective—which is essentially Noise Contrastive Estimation where each unpreferred segment serves as a negative—inherits this property. The connection the paper draws to InfoNCE (Oord et al., 2018) for ranking data (Appendix A.5) reinforces this: more comparisons per batch provide richer contrastive signal.

The practical implication is significant for how preference data should be collected. If CPL is the learning algorithm, it may be more valuable to elicit dense rankings over a smaller number of behavior segments than sparse pairwise comparisons over many segments. This shifts the emphasis from coverage (sampling diverse state-action pairs) to contrast (ensuring that each segment is compared against many others, providing a rich relative ordering). The paper does not fully explore this tradeoff—the dense-vs-sparse comparison in Table 1 confounds segment count with comparison density—but the direction is clearly indicated. In the extreme, CPL with dense comparisons on only 2,500 segments (row 1 of Table 1) often outperforms CPL with sparse comparisons on 20,000 segments (row 3), despite having 8× fewer state-action pairs. This is a substantial reversal of the usual "more data is better" assumption and suggests that preference structure can substitute for preference volume when the learning algorithm can exploit it.

Innovation 4: Verifier-Free RLHF That Scales to High-Dimensional Sequential Control Without the Contextual Bandit Approximation

The paper's empirical contribution is demonstrating that an RL-free, fully off-policy method for learning from preferences can operate on temporally extended manipulation tasks with high-dimensional image observations—a regime where prior RLHF methods have either failed or not been attempted. This is not merely a "CPL works" result but a proof of concept that the RLHF problem in its full MDP generality does not require RL.

The significance of this demonstration becomes clear when placed in context. Contemporary RLHF for language models (Ouyang et al., 2022; Ziegler et al., 2019; Rafailov et al., 2023) operates in the contextual bandit setting, where the sequential structure of interaction is flattened into single-step reward feedback. This is widely acknowledged to be an approximation—multi-turn dialogue is inherently sequential—but the approximation is tolerated because RL algorithms that handle sequential credit assignment (like PPO) are difficult to tune at scale, as the paper's own PPO results illustrate: "PPO was very sensitive to the KL-constraint coefficient on reward, which makes it difficult to tune" (Section 4.1), and its performance on MetaWorld, despite 3.84 million additional online transitions, is inconsistent and often below CPL (Table 1). On the robotics side, prior RLHF methods (Christiano et al., 2017; Lee et al., 2021; Hejna & Sadigh, 2023) have been limited to state-based control, where the observation space is low-dimensional and the representation learning burden is minimal.

CPL breaks through both constraints simultaneously. On image-based MetaWorld tasks with 64×64 pixel observations (Table 1, rows 2 and 4), CPL learns effective policies using only offline sub-optimal data and a CNN-based DrQv2 architecture. It requires no online interaction, no value function, no Q-function, and no reward model. The comparison to P-IQL is instructive: P-IQL also works on images and sometimes outperforms CPL (e.g., Bin Picking with dense preferences: 83.7% vs. 80.0%), but it does so at the cost of learning three additional neural networks and taking 1.6× longer per training step (Table 2). The paper does not claim CPL is strictly better than P-IQL on images—the results are mixed—but rather that CPL achieves comparable or better performance with dramatically less machinery.

This matters because it expands the design space for RLHF systems. Prior to CPL, the choice was: (a) use RL-based methods, accept their complexity and tuning difficulty, and restrict the problem setting to make them work (bandits or low-dimensional states), or (b) use supervised methods like DPO, but restrict to the bandit setting. CPL demonstrates a third path: supervised simplicity with full MDP generality. The cost—and the paper is clear about this—is that CPL requires segment-level preferences with known temporal discounting $\gamma$, and benefits most when comparisons are dense. But in domains where these conditions can be met (robotics with real-time preference elicitation, or LLM fine-tuning on multi-turn conversation data if such datasets existed), CPL offers a qualitatively simpler scaling path than RL-based alternatives.

The paper's negative result on PPO is also significant here. The fact that PPO—the algorithm used to train ChatGPT—fails to consistently outperform CPL on MetaWorld despite access to 3.84 million online transitions (Table 1, rows 1 and 3) underscores how brittle RL can be in the continuous control setting. The paper attributes this to sensitivity of the KL-constrained reward and high variance of policy gradients with longer horizons. This is not a theoretical argument against RL but an empirical demonstration that, for certain practically important problem classes, avoiding RL is not just conceptually elegant but performance-competitive or superior.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary experiments use six manipulation tasks from the MetaWorld robotics benchmark (Yu et al., 2020): Bin Picking, Button Press, Door Open, Drawer Open, Plate Slide, and Sweep Into. These environments are modified from the standard MetaWorld v2 release: the goal is made observable (rather than hidden, as in meta-RL variants), proprioceptive history is removed to improve Markovianity, and the initial arm position is randomized (Appendix D.1). For each task, 2,500 sub-optimal rollout episodes are collected from a stochastic SAC policy checkpoint achieving approximately 50% success rate (exact rates in Table 4, ranging from 15.56% for Button Press to 60.12% for Drawer Open). Synthetic regret-based preference labels are generated using the Q-function and policy of an oracle SAC model trained to 100% success on a combination of the sub-optimal rollout data and online data (Appendix D.2). Two dataset configurations are considered: dense (2.5K segments, all pairwise comparisons labeled, producing ~3.1 million comparisons) and sparse (20K segments, one comparison per pair, producing 10K comparisons). Segment length is fixed at 64 time steps. For human-preference experiments, the paper uses four locomotion datasets from the D4RL benchmark (Fu et al., 2020)—Walker2D-Medium-Expert, Walker2D-Medium-Replay, Hopper-Medium-Expert, Hopper-Medium-Replay—with real human preferences from Kim et al. (2023), comprising either 100 (expert) or 500 (replay) human queries.

• Base model(s). All MetaWorld policies are stochastic Gaussian policies with a fixed variance (state) or learned diagonal Gaussian (D4RL). For state-based experiments, policies use a 2-layer MLP with hidden size 512 and dropout 0.25. For image-based experiments (64×64 pixel observations), the DrQv2 architecture from Yarats et al. (2022) is used with dropout 0.5 and random shift augmentations following Laskin et al. (2020). For D4RL experiments, the network architecture from Kim et al. (2023) is adopted. The paper does not compare across model scales—all experiments use a single architecture size—so there is no "base model family" in the sense of PaLM or GPT variants.

• Metrics. The primary metric is success rate (in percent), defined as the fraction of evaluation episodes where the agent achieves the task goal (e.g., grasping and depositing the object for Bin Picking). For MetaWorld, evaluation runs 25 episodes every 5,000 (state) or 2,500 (image) training steps. The reported number is the maximum of the running average across eight neighboring checkpoints (200 episodes total), averaged over seeds, following a maximum-of-the-average protocol that the paper positions as "a middle-of-the-road approach" between the over-optimistic per-seed maximum used in imitation learning (Mandlekar et al., 2021) and the fixed-training-step reporting common in offline RL (Kostrikov et al., 2022). For D4RL, evaluation follows Kim et al. (2023): 10 episodes every 5,000 steps, with results reported as normalized scores (where 100 corresponds to an expert policy).

• Baselines. Four baselines are evaluated, plus two oracle references:

  1. Supervised Fine-Tuning (SFT): Behavior cloning on all segments in the dataset, followed by additional BC fine-tuning on only the preferred segments ($\sigma^+$) from $\mathcal{D}_{\text{pref}}$.
  2. Preference IQL (P-IQL): Learns a reward function from $\mathcal{D}_{\text{pref}}$ assuming the partial-return preference model, then optimizes it using Implicit Q-Learning (Kostrikov et al., 2022), a state-of-the-art offline RL algorithm. The paper notes that P-IQL in this configuration is "equivalent to TREX on pairwise preferences" (Brown et al., 2019). This baseline uses the implementation from Hejna & Sadigh (2023) and learns a reward network, a Q-network, a value network, and a policy network.
  3. PPO with KL-constrained reward: Uses proximal policy optimization to maximize a learned reward function with a KL penalty toward a reference policy (Ziegler et al., 2019; Ouyang et al., 2022). This baseline receives 3.84 million additional online state-action pairs collected during RL training—25× more data than CPL 2.5K Dense and 4× more than CPL 20K Sparse—and is delineated with a dashed line in results tables to emphasize it is "not a fair comparison" (Section 4). The paper notes PPO was "very sensitive to the KL-constraint coefficient" and required bounding reward values between 0 and 1 for stability (Table 8, Appendix D.4).
  4. %BC (Oracle): Behavior cloning on only the top X% of rollouts according to the ground-truth reward $r_E$. This is an oracle baseline since $r_E$ is not available during RLHF.
  5. Preference Transformer (PT): For D4RL human-preference experiments only, the method from Kim et al. (2023), which learns a transformer-based reward model and then uses IQL for policy optimization.

• Generation budget / compute accounting. Since CPL is a pure offline method that does not interact with the environment, the "compute budget" is not measured in environment steps as in standard RL. Instead, fairness is ensured by fixing the dataset and labeling budget across methods. All methods receive the same offline rollouts and the same preference comparisons. The paper reports parameter counts and wall-clock training time as the relevant efficiency metrics. For the PPO baseline, the online interaction budget (3.84M additional transitions) is reported separately and explicitly noted as additional data. For computational efficiency, Table 2 reports that CPL trains in 10.2 hours with 2.1M parameters versus P-IQL's 16.5 hours with 9.6M parameters on a single TitanRTX GPU for image-based tasks—a 1.62× speedup with less than a quarter of the parameters.

• Cross-validation / statistical protocol. Results are reported over four seeds for state-based experiments and three seeds for image-based experiments, with standard error or standard deviation (the paper reports $\pm$ values in Table 1 without specifying whether these are standard deviation or standard error, though the context and learning curves in Appendix C are averaged across seeds). For D4RL experiments, results follow the protocol from Kim et al. (2023). Hyperparameters are tuned on the 10K comparison (sparse) dataset configuration, then held fixed for data scaling experiments, which the paper notes is a limitation: "P-IQL ... consequently more sensitive to changes in the dataset. We tuned hyperparameters for all methods with 10K comparison, then left them the same for scaling experiments" (Section 4.2).

---

Main Quantitative Results

State-Based MetaWorld with Dense Preferences (2.5K Segments)

The headline results for the strongest signal regime are in Table 1, row 1. CPL achieves the best or statistically tied-for-best performance on 5 of 6 tasks, with success rates as follows (CPL vs. best baseline in parentheses):

CPL's margin over P-IQL is substantial in Button Press (8.3 percentage points), Drawer Open (12.5 points), and Sweep Into (9.8 points). PPO beats CPL in Bin Picking (83.7 vs. 80.0) but requires 25× more data. CPL consistently outperforms %BC (bottom of Table 1), with 10% BC achieving only 62.6% on Bin Picking and 5% BC achieving 64.6%—well below CPL's 80.0%—indicating that CPL is not merely selecting the best segments from the data but is exhibiting policy improvement beyond imitation of the top fraction of demonstrations.

Key observations from the learning curves in Figure 3 (Appendix C.1): CPL's performance rises sharply after the BC pretraining phase (dotted vertical line at 200K steps) and continues to improve through 500K total steps. P-IQL learns more gradually and sometimes plateaus earlier. SFT's performance is consistently lower, often declining after the pretraining phase as it overfits to the preferred segments.

State-Based MetaWorld with Sparse Preferences (20K Segments, 10K Comparisons)

Table 1, row 3 shows results in the sparser regime. CPL outperforms baselines in 3 of 6 tasks and is statistically tied in the remaining 3, with a particularly large margin in Bin Picking (83.2 vs. P-IQL's 75.0 and SFT's 67.0) and Sweep Into (81.2 vs. P-IQL's 73.4). The task-by-task breakdown:

Comparing dense (row 1) to sparse (row 3) within CPL: dense preferences generally yield higher or comparable performance despite having 8× fewer state-action pairs (2.5K vs. 20K segments). This is the paper's central empirical finding about preference density—more comparisons per segment can compensate for fewer segments. However, the effect is not uniform: on some tasks (Button Press, Drawer Open), CPL performs slightly better with sparse data (29.8 vs. 24.5 for Button Press; 79.1 vs. 83.6 for Drawer Open), though confidence intervals overlap.

PPO's performance with sparse comparisons (row 3) is more competitive than with dense comparisons (row 1), achieving top marks on Door Open (82.8) and Plate Slide (60.7). The paper does not explain why PPO benefits from sparser data, though presumably fewer comparisons mean a simpler reward model with less underfitting—consistent with the observation that "P-IQL's performance sometimes goes down" with more comparisons due to "reward underfitting" (Appendix C.3).

Image-Based MetaWorld Experiments

Table 1, row 2 (2.5K dense) and row 4 (20K sparse) report results with 64×64 image observations, using DrQv2 architectures and random shift augmentations. With dense preferences, CPL outperforms P-IQL in 4 of 6 tasks and ties on Sweep Into. With sparse preferences, the two methods are comparable, each winning on approximately half the tasks.

Dense image results (row 2):

Sparse image results (row 4):

There are three notable trends in the image results. First, data augmentation benefits P-IQL disproportionately: P-IQL's image performance is markedly higher than its state performance in several tasks (e.g., Bin Picking: 83.7 image vs. 70.6 state on dense; Sweep Into: 67.7 image vs. 60.6 state on dense), while CPL's is comparable or slightly lower. The paper attributes this to "data-augmentation, which is inapplicable in state, plays a key role in improving value representation for P-IQL" (Section 4.1). Second, the gap between CPL and P-IQL narrows in the sparse regime—the contrastive objective's advantage diminishes with fewer comparisons per segment. Third, both methods generally improve from state to image on some tasks (Button Press: CPL 24.5 state → 27.5 image on dense), which the paper does not explain but may reflect the richer visual features learned by the DrQv2 CNN encoder.

The computational efficiency advantage of CPL is most stark in the image setting: CPL trains in 10.2 hours with 2.1M parameters versus P-IQL's 16.5 hours with 9.6M parameters (Table 2). The 1.62× speedup and 4.6× parameter reduction derive from CPL's elimination of the reward network, Q-network, and value network—P-IQL must learn all four components.

Data Scaling: Comparisons Per Segment and Dataset Size

Figure 2 (left) and Figure 8 (Appendix C.3) show CPL and P-IQL's performance as the number of comparisons per segment increases from 2 to 16, holding the dataset fixed at 5,000 segments. CPL's performance improves monotonically with comparison density on 5 of 6 tasks (Bin Picking, Button Press, Door Open, Drawer Open, Sweep Into), while P-IQL's performance sometimes degrades (Door Open, Sweep Into) or saturates early. Plate Slide is the exception, where both methods are relatively flat. The paper posits that P-IQL's degradation stems from reward model underfitting: "Inspecting our training logs reveals that this is likely due to the reward function underfitting. For example, on Door Open, the reward modeling loss is near zero with only 2 comparisons per segment (10K comparisons total ...). With 16 comparisons per segment, the loss ends near 0.16" (Appendix C.3).

Figure 7 (Appendix C.3) varies the total number of segments (1,000 to 7,500) while keeping comparisons dense. Both CPL and P-IQL generally improve with more segments, but CPL benefits more proportionally. On Drawer Open, CPL goes from ~0.80 at 1K segments to ~0.85 at 7.5K; P-IQL goes from ~0.72 to ~0.82. On Bin Picking, the trends are similar in absolute terms but CPL maintains a consistent ~5-point advantage.

Together, these scaling experiments establish that CPL benefits more from increasing comparison density than from increasing state-action coverage, while P-IQL's bottleneck appears to be reward model capacity rather than data quantity. This is a practically significant finding for data collection: when annotating preferences for CPL, it may be more valuable to elicit many comparisons over a moderate number of segments than to collect many segments with sparse labels.

Segment Size Ablation

Figure 9 (Appendix C.4) varies segment length from 8 to 64 on Drawer Open with 20K sparse segments. CPL's performance increases with segment length, from ~0.75 at segment size 8 to ~0.79 at segment size 64. P-IQL and SFT are less sensitive to segment size. The paper does not explore segment sizes beyond 64, nor does it discuss the memory implications of larger segments (each comparison stores $2 \times k$ state-action pairs in GPU memory, which "requires a substantial amount of GPU memory for large segment sizes"—Section 6, Limitations).

Real Human Preferences on D4RL

Table 3 reports results with 100 (Medium-Expert) or 500 (Medium-Replay) real human preferences from Kim et al. (2023). CPL achieves the best performance on 3 of 4 tasks, with particularly large margins on Hopper-Medium-Expert (109.1 vs. PT's 86.7 and P-IQL's 88.6):

CPL fails on Walker-Medium-Replay (48.3 vs. PT's 76.6 and P-IQL's 71.6), a 23–28 point gap. The paper speculates that "preferences for this dataset may not closely follow the regret-based model as per discussions with the authors of Kim et al. (2023) they were collected by a single user with a pre-planned rules-based approach" (Section 4.3). This is a notable negative result: CPL's performance depends on the regret model accurately describing how preferences were generated. When humans provide preferences according to a different cognitive model (e.g., reward-based or rule-based), CPL's inductive bias becomes a liability rather than a strength.

For these experiments, CPL was modified to handle the extremely limited number of queries (100–500) by first training a logistic regression model to predict the user's preference $P[\sigma^+ \succ \sigma^-]$ from full segments, then using this model to relabel the offline D4RL data with dense preferences for CPL's contrastive objective. The segment-level preference predictor uses the transformer architecture from Kim et al. (2023) and is not a reward or advantage function. No BC pretraining was used for D4RL experiments. Learning curves are in Figure 11 (Appendix C.5).

---

Ablation Studies and Robustness Checks

Temperature $\alpha$ (Figure 2, right): On Drawer Open state, $\alpha = 0.1$ performs best (reaching ~0.89 success), while $\alpha = 0.02$ (colder) plateaus at ~0.82 and $\alpha = 0.5$ (warmer) at ~0.79. The authors note that "higher performance could have been attained with further hyper-parameter tuning" and did not perform a systematic per-environment sweep.

Bias regularizer $\lambda$ (Figure 2, right): $\lambda = 0.5$ reaches ~0.89 on Drawer Open, while $\lambda = 0.25$ and $\lambda = 0.75$ both plateau around ~0.82–0.84. The ordering is not monotonic—$\lambda = 0.25$ initially learns faster but saturates lower. The paper used $\lambda = 0.5$ uniformly across all MetaWorld experiments and $\lambda = 0.5$ for D4RL as well (Table 6).

Pretraining with BC (Figure 10, Appendix C.5): Removing BC pretraining from CPL improves performance on Bin Picking (reaching ~0.85 without vs. ~0.80 with) and Button Press, while maintaining or slightly reducing performance on the other four tasks. The paper notes this mixed effect and hypothesizes that pretraining helps "when a policy closer to the data distribution is desirable," but may slow the contrastive phase by biasing the policy toward sub-optimal actions.

CPL variants (Figures 3–6, Appendix C.1–C.2): The BC-regularized variant (CPL-BC) and the KL-constrained variant (CPL-KL) are evaluated alongside the main bias-regularized CPL in the learning curve figures. CPL-BC is consistently the weakest variant, often failing to surpass SFT. CPL-KL is competitive on some tasks (e.g., state-based Door Open with sparse data, Figure 4) but generally underperforms bias-regularized CPL. The paper does not report final numbers for these variants in Table 1, indicating they were not primary baselines.

Conservative regularizer choice (Appendix B): In addition to the bias regularizer, the paper tested a BC-constrained Lagrangian relaxation (CPL-BC) and a KL-constrained variant (CPL-KL), as described in Section 3.4. The learning curves (Appendix C) consistently show CPL (bias) outperforming CPL-BC, with CPL-KL intermediate. The paper selected the bias regularizer as the primary variant based on these results, attributing its effectiveness to Proposition 2's theoretical justification.

Data augmentation for images: Random shift augmentations (Laskin et al., 2020) are used for all image-based experiments. The paper does not ablate this choice, but notes that augmentation "drastically improve[s] the performance of RL from images" and significantly benefits P-IQL's value representation learning (Section 4.1). CPL's performance on images—where it cannot leverage value-based improvements from augmentation—is thus a conservative test relative to methods that can.

Transitive augmentation for dense CPL (Appendix B): For image-based dense experiments, CPL uses an augmented loss that exploits transitivity: all $\binom{b}{2}$ pairwise comparisons are computed within each batch of $b$ segments. The paper reports this "lead to a slight increase in performance for some tasks" but does not quantify the improvement or evaluate the interaction with segment size and GPU memory constraints.

Preference predictor for limited human data (Section 4.3, Appendix C.5): In D4RL experiments with only 100–500 human preferences, CPL first trains a segment-level preference predictor (a logistic regression or transformer model) to predict $P[\sigma^+ \succ \sigma^-]$, then uses it to relabel the offline data with dense synthetic preferences. This is a significant architectural addition not used in the MetaWorld experiments and represents a form of semi-supervised learning—the human labels supervise the preference predictor, and the predictor's output labels supervise CPL. The paper does not ablate this choice or compare it to directly training CPL on the limited human queries.

---

Critical Assessment

Claim 1 (from executive summary): CPL matches or exceeds PPO and Preference IQL baselines while being 1.6× faster and using less than a quarter of the parameters.

The speed and parameter efficiency claims are well-supported by Table 2 (10.2 hrs, 2.1M params for CPL vs. 16.5 hrs, 9.6M params for P-IQL on images). The performance claim holds but with important qualifiers: (a) CPL outperforms P-IQL decisively on state-based dense data (5 of 6 tasks, Table 1 row 1) but the gap narrows on sparse data (competitive on state, mixed on images); (b) the comparison to PPO is asymmetric because PPO receives 3.84M online transitions, and even with that advantage, CPL matches or exceeds PPO on most tasks—but this is a demonstration that PPO underperforms in this setting, not that CPL is universally better than RL. The paper does not evaluate CPL against a well-tuned online RLHF method with matched online data, which would be a stronger test.

Claim 2: CPL scales to high-dimensional control problems and larger networks.

The image-based MetaWorld results (Table 1, rows 2 and 4) support this claim in the sense that CPL successfully learns manipulation policies from 64×64 pixel observations using a CNN policy network. However, "scales" is a strong word for experiments on six tasks with a single architecture (DrQv2). The paper does not evaluate CPL on larger image sizes, more complex visual domains, or bigger networks that would stress-test the supervised objective's memory and optimization properties. The acknowledgment that CPL's loss function "requires a substantial amount of GPU memory for large segment sizes" (Section 6) is relevant: scaling to longer segments or larger batches on high-resolution images may be constrained by GPU memory in ways not tested. The 1.62× speedup over P-IQL is measured at a relatively small scale (DrQv2, batch size 48, single GPU). The paper's claim that "as networks get larger and larger, the performance gain from using CPL would only increase" is extrapolation without evidence.

Claim 3: CPL benefits more from dense preferences than RL-based methods, suggesting contrastive methods are particularly suited to dense comparative feedback.

This is the paper's most robust and novel empirical finding. The monotonic improvement of CPL with comparison density (Figure 8, Appendix C.3) versus P-IQL's degradation is striking and well-documented across five of six tasks. However, the causal mechanism is incompletely characterized. The paper attributes P-IQL's degradation to reward model underfitting, which is plausible but not directly tested—no experiment varies reward model capacity to see if larger reward models can absorb denser comparisons. An ablation training P-IQL's reward model for more steps or with larger networks on dense data would distinguish between "reward learning fundamentally struggles with dense comparisons" and "reward learning needs more capacity to handle dense comparisons." The former would be a strong argument for CPL; the latter would suggest P-IQL could be fixed with better engineering.

Claim 4: CPL works with real human preferences, not just synthetic regret-based labels.

Supported in part by the D4RL results (Table 3): CPL achieves top performance on 3 of 4 tasks. The failure on Walker-Medium-Replay (48.3 vs. PT's 76.6) is a critical negative result that limits the generality of the claim. The paper's speculation that this environment's preferences were generated differently (rules-based rather than regret-based) is a hypothesis, not a demonstrated fact, and was not verified experimentally. This is a significant weakness—CPL's theoretical guarantees depend on the regret preference model being correct, and if real human preferences systematically deviate from this model in certain contexts, CPL will underperform. A broader study across multiple human preference datasets with different collection protocols would be needed to characterize where the regret model holds empirically.

Missing experiments. Several experiments would strengthen the paper's claims but are absent:

1. No experiments with RLHF-trained reward model + CPL policy extraction. CPL's theoretical derivation assumes the regret preference model; what if preferences are actually generated by a partial-return model but CPL is applied anyway? This would test the paper's claim that the regret model is more accurate—if CPL works well even on reward-model preferences, the choice of preference model may matter less than the choice of optimization method.

2. No comparison to DPO on sequential tasks. Since DPO is a special case of CPL (Appendix A.6), a direct comparison on MetaWorld—DPO operating on single-step segment slices versus CPL on length-64 segments—would quantify the benefit of multi-step contrastive learning over bandit-style learning. This is a natural ablation that would directly test the paper's claim that sequential preferences are important.

3. No ablation of segment length $\times$ comparison density. The experiments vary either segment count (Figure 7) or comparisons per segment (Figure 8) independently but never jointly for a fixed total comparison budget. Is it better to have 1,000 segments with 16 comparisons each or 4,000 segments with 4 comparisons each? This trade-off is central to practical preference collection and is not addressed.

4. No difficulty analysis. The paper does not analyze whether CPL's performance varies with episode difficulty, success rate of the data-generating policy, or any other stratification variable. Understanding where CPL succeeds and fails—beyond task-level averages—would strengthen the practical guidance for deployment.

5. Confidence intervals on efficiency comparisons. Table 2 reports a single runtime number without variance across runs or seeds. The 1.62× speedup is measured at one training step budget on one GPU, and it is unclear whether this advantage holds across different hardware, batch sizes, or training durations.

Where claims hold conditionally. The paper's central selling point—that CPL eliminates RL while matching or exceeding RL-based methods—holds clearly when (a) preferences are dense, (b) preferences follow the regret model, and (c) the task involves sequential decision making with sub-optimal offline data. The advantages diminish when preferences are sparse (rows 3–4 vs. row 1 of Table 1), when data augmentation benefits value-based methods (image results), or when human preferences deviate from the regret model (Walker-Medium-Replay). The paper is generally transparent about these conditions but sometimes overstates the generality of the conclusions, particularly regarding scaling behavior beyond the tested domains and architectures.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Dominates the Headline Efficiency Gains

The assumption or constraint. CPL's contrastive objective requires preferences to be structured as pairwise comparisons between behavior segments, where each segment is a length-64 sequence of state-action pairs. The paper's primary experiments use synthetic "dense" datasets where all $\binom{2500}{2} \approx 3.1$ million pairwise comparisons among 2,500 segments are labeled—a regime the paper explicitly associates with CPL's strongest performance. As Section 4.2 notes:

"the gap in performance between CPL and baselines is higher for datasets with denser comparisons... underscoring the importance of informative negatives."

The experiments do not account for the cost of generating these dense annotations. In the MetaWorld experiments, preferences are synthetically labeled using an oracle SAC model that has already achieved 100% success on the task—a model whose training required access to both the sub-optimal rollout data and additional online environment interactions (Appendix D.2). The oracle SAC model is precisely the kind of capability CPL is intended to acquire without having.

The consequence. The headline results—CPL outperforming P-IQL on 5 of 6 tasks with 2.5K dense segments (Table 1, row 1), achieving 4× parameter efficiency (Table 2)—all assume access to a fully trained oracle policy that already solves the task. In any real deployment where the goal is to learn a policy from human feedback, no such oracle exists. Without it, generating dense preference labels would require humans to provide millions of pairwise comparisons across 2,500 length-64 trajectory segments—an annotation burden that is completely impractical. The 1.62× training speedup (Table 2) is computed after preferences are generated, ignoring the cost of acquiring them.

Conversely, if preferences are sparse (20K segments, 10K comparisons)—a more realistic annotation scenario—CPL's advantage over P-IQL narrows substantially. On state-based sparse data (Table 1, row 3), CPL outperforms or ties baselines but margins are smaller (e.g., Drawer Open: 79.1 vs. 76.2; Door Open: 77.9 vs. 82.8). On image-based sparse data (Table 1, row 4), CPL and P-IQL are essentially at parity, each winning on half the tasks. The paper's own scaling experiments (Figure 8, Appendix C.3) show that CPL's performance degrades substantially as comparisons-per-segment decreases: on Drawer Open with 5,000 segments, CPL drops from ~0.89 at 16 comparisons per segment to ~0.82 at 2 comparisons per segment. P-IQL is comparatively flat across this range. This means the $4\times$ parameter efficiency claim is measuring the cost of policy optimization only, not the total cost of preference acquisition + policy optimization—and the former dominates in the sparse regime where CPL's advantage is smallest.

What evidence exists in the paper. The gap between dense and sparse performance is directly visible in Table 1 and Figures 7–8. The paper does not report the computational or human-annotation cost of generating the dense preference datasets. The oracle SAC model used to label preferences is described in Appendix D.2 but its training cost (environment interactions, wall-clock time) is not quantified. The D4RL experiments (Section 4.3) partially address this by training a preference predictor on only 100–500 real human queries and then relabeling offline data with dense synthetic preferences—but this introduces an auxiliary model whose training and accuracy are not ablated, and it does not address the circularity of needing a strong preference predictor to achieve the dense regime where CPL excels.

Mitigation status. The paper acknowledges this limitation implicitly through its data scaling experiments (Section 4.2, Appendix C.3) but does not frame it as a limitation of the method. No proposal is made for reducing the comparison density requirement, no experiment tests CPL with budgeted annotation (e.g., active query selection), and no cost model is provided for trading off segment count versus comparisons per segment. The paper's suggestion to use transitive augmentation (computing all $\binom{b}{2}$ pairs within a batch; Appendix B) reduces GPU memory requirements but does not reduce the number of labeled comparisons needed in the dataset—those comparisons must still be generated and stored. The D4RL relabeling trick is practical but relies on the preference predictor generalizing correctly, and the paper provides no analysis of its error propagation into CPL's policy.

---

The Regret Preference Model Is an Assumption, Not a Verified Fact, and CPL Breaks When It Is Wrong

The assumption or constraint. CPL's theoretical guarantees—Theorem 1 (convergence to $\pi^*$ with unbounded data) and Proposition 1 (consistency)—both assume that human preferences are generated according to the regret-based Boltzmann model:

$P_{A^*}[\sigma^+ \succ \sigma^-] = \frac{\exp \sum_{\sigma^+} \gamma^t A^*(s_t, a_t)}{\exp \sum_{\sigma^+} \gamma^t A^*(s_t, a_t) + \exp \sum_{\sigma^-} \gamma^t A^*(s_t, a_t)}$

The paper cites Knox et al. (2022) as evidence that "humans instead provide preferences based on the regret of each behavior under the optimal policy" but acknowledges in Section 6: "no model of human behavior is perfect." The synthetic MetaWorld experiments label preferences using this exact model (via an oracle SAC model's Q-function and policy), meaning they test whether CPL can recover the optimal policy from data that matches its assumptions, not whether it works when human preferences follow a different generative process. There is no experiment where preferences are generated by a partial-return model but CPL is applied anyway—which would test robustness to model misspecification.

The consequence. When the regret model is wrong, CPL has no theoretical guarantees, and the paper provides direct empirical evidence of failure. On Walker-Medium-Replay from the D4RL benchmark (Table 3), CPL achieves 48.3 ± 3.7 normalized score compared to Preference Transformer's 76.6 ± 3.2 and P-IQL's 71.6 ± 5.9. This is a 23–28 percentage-point gap on a task where the baselines succeed. The paper attributes this to the preference collection process:

"preferences for this dataset may not closely follow the regret-based model as per discussions with the authors of Kim et al. (2023) they were collected by a single user with a pre-planned rules-based approach" (Section 4.3)

This admission is significant: it means CPL's performance is sensitive to how human preferences are elicited, not just to their quantity or density. A rules-based annotator (e.g., "prefer the segment that reaches further along the track") may generate preferences that are better modeled by a partial-return or heuristic model than by regret under an optimal policy. Since real-world RLHF deployments often use crowd workers with explicit annotation guidelines or automated quality checks, the gap between the regret model and actual human annotation behavior may be common.

What evidence exists in the paper. The Walker-Medium-Replay failure is the only direct test of model misspecification. The paper does not systematically vary the preference generation process to characterize CPL's robustness. No experiment compares CPL trained on regret-model preferences versus partial-return-model preferences on the same tasks to quantify the performance gap under misspecification. The three other D4RL tasks show CPL performing well (Hopper-Medium-Expert: 109.1 vs. PT's 86.7; Walker-Medium-Expert: 109.2 vs. PT's 110.2), but these successes do not establish that the regret model holds for those tasks—they only show that CPL performs well on them, which could be due to other factors (the preference predictor's accuracy, CPL's implicit regularization, or the regret model being approximately correct for those environments).

Mitigation status. None beyond the acknowledgment in Section 6. The paper does not propose a diagnostic for detecting when preference data violates the regret model, does not develop a robust variant of CPL that degrades gracefully under misspecification, and does not recommend preference elicitation protocols that encourage regret-consistent responses. The suggestion that "an online version of CPL could be developed that works with online human feedback" (Section 6) would not address the model misspecification problem—it would only change when and how preferences are collected, not their underlying cognitive model. A practitioner deploying CPL would need to independently verify that their human annotators' preferences align with the regret model, and the paper provides no tools for doing so.

---

Segment-Based Loss Requires Known Discount Factor and Incurs Memory Costs That May Limit Scaling

The assumption or constraint. CPL's loss function (Eq. 5) sums discounted log-probabilities along entire behavior segments of length $k$:

$\mathcal{L}_{\text{CPL}} = \mathbb{E}_{(\sigma^+, \sigma^-)} \left[-\log \frac{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t)}{\exp \sum_{\sigma^+} \gamma^t \alpha \log \pi_\theta(a^+_t|s^+_t) + \exp \sum_{\sigma^-} \gamma^t \alpha \log \pi_\theta(a^-_t|s^-_t)}\right]$

This requires two things that are not required by standard RLHF methods like PPO or DPO. First, the discount factor $\gamma$ must be known and must match the human's internal temporal discounting. The paper uses $\gamma = 1$ for all MetaWorld experiments (Table 6), meaning all time steps in a segment are weighted equally. Second, the loss function operates on full segments stored in GPU memory: each comparison in a batch stores $2k$ state-action pairs, and computing the log-probability sum requires a forward pass through the policy network for each of those $2k$ pairs. The paper acknowledges this explicitly in Section 6:

"as CPL's loss function is computed over segments, it requires a substantial amount of GPU memory for large segment sizes."

The consequence. For the discount factor: if the human's temporal preferences differ from $\gamma$, CPL's implied optimal policy will optimize the wrong objective. A human who strongly prefers immediate progress over delayed progress ($\gamma < 1$) would be modeled as indifferent to timing when $\gamma = 1$, causing CPL to potentially overweight late-segment actions relative to the human's true preferences. Conversely, using $\gamma < 1$ when the human evaluates all actions equally would cause early-segment actions to dominate the preference signal. The paper does not explore how sensitive CPL is to $\gamma$ misspecification—no experiment varies $\gamma$ to see if performance degrades. This is practically relevant because real humans cannot articulate their discount factor, and different annotators may have different implicit discounting.

For memory costs: the batch sizes used in the paper's experiments are already constrained by GPU memory—96 comparisons for state-based experiments, reduced to 48 for image-sparse and 32 for image-dense (Table 5). With segment length 64, a batch of 96 comparisons stores $96 \times 2 \times 64 = 12,288$ state-action pairs. For image-based experiments with 64×64 RGB observations, this is approximately $12,288 \times 3 \times 64^2 \approx 151$ million pixel values per batch—before considering the CNN encoder's intermediate activations. The reduction from batch size 96 (state) to 32 (image-dense) is a 3× decrease, directly attributable to memory constraints. Larger segment sizes—which Figure 9 (Appendix C.4) shows improve CPL performance—would further strain memory. Scaling CPL to longer-horizon tasks (hundreds or thousands of steps per segment), higher-resolution images, or larger policy networks would require either reducing batch size (which may hurt optimization stability and contrastive signal quality) or distributing across more GPUs (increasing cost).

What evidence exists in the paper. The memory constraint is visible in the batch size scaling across observation modalities (Table 5). The segment size ablation (Figure 9, Appendix C.4) tests sizes 8, 16, 32, and 64 on Drawer Open state, showing CPL improves with larger segments but does not test beyond 64—the practical ceiling on a single TitanRTX GPU with batch size 96. The $\gamma = 1$ choice is stated without justification (Table 6) and is not ablated. No experiment reports GPU memory usage, peak batch sizes, or wall-clock time as a function of segment length. The paper does not evaluate whether CPL's performance with segment-length 64 and batch-size 32 (image) is better or worse than with shorter segments and larger batches—a natural trade-off that a practitioner would need to resolve.

Mitigation status. The paper acknowledges both issues in Section 6 but proposes no solutions for the current method. For $\gamma$: "to relax the assumption that $\gamma$ is known, one might instead include it in the expressivity of CPL or other RLHF approaches." For memory: no mitigation is proposed. A practitioner would need to tune segment length and batch size within hardware constraints without guidance on the performance implications of that trade-off. Methods like gradient accumulation across smaller sub-batches, segment truncation with importance weighting, or off-policy segment replay (storing segments in CPU memory and streaming to GPU) could potentially address the memory constraint but are not explored.

---

No Online or Iterative Refinement: CPL Is Purely Offline and Cannot Improve Beyond Its Preference Data

The assumption or constraint. CPL is a strictly offline algorithm: it learns a policy from a fixed dataset of preferences and sub-optimal rollouts, with no mechanism for collecting new data, querying humans for additional preferences, or iteratively improving the policy through environment interaction. The paper's experiments reflect this: all MetaWorld results use a single static dataset of 2,500 rollouts from a ~50% success rate policy (Table 4), and all preferences are labeled once and never updated. The PPO baseline, by contrast, collects 3.84 million additional online state-action pairs (Appendix D.4). Section 6 notes this as a limitation:

"our work only considers offline data generated by suboptimal policies. An online version of CPL could be developed that works with online human feedback, allowing policies to continually improve."

The consequence. CPL's performance is fundamentally bounded by the quality and coverage of the offline dataset. If the dataset does not contain any trajectory that achieves the task—or contains only trajectories that fail in ways the preference labels cannot distinguish—CPL cannot discover the optimal policy, because it has no mechanism for generating novel behavior. This is a well-known fundamental limitation of offline RL (Levine et al., 2020) and applies to CPL as well.

The paper provides indirect evidence of this bound in the %BC comparison (Table 1, bottom) and the data scaling experiments (Figure 7, Appendix C.3). CPL does outperform behavior cloning on the top 5% and 10% of trajectories—meaning it is combining information across sub-optimal segments to produce a policy better than any single trajectory in the dataset. But the absolute performance ceiling is unclear. On Button Press, even CPL with dense preferences achieves only 24.5% success (Table 1, row 1)—the data-generating policy itself had a 15.56% success rate (Table 4), so CPL only modestly exceeds it. Whether additional offline data from better policies, or online interaction, could push CPL higher is not tested.

In the D4RL experiments, CPL uses a preference predictor trained on 100–500 human queries to relabel the offline D4RL dataset with dense preferences. This is a form of semi-supervised learning where the human labels provide weak supervision for the preference predictor, and the predictor's output provides dense supervision for CPL. But this pipeline is still fundamentally offline—the preference predictor does not actively query for labels on uncertain comparisons, and CPL does not request additional human feedback on states where its policy's implicit advantage estimates have high variance. This means that if the initial 100–500 human queries are insufficient to learn a good preference predictor (e.g., if they cover only a narrow region of behavior space), CPL's performance will suffer regardless of how much offline data is available for relabeling.

What evidence exists in the paper. The data scaling experiments (Figures 7–8, Appendix C.3) show that CPL's performance improves with more offline data and more comparisons, but no experiment tests whether it asymptotically approaches the optimal policy or plateaus at some sub-optimal level below 100%. The Button Press results (24.5% dense, 29.8% sparse, Table 1) are notably low across all methods, suggesting that the offline data may simply not contain information sufficient to learn a high-performing policy for that task. The paper does not report what the data-generating policy's success rate was on each task (Table 4 provides only aggregate rates), so it is not possible to assess how much CPL improves over the best trajectory in the dataset versus the data-generating policy's average. The gap between CPL and the oracle SAC policy (which achieves 100% by construction) is not quantified.

Mitigation status. The paper explicitly flags this as future work: "An online version of CPL could be developed that works with online human feedback, allowing policies to continually improve" (Section 6). No online variant is proposed or evaluated. Extending CPL to an online setting would require addressing several challenges not discussed: how to select which states or segments to query for human feedback (active preference elicitation), how to combine online queries with the offline contrastive objective without catastrophic forgetting or distribution shift, and how to trade off exploration (gathering new behavior segments) versus exploitation (refining the policy from existing preferences). These are non-trivial problems that the offline-only version of CPL sidesteps entirely.

---

Single Benchmark, Single Task Family, and No Demonstration on Language or Multi-Turn Dialogue

The assumption or constraint. All of CPL's experimental validation is conducted on continuous control tasks: six MetaWorld robotic manipulation tasks and four D4RL locomotion tasks. The paper's introduction and motivation (Section 1) frame RLHF primarily in the context of large language models, explicitly citing LLM fine-tuning (Ouyang et al., 2022), image generation (Lee et al., 2023), and robot policies (Christiano et al., 2017) as the application domains. The paper then claims CPL can be applied to "arbitrary MDPs" (Section 1) and that it "enables RLHF on multi-step dialogue" (Section 6, Future Directions). However, no experiment tests CPL on language generation, multi-turn conversation, code generation, or any discrete-action domain.

The consequence. The paper's claims about applicability to LLMs are entirely speculative. LLM fine-tuning with RLHF differs from MetaWorld robotics in several ways that may affect CPL's performance:

1. Discrete vs. continuous actions. MetaWorld uses continuous action spaces with Gaussian policies, where log-probabilities are well-defined and differentiable. LLMs use discrete token sequences where the policy is a categorical distribution over a vocabulary of tens of thousands of tokens. Computing $\log \pi_\theta(a_t|s_t)$ for each token in a sequence and summing discounted log-probabilities over potentially hundreds of tokens raises questions about numerical stability (summing many log-probabilities of individually small values), gradient flow through long sequences, and the implicit advantage representation (is $\alpha \log \pi(a|s)$ meaningful when $\pi(a|s)$ is a probability over a discrete vocabulary?).

2. Segment semantics. In MetaWorld, a "segment" is a contiguous 64-step trajectory fragment with clear physical meaning (e.g., approaching and grasping an object). In dialogue, a "segment" might span multiple turns of a conversation, and its boundaries are semantic rather than temporal—where does a "segment" begin and end? What is the equivalent of a 64-step segment when turns vary in length? The paper offers no guidance on segment construction for discrete or language domains.

3. Preference density. The paper's central finding—that CPL benefits from dense comparisons—is predicated on having many pairwise rankings among segments. In LLM RLHF, preferences are typically collected as pairwise comparisons of complete model responses to a single prompt (Ouyang et al., 2022), not as rankings among many trajectory fragments from the same starting state. The paper's D4RL relabeling trick (train a preference predictor, then label offline data densely) may not transfer: dialogue preferences are prompt-specific, and training a general preference predictor from limited human feedback across diverse prompts is significantly harder than predicting preferences among locomotion trajectories in a single MDP.

4. Reference policies and KL constraints. CPL does not incorporate a KL penalty toward a reference policy in its standard form. The KL-constrained variant (CPL-KL, Appendix B) is mentioned but is not the primary method evaluated. Contemporary LLM RLHF methods (Ziegler et al., 2019; Ouyang et al., 2022; Rafailov et al., 2023) universally include a KL penalty toward the pretrained model to prevent reward hacking and preserve general capabilities. Whether CPL without a KL penalty—or with the Appendix B KL variant—would prevent language model degeneration is completely untested.

What evidence exists in the paper. No language experiments exist. No discrete-action experiments exist. The paper does not discuss implementation considerations for discrete action spaces or sequence models. The closest the paper comes to language is the theoretical connection to DPO (Appendix A.6), which shows DPO is a special case of CPL when segments are length-1 and share the same starting state. But this connection is retrospective—it shows CPL subsumes DPO's bandit formulation but does not demonstrate that CPL's multi-step generalization adds value in language domains or even works in discrete action spaces.

Mitigation status. The paper acknowledges this implicitly by leaving LLM experiments as future work: "One potentially exciting application is LLMs, where CPL enables fine-tuning on multiple steps of turn-based dialogue. To our knowledge, no multi-step preferences dataset currently exists for LLMs" (Section 6). This is candid but also revealing: the lack of multi-step preference datasets for LLMs is precisely the gap that makes CPL's claimed applicability to language speculative. The paper does not create such a dataset, does not simulate one from existing single-turn preference data, and does not propose a method for constructing multi-step preferences from single-turn annotations. A practitioner interested in applying CPL to language would need to solve the dataset construction problem, the discrete-action implementation problem, and the KL-regularization problem—all without guidance from the paper's experiments.

The Test Set Sizes Are Small, and Strategy Selection Is Not Cross-Validated for the Compute-Optimal Policy

The assumption or constraint. The paper's main evaluation reports success rates on six MetaWorld tasks with 4 seeds (state) or 3 seeds (image), each evaluated over a 200-episode window from the maximum running-average checkpoint. D4RL experiments use 4 environments with a single evaluation protocol from Kim et al. (2023). The total number of independent evaluations per task-method pair is: 4 × 200 = 800 episodes for MetaWorld state, 3 × 200 = 600 episodes for MetaWorld image, and the D4RL protocol's episodes for D4RL.

The consequence. Success rates on six tasks with small seed counts provide limited statistical power for distinguishing between methods. Many of the comparisons in Table 1 show overlapping or nearly overlapping confidence intervals. For example, on state-based Door Open with dense preferences (Table 1, row 1): CPL achieves 80.0 ± 6.8, P-IQL 69.0 ± 6.2, and PPO 79.3 ± 1.2. The ±6.8 on CPL means its 95% confidence interval spans approximately [73.2, 86.8] if those are standard errors, or [66.4, 93.6] if standard deviations (the paper does not specify which). Either way, CPL's interval overlaps substantially with PPO's, making the claim that CPL "matches or exceeds" baselines less statistically robust than the point estimates suggest.

On the D4RL tasks, results are reported with only a single seed? The paper does not specify the number of seeds for D4RL—Table 3 reports means ± something (likely standard deviation across evaluation episodes, not seeds) and Figure 11's learning curves show single lines without error bands. The number of independent training runs is unclear. Walker-Medium-Replay, where CPL fails (48.3 vs. PT's 76.6), is reported with error bars of ±3.7 for CPL and ±3.2 for PT—if these are standard deviations across evaluation episodes from a single training run, the statistical significance of the gap is not established.

Additionally, the paper performs no cross-validation for hyperparameter selection across the MetaWorld tasks. Hyperparameters (α, λ, learning rate, pretraining steps, batch size, segment length) are tuned on the sparse 10K comparison dataset configuration and then applied to all experiments (Section 4.2). This means the hyperparameters are implicitly optimized on the test distribution—the evaluation tasks are the same six MetaWorld environments on which hyperparameters were selected. In the D4RL experiments, CPL's hyperparameters (α = 0.2, λ = 0.5; Table 6) are a single fixed setting applied to all four tasks. There is no held-out task for hyperparameter validation, and no sensitivity analysis across tasks to determine whether the chosen settings are near-optimal or merely not catastrophic.

What evidence exists in the paper. The learning curves (Figures 3–6, Appendix C) show seed-level variance across training, but the reported numbers in Table 1 aggregate across seeds and across an 8-checkpoint evaluation window. The paper describes the evaluation protocol as "a middle-of-the-road approach" (Appendix D.3) that balances overfitting concerns in supervised learning with the fixed-training-step protocol of offline RL. But this protocol is not standard and makes direct comparison to prior work difficult. The ± values in Table 1 are not explicitly defined as standard deviation or standard error of the mean. The D4RL results do not specify the number of training seeds. Hyperparameters were tuned on the same set of six tasks used for evaluation, with no held-out task for validation. The paper does not report statistical tests (t-test, bootstrap confidence intervals, or Mann-Whitney U) for any of the pairwise comparisons between CPL and baselines.

Mitigation status. None. The paper does not acknowledge the statistical power limitations of a 6-task, 4-seed evaluation design. No confidence intervals are provided for the efficiency comparisons (Table 2's 1.62× speedup). No sensitivity analysis is conducted for hyperparameters across tasks—hyperparameters are swept only on Drawer Open (Figure 2, right) and then applied uniformly. For a paper that claims to introduce a "new family of algorithms," the empirical validation is narrow in both task diversity and statistical rigor. A practitioner choosing between CPL and P-IQL for a new domain would need to perform their own hyperparameter tuning and statistical comparison, as the paper's results provide point estimates but not the variance characterization needed for robust method selection.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reorients the RLHF problem from a two-phase optimization pipeline—learn a reward function, then optimize it with RL—into a single supervised learning problem. This is not merely an algorithmic substitution but a conceptual reframing of what it means to learn from human preferences: the preference model itself determines what representation is needed, and by choosing the regret model, that representation becomes the policy directly. The intellectual consequence is that learning from comparative feedback is recast as a contrastive representation learning problem rather than a control problem. This places RLHF in the same algorithmic family as SimCLR (Chen et al., 2020), MoCo (He et al., 2020), and CLIP (Radford et al., 2021)—methods that have scaled to massive datasets and architectures with minimal optimization difficulty—rather than in the family of policy gradient and approximate dynamic programming methods, which have well-documented scaling challenges (Marbach & Tsitsiklis, 2003; Van Hasselt et al., 2018).

The magnitude of this shift is substantial but bounded. CPL does not render RL obsolete for preference-based learning—the paper's own results show that P-IQL matches CPL on image-based sparse tasks (Table 1, row 4) and that PPO with enough online data can outperform CPL on some environments (Bin Picking dense, Table 1 row 1). Rather, CPL expands the design space by demonstrating that RL is not necessary for general MDP RLHF, contradicting the implicit assumption—embedded in methods from Christiano et al. (2017) through Ouyang et al. (2022)—that reward learning plus policy optimization is the only viable path. The practical consequence is that researchers and practitioners now have a choice between two qualitatively different approaches (supervised contrastive vs. RL-based), each with different strengths: CPL scales with comparison density and requires less parameterization; RL-based methods scale with environment interaction and may be more robust to preference model misspecification.

The paper reconciles a latent tension in the RLHF literature between the bandit approximation (Ouyang et al., 2022; Rafailov et al., 2023) and the full MDP formulation (Christiano et al., 2017; Hejna & Sadigh, 2023). Prior to CPL, the RLHF community effectively accepted a tradeoff: use the full MDP but pay the price of RL's optimization challenges, or simplify to a bandit and enjoy supervised learning's stability. The paper's derivation in Appendix A.6—showing DPO is CPL with length-1 segments and shared starting states—unifies these regimes under a single framework. The implication is that the bandit case is not a separate problem requiring its own algorithms but rather the degenerate limit of a general segment-based formulation. This unification makes the research landscape more coherent: future work on sequential preferences can build on CPL's contrastive formulation without abandoning the theoretical and practical insights developed in the bandit DPO literature. The DPO-to-CPL correspondence in Appendix A.6 is not just a theoretical curiosity—it means that any improvement to CPL's multi-step contrastive objective has a natural bandit special case, and conversely, any failure mode identified in the bandit setting likely has a sequential analog that CPL's framework can diagnose.

The paper also establishes preference density as a first-class scaling axis. Prior RLHF work implicitly assumed that more segments (more state-action coverage) was the primary driver of performance—methods sampled one comparison per segment pair and focused on coverage of the state space. CPL's scaling experiments (Figure 8, Appendix C.3) show that comparison density can substitute for segment count: CPL with 2,500 dense segments often outperforms CPL with 20,000 sparse segments (Table 1, rows 1 vs. 3), and CPL's performance improves monotonically with comparisons-per-segment while P-IQL's sometimes degrades. This finding reorients how preference data should be collected and valued. A research program organized around eliciting rich relative orderings over smaller behavior sets may now be more productive than one focused on maximizing the number of preference-labeled segments. This has downstream implications for active preference learning (where queries should be designed to maximize contrastive informativeness rather than state-space coverage) and for annotation budget allocation (where dense rankings over a curated set may outperform sparse comparisons over a larger set at fixed cost).

One research direction that becomes less attractive is the development of ever-more-complex two-phase RLHF architectures that learn reward functions, Q-functions, value functions, and policies simultaneously. Table 2's efficiency comparison—CPL trains in 10.2 hours with 2.1M parameters versus P-IQL's 16.5 hours with 9.6M parameters—quantifies the overhead of learning multiple value-related components. Unless those components provide substantial performance benefits that CPL cannot match (which the paper's results do not demonstrate, except perhaps in the image-sparse regime or under preference model misspecification), the additional complexity is hard to justify. Similarly, the paper's PPO results—which required 3.84 million online transitions, extensive hyperparameter tuning, and still underperformed CPL on most tasks—suggest that scaling policy gradient methods to sequential RLHF is an uphill battle that may not be worth fighting when supervised alternatives exist. The research portfolio should shift toward improving the components CPL retains (the policy network, the contrastive objective, the preference model) rather than re-introducing the components it eliminates.

Follow-Up Research This Work Enables

Characterizing the regret model's empirical validity on real human preference data. The paper's most significant failure—CPL drops to 48.3 on Walker-Medium-Replay versus PT's 76.6 (Table 3)—is attributed to preferences that "may not closely follow the regret-based model" because they were "collected by a single user with a pre-planned rules-based approach." This is a hypothesis, not a verified fact. A direct follow-up would collect human preferences on a set of control or language tasks under two protocols: one designed to elicit regret-based judgments (e.g., asking annotators "which behavior is closer to optimal?" or "which would you prefer an expert to take?") and one designed to elicit reward-based judgments (e.g., "which behavior achieves more of the stated objective?"). Training CPL on both datasets and measuring the performance gap would (a) quantify CPL's sensitivity to preference elicitation protocol, (b) validate whether the regret model better describes human cognition when preferences are elicited appropriately, and (c) provide guidance for how to instruct annotators when CPL is the intended learning algorithm. If the gap is large—CPL succeeds under regret-elicited preferences but fails under reward-elicited ones—then preference elicitation protocol becomes a critical design choice in CPL deployments and should be standardized. If the gap is small, the regret-versus-reward model distinction matters less than CPL's contrastive optimization advantage, which would strengthen CPL's robustness claims.

Online CPL with active preference querying. The paper explicitly leaves online extension to future work (Section 6). A natural next step is to combine CPL with an uncertainty-aware query strategy: as the agent interacts with the environment, it generates candidate behavior segments from its current policy, estimates the variance of the implicit advantage difference between segment pairs (e.g., via an ensemble of CPL policies or via Monte Carlo dropout on the policy network), and queries the human for preferences on the pairs with highest predictive entropy. This would address two limitations simultaneously: the offline-only constraint (CPL cannot improve beyond its dataset) and the preference density requirement (active querying can focus annotation budget on informative comparisons). A strong experiment would benchmark online CPL against online PPO-based RLHF on a set of continuous control tasks with a fixed human annotation budget, measuring both final policy performance and the efficiency with which human feedback is converted into policy improvement. The key metric would be area under the success rate vs. number of human queries curve—how much performance does each method extract per unit of human effort? CPL's contrastive objective may prove more query-efficient than PPO's reward-learning-plus-RL pipeline because each preference directly shapes the policy rather than going through an intermediate reward representation.

CPL for multi-turn language model fine-tuning. The paper motivates CPL's MDP generality by invoking multi-step dialogue (Section 1), but no multi-turn preference dataset exists and no language experiments are conducted. A concrete follow-up would construct such a dataset: take an existing single-turn preference dataset (e.g., Anthropic's Helpful and Harmless, or OpenAI's summarization comparisons), but instead of labeling individual responses in isolation, present annotators with conversation prefixes followed by two possible continuations (each spanning multiple turns). The annotator's task is to judge which continuation represents better overall dialogue, implicitly evaluating multi-step properties like coherence maintenance, appropriate question-asking, and topic management. Training CPL on this dataset and comparing to DPO (which would treat each turn independently as a bandit) would directly test CPL's central claim: that modeling sequential preferences improves policy learning when the underlying problem is sequential. The experiment should measure both turn-level metrics (relevance, factuality per turn) and conversation-level metrics (overall coherence, task completion rate, user satisfaction in simulated dialogues). A null result—CPL performing no better than DPO on multi-turn metrics—would suggest that the bandit approximation is sufficient even for sequential language tasks, which would substantially weaken CPL's practical case for language. A positive result would open the door to RLHF for dialogue systems, interactive coding assistants, and multi-step reasoning that currently operate under the bandit assumption.

Scaling CPL to larger networks and measuring how the parameter efficiency gap grows. Table 2 reports a 4.6× parameter reduction (2.1M vs. 9.6M) at the DrQv2 scale. The paper speculates that "as networks get larger and larger, the performance gain from using CPL would only increase" (Section 4.1), but this is untested extrapolation. A scaling study that varies policy network size (e.g., from 2M to 200M parameters) and measures both CPL and P-IQL's (a) wall-clock training time, (b) GPU memory consumption, (c) final task performance, and (d) sample efficiency (performance vs. number of preference comparisons) would characterize how the complexity advantage scales. The hypothesis is that P-IQL's overhead—three additional networks that must all be scaled proportionally—causes the gap to grow superlinearly, making CPL increasingly attractive for large-scale deployments. A negative result—the gap saturating or P-IQL catching up at larger scales due to better representation learning from value function bootstrapping—would bound the practical advantage and suggest that CPL's benefits are most pronounced in the moderate-scale regime tested in the paper.

Diagnosing and mitigating CPL's failure under preference model misspecification. The Walker-Medium-Replay failure (Table 3) raises a fundamental question: can CPL be made robust to preferences that deviate from the regret model? One approach is to embed CPL's objective within a broader family of preference models that nests both the regret model and the partial-return model, and to learn the model parameters (e.g., a mixing coefficient) jointly with the policy. Concretely, one could parameterize the segment score as a convex combination of the discounted sum of advantages and the discounted sum of rewards, where the mixing weight is an additional learned parameter optimized to maximize the likelihood of observed preferences. If the data supports the regret model, the weight should approach 1; if it supports the partial-return model, the weight should approach 0. This would make CPL self-diagnosing: the learned mixing weight indicates which model better describes the annotator. A strong experiment would test this adaptive CPL on the four D4RL human-preference tasks plus synthetic datasets generated from both preference models, measuring whether (a) the learned mixing weight correctly identifies the generative model, and (b) the adaptive variant matches CPL's performance under the regret model while avoiding catastrophic failure under the partial-return model.

CPL with variable-length segments and temporal credit assignment within segments. The paper uses fixed-length segments of 64 steps and sums discounted log-probabilities uniformly. This treats all 64 steps as equally informative about the preference, but in practice, certain steps (e.g., the grasp in a pick-and-place task) likely carry more preference-relevant information than others (e.g., the approach trajectory). A follow-up could introduce learned attention weights over timesteps within a segment, where the segment score becomes $\sum_t w_t \gamma^t \alpha \log \pi(a_t|s_t)$ and $w_t$ is produced by a lightweight attention module trained end-to-end with the CPL loss. This would serve two purposes: (a) it could improve performance by focusing the contrastive signal on decision-relevant timesteps, and (b) the learned attention weights would provide interpretability—visualizing which timesteps the model considers important for preference judgments could reveal whether CPL is attending to task-relevant events or spurious correlations. The experiment would compare attention-weighted CPL against uniform CPL on tasks where the "important" moments are known a priori (e.g., MetaWorld tasks where success depends on a specific contact event), measuring both performance and whether attention weights peak at the task-critical timesteps.

Practical Applications and Downstream Use Cases

Offline robot policy refinement from human video comparisons. A robotics lab collects 2,500 rollout videos from a suboptimal policy on a manipulation task (e.g., a ~50% success policy for bin picking). Instead of designing a shaped reward function or collecting demonstrations, a human watches pairs of rollout videos and indicates which one looks "better"—closer to how they would want the robot to perform. CPL converts these 3.1 million pairwise preferences (all comparisons among 2,500 videos) into a policy that achieves 80% success (Table 1, row 1) using only a policy network with 2.1M parameters, trained in 10.2 hours on a single GPU (Table 2). The entire pipeline—preference annotation, CPL training, policy deployment—never requires writing a line of reward code, running an RL algorithm, or collecting online robot data. This matters practically because reward engineering for manipulation is notoriously difficult (what is the mathematical expression for "grasp the object securely and place it in the bin without dropping it"?), and online RL on real robots is slow and potentially unsafe. The tradeoff is annotation effort: 3.1 million pairwise video comparisons is far too many for a single human, but could be distributed across crowd workers or amortized over multiple projects. The paper's sparse results (20K comparisons achieving 83.2% on bin picking, Table 1 row 3) suggest that even moderate labeling budgets can produce useful policies, and the scaling curves (Figure 8, Appendix C.3) provide a basis for estimating how much annotation to budget for a target performance level.

Fine-tuning open-source chat models on multi-turn conversation quality. A company deploys a 7B-parameter chat model and collects conversation logs where users engage in extended multi-turn dialogues (clarification questions, follow-up requests, multi-step problem-solving). A subset of these conversations is annotated by experienced raters who compare pairs of conversation continuations (e.g., "given the first 3 turns, which of these two possible next 2-turn responses is better?"). CPL fine-tunes the model directly on these multi-turn preferences using the CPL-KL variant (Appendix B), which incorporates a KL penalty toward the pretrained model to prevent catastrophic forgetting. The benefit over DPO—which would treat each turn as an independent bandit—is that CPL can learn turn-taking behavior, appropriate follow-up questioning, and multi-step reasoning chains that only manifest across multiple turns. The paper's theoretical connection (Appendix A.6) guarantees that if the preferences follow the regret model, CPL recovers the optimal policy for the full conversation MDP, not just for single-turn responses. The benefit over PPO-based RLHF is implementation simplicity: no reward model to train, no value function to bootstrap, no policy gradient to estimate—just supervised fine-tuning with a logistic contrastive loss. This reduces the engineering burden from maintaining an RL pipeline (with its associated hyperparameters, stability issues, and compute requirements) to running a standard supervised training loop that fits naturally into existing fine-tuning infrastructure.

Preference-based reward specification for game-playing agents. A game development studio wants to train NPCs that exhibit "human-like" behavior in a complex environment, but hard-coding a reward function that captures nuanced stylistic preferences (aggressiveness, risk-taking, cooperation) is intractable. The studio generates 20,000 trajectory segments from a heuristic policy, has designers provide 10,000 pairwise preferences ("which segment looks more like how an experienced human player would act?"), and trains CPL. The resulting policy captures the designers' stylistic intent without explicit reward engineering. Because CPL is fully off-policy, it can use any suboptimal data source—the segments can come from heuristic bots, rule-based systems, or even human play sessions where the human's actions are recorded but not treated as demonstrations. The key practical benefit is decoupling behavior specification from behavior generation: designers specify intent through comparisons (which requires them to recognize desired behavior, not produce it) and CPL converts those comparisons into an executable policy. This matters in domains like game AI where the desired behavior is easy to judge but hard to generate (e.g., "play like a skilled but not superhuman opponent") and where traditional inverse reinforcement learning has struggled due to the complexity of the state space.

When to Prefer This Method

The paper does not articulate an explicit decision rule for choosing CPL over named alternatives like PPO-based RLHF, DPO, or P-IQL. However, the experimental results and theoretical properties imply a set of conditions under which CPL is the preferred approach. These conditions are not presented as a tradeoff matrix by the authors but emerge from the paper's evidence:

Prefer CPL when: (1) preferences can be collected densely—either by labeling all pairwise comparisons among a moderate number of segments or by training a preference predictor on limited human data and relabeling offline data—since CPL's advantage over P-IQL grows with comparison density (Figure 8, Appendix C.3); (2) the preference elicitation protocol can be designed to encourage regret-based judgments ("which behavior is closer to optimal?") rather than reward-based judgments ("which behavior achieved more of X?"), since the Walker-Medium-Replay failure (Table 3) suggests CPL degrades under model misspecification; (3) parameter efficiency and implementation simplicity are valued—CPL requires only a policy network (2.1M parameters) versus P-IQL's four networks (9.6M parameters; Table 2), and its supervised objective integrates into standard training pipelines without RL-specific infrastructure; (4) the task involves sequential decision-making with temporal structure that matters for preference judgments, since CPL is designed for the general MDP setting while DPO restricts to the bandit case; (5) offline data from a suboptimal policy is available and online interaction is expensive or unsafe, since CPL is fully off-policy and never requires environment interaction during training.

Prefer RL-based methods (P-IQL, PPO) when: (1) preferences are sparse—only a single comparison per segment pair is available—and data augmentation benefits value-based learning, since P-IQL closes the gap with CPL in the image-sparse regime (Table 1, row 4) and may benefit disproportionately from augmentation (Section 4.1); (2) preference labels are generated by rules-based or reward-based annotators whose judgments may violate the regret model, since P-IQL does not rely on the regret assumption and can learn from any consistent preference structure; (3) online environment interaction is cheap and offline data coverage is poor, since online RL methods like PPO can explore to fill coverage gaps while CPL is bounded by its dataset; (4) the discount factor $\gamma$ for temporal preferences is unknown and cannot be communicated to annotators, since CPL's segment-based loss requires committing to a specific $\gamma$ (set to 1 in all experiments; Table 6) while RL-based methods can treat discounting as a tunable hyperparameter in the policy optimization phase.

Prefer DPO (Rafailov et al., 2023) when: the problem genuinely satisfies the contextual bandit assumption—single-step responses to fixed prompts with no sequential dependence across turns—since DPO is simpler than CPL (no segment construction, no discount factor summation, no multi-step contrastive loss) and the two methods are equivalent in this limit (Appendix A.6). The paper's theoretical contribution clarifies this boundary: DPO is CPL with $k=1$ and shared starting state. If that is the problem structure, there is no reason to use CPL's additional machinery. If the problem has sequential structure that matters, DPO is solving a simplified version and CPL's generalization may yield improvements, though the paper does not empirically demonstrate this since no language experiments are conducted.