KTO: Model Alignment as Prospect Theoretic Optimization

🎯 Pitch

Alignment methods like DPO implicitly encode human cognitive biases like loss aversion, which explains their surprising effectiveness. By explicitly modeling human utility using Kahneman and Tversky’s prospect theory, the proposed KTO method learns from simple binary feedback (desirable/undesirable) yet matches or exceeds DPO that requires costly pairwise preferences—even improving math reasoning by over 13 points in some settings.

1. Executive Summary

This paper proposes Kahneman-Tversky Optimization (KTO), a new alignment method for LLMs that directly maximizes the utility of generations—modeled through a Kahneman-Tversky human value function—instead of maximizing the log-likelihood of preferences as existing methods do (e.g., DPO). Using the Pythia (1.4B–12B) and Llama (7B–30B) model families aligned on a combination of Anthropic-HH, OpenAssistant, and SHP, KTO matches or exceeds DPO performance while learning from only a binary signal of whether an output is desirable or undesirable, converting each preference pair into two independent examples. On a FLOPs-equivalent comparison using identical preference data, KTO achieves performance parity with DPO at scales from 1B to 30B, and when aligning Mistral-7B derivatives on UltraFeedback, swapping DPO for KTO improves GSM8K accuracy by 13.5 points—establishing that binary feedback, when coupled with a prospect-theoretic value function that is concave in gains and convex in losses, can be as effective as preference data when the feedback contains sufficient noise and intransitivity, as most real-world human feedback does.

2. Context and Motivation

The Core Problem: Alignment Methods Demand Preference Data, but Preferences Are Scarce and Expensive

The fundamental problem this paper tackles is a mismatch between what alignment methods demand and what the real world provides. The most successful approaches for aligning LLMs with human values—RLHF (Christiano et al., 2017; Ouyang et al., 2022) and Direct Preference Optimization (DPO; Rafailov et al., 2023)—require human feedback in the form of preferences: given an input $x$ , a human annotator must compare two outputs $y_w$ and $y_l$ and indicate which one is better ( $y_w \succ y_l$ ). This preference format is necessary because both RLHF and DPO are built around the Bradley-Terry model of pairwise comparisons:

$p^*(y_w \succ y_l | x) = \sigma(r^*(x, y_w) - r^*(x, y_l))$

The reward function $r^*$ is inferred by maximizing the likelihood of observed preferences, and the policy is optimized to produce outputs with high inferred reward.

But preferences are, as the authors note in Section 1, "relatively scarce and expensive to collect in practice." This is not a minor inconvenience—it is a structural bottleneck. Collecting preference data requires annotators to read two outputs per input, compare them carefully, and render a judgment. This is cognitively demanding, time-consuming, and therefore costly. In contrast, binary feedback—simply indicating whether a single output is desirable or undesirable for a given input—is more abundant, cheaper, and faster to collect. A user clicking "thumbs up" or "thumbs down" on a chatbot response, a developer marking a code completion as "accepted" or "rejected," or a content moderator flagging a generation as "harmful" or "safe"—these are all naturally occurring binary signals that exist at far greater volume than pairwise preferences.

The gap, then, is this: the alignment methods that work best are built for preference data, but the data that is most abundant is binary. The paper's central motivating question is whether this gap can be closed—whether alignment can be made to work effectively from binary feedback alone, without the pairwise structure that DPO and RLHF depend on.

Why This Matters: Scaling Alignment in Production

The practical stakes are substantial. As LLMs are deployed in production, the need to continuously align and re-align them with evolving human expectations grows. Consider a production chatbot: every user interaction generates implicit binary feedback (was the response helpful? did the user ask a follow-up? did they copy the code snippet?). If alignment methods could work from these binary signals, the feedback loop could be automated and scaled enormously. If they require preferences, each piece of feedback must be deliberately constructed and collected.

The paper cites Casper et al. (2023), which surveys open problems in RLHF and identifies the data bottleneck as a fundamental limitation. Preferences are not just expensive—they are also inherently lossy when deconstructed from richer forms of feedback. If a user provides a rating on a 5-point scale, converting that to preferences requires arbitrary binarization decisions (is a 4-vs-2 comparison equivalent to a 5-vs-3 comparison?). If an annotator flags an output as "toxic," there is no natural dispreferred counterpart to form a pair. The preference format forces a particular structure onto feedback that does not naturally fit many real-world collection pipelines.

Beyond the data bottleneck, there is a deeper theoretical question motivating this work: why do alignment methods like DPO work so well in the first place? The paper observes that DPO and PPO-Clip (the algorithm typically used for RLHF) outperform simpler baselines like conditional SFT or margin-based losses, but the reason for this superiority is not well understood. Is it because they use preference data? Because they incorporate a KL divergence penalty? Because their loss functions have particular mathematical properties? The paper sets out to answer this question by reframing alignment through the lens of prospect theory, the Nobel Prize-winning framework from behavioral economics that describes how humans make decisions under uncertainty (Kahneman & Tversky, 1979; Tversky & Kahneman, 1992).

Prior Approaches and Where They Fall Short

RLHF (Christiano et al., 2017; Ouyang et al., 2022). The standard RLHF pipeline works in three stages: pretraining, supervised finetuning (SFT), and then reinforcement learning with a learned reward model. The reward model is trained on preference data to predict $r_\phi(x, y)$ , and the policy $\pi_\theta$ is optimized using PPO to maximize this reward while staying close to the SFT model via a KL penalty:

$\max_{\pi_\theta} \mathbb{E}_{x \in \mathcal{D}, y \in \pi_\theta}[r_\phi(x, y)] - \beta D_{\text{KL}}(\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x))$

RLHF has been successfully used to make models more helpful, factual, and ethical (Ouyang et al., 2022; Bai et al., 2022; Tian et al., 2023). However, it suffers from significant practical drawbacks that the paper highlights: it is slow (because generations must be sampled from the current policy at each step), unstable (especially in distributed settings, where the interaction between the policy and reward model can diverge), and requires a separately trained reward model that adds complexity and potential for reward hacking. The paper's authors explicitly note in Section 3.3 that "RLHF is also quite unstable in a distributed setting."

DPO (Rafailov et al., 2023). DPO addressed many of RLHF's practical limitations by deriving a closed-form loss that directly optimizes the policy from preference data, without needing to train a separate reward model or run an RL algorithm. The key insight is that the optimal policy under the RLHF objective has a closed form:

$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} r^*(x, y)\right)$

By rearranging this equation, the reward can be expressed in terms of the policy, and plugging this into the Bradley-Terry preference model yields the DPO loss:

$\mathcal{L}_{\text{DPO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x, y_w, y_l \sim \mathcal{D}}\left[-\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$

DPO is simpler, faster, and more stable than RLHF. But it still fundamentally requires preference data: the loss is defined over pairs $(y_w, y_l)$ , and the gradient pushes the model to increase the relative likelihood of $y_w$ versus $y_l$ . The paper's contribution is not to criticize DPO's performance—in fact, DPO is the primary baseline that KTO matches or exceeds—but rather to question whether the preference format is necessary to achieve DPO-level results, and to understand what property of the DPO loss makes it effective.

Conditional SFT (Korbak et al., 2023). This method prepends control tokens like ⟨|good|⟩ or ⟨|bad|⟩ to outputs during training, then at inference appends the ⟨|good|⟩ token to elicit desirable generations. It works with binary feedback (good/bad) rather than preferences, making it directly comparable in data requirements to KTO. However, the paper shows that CSFT underperforms HALOs (human-aware losses) at every scale (Figure 2), and in Section 3.3 the authors prove that CSFT is not a HALO because it cannot satisfy the definition—specifically, its reference point is fixed for a given $x$ but must equal $-\log \pi_{\text{ref}}(y|x)$ , which is impossible when $\pi_{\text{ref}}$ is a non-uniform distribution.

SLiC (Zhao et al., 2023). Sequence Likelihood Calibration combines a max-margin loss over preferences with a language modeling regularization term:

$\mathcal{L}_{\text{cal}}(\pi_\theta) = \mathbb{E}_{x, y_w, y_l \sim \mathcal{D}}\left[\max\left(0, \delta - \log \frac{\pi_\theta(y_w|x)}{\pi_\theta(y_l|x)}\right)\right]$

$\mathcal{L}_{\text{reg}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\text{ref}}(x)}[-\log \pi_\theta(y|x)]$

SLiC is not a HALO for the same reason as CSFT, and the paper's experiments (Figure 2) show it underperforming HALOs at scale.

Unlikelihood Training (Welleck et al., 2019). One of the earliest methods to align language models using binary feedback, unlikelihood training simply penalizes the model for generating tokens from undesirable outputs. However, Korbak et al. (2023) found it to be worse than CSFT, and the paper does not include it as a primary baseline—its weakness is taken as established.

Reference-free methods (ORPO; Hong et al., 2024). ORPO combines SFT and alignment into a single stage without a reference model, using an odds-ratio-based penalty. The paper compares KTO's memory-efficient variant (which also operates without a reference model) to ORPO and finds it strictly outperforms ORPO while using less memory (Table 2, Section 4.3).

The offline PPO variant (Section 3.3). One of the paper's most striking preliminary findings comes from a baseline they construct: an offline variant of PPO that operates on dummy +1/-1 rewards (good outputs get +1, bad outputs get -1) rather than learned rewards. This method is described in Section 3.3 and Appendix C. Despite its simplicity—no reward model, no online sampling, just binary signals converted to dummy rewards—it performs as well as DPO for all models except Llama-30B (Figure 2). This result challenges the conventional wisdom that reward learning is critical (as argued by Casper et al., 2023) and suggests that the inductive bias of the loss function matters more than the quality of the reward signal. However, the offline PPO variant suffers from hyperparameter sensitivity and training instability, making it difficult to use reliably in practice. It serves as an existence proof: binary feedback can work if the loss function has the right properties, motivating the search for a more principled approach.

Where Prior Work Falls Short: A Unified Gap

Collectively, these prior approaches exhibit several limitations that this paper directly addresses:

Dependence on preference data. RLHF, DPO, SLiC, and similar methods require pairwise comparisons. This is a data format mismatch with the abundant binary feedback available in production systems.
Lack of theoretical understanding of why alignment losses work. Before this paper, there was no framework explaining why DPO outperforms CSFT, or why the KL penalty in RLHF is beneficial beyond keeping the model close to the reference. The field relied on empirical comparisons without a unifying theory of what makes an alignment loss effective.
No principled approach to binary feedback. Methods that do use binary feedback (CSFT, unlikelihood training) underperform preference-based methods. The offline PPO variant shows that binary feedback can compete, but it is a crude hack—dummy rewards fed into a clipped policy gradient algorithm—without theoretical grounding. What was missing is a loss function designed from first principles to maximize human utility from binary feedback.
Reward-centric thinking. The alignment literature is organized around the idea of learning a reward function and optimizing against it. DPO cleverly avoids explicitly learning a reward by reparameterizing it in terms of the policy, but it still inherits the reward-maximization framing via the Bradley-Terry model. The paper questions whether reward maximization is the right objective at all, suggesting instead that utility maximization—in the prospect-theoretic sense of how humans actually perceive value—may be more appropriate.

How This Paper Positions Itself

The paper introduces a new framing that cuts across these limitations: alignment objectives as human-aware losses (HALOs). This framing serves several strategic purposes:

It explains why existing methods work. By showing that DPO and PPO-Clip are HALOs (Theorem 3.5)—their loss functions can be expressed as prospect-theoretic value functions applied to an implied reward—the paper provides a unifying explanation for their success. The inductive biases that make them effective (reference dependence, diminishing sensitivity, loss aversion) are the same biases that prospect theory identifies in human decision-making. In other words, DPO works well partly because its loss function thinks like a human about the relative goodness of outputs.

It explains why some methods don't work. CSFT and SLiC are not HALOs, and they underperform. The paper is careful not to claim that being a HALO is necessary for an alignment method to work—the function class of HALOs and non-HALOs are both diverse—but the empirical pattern in Figure 2 is suggestive: among the methods tested, HALOs match or outperform non-HALOs at every scale, and only HALOs achieve above-chance winrates against the SFT baseline at 13B+ parameters.

It provides a design framework for new losses. The HALO definition (Definition 3.4) specifies the necessary components: an implied reward $r_\theta(x, y) = l(y) \log[\pi_\theta(y|x) / \pi_{\text{ref}}(y|x)]$ , a reference point $\mathbb{E}_Q[r_\theta(x, y')]$ representing the human's baseline expectation, and a value function $v$ that is non-decreasing everywhere and concave in gains. Any loss function satisfying these criteria inherits the prospect-theoretic inductive biases. KTO is constructed by instantiating this framework with the specific value function from Tversky & Kahneman (1992), adapted with a logistic sigmoid for numerical stability and with separate loss aversion coefficients $\lambda_D, \lambda_U$ for desirable and undesirable outputs.

It shifts the objective from preference likelihood to human utility. This is the paper's most fundamental repositioning. DPO maximizes the log-likelihood of observed preferences under a Bradley-Terry model. KTO maximizes something different: the prospect-theoretic utility of generations, where utility is defined by how humans actually perceive gains and losses relative to a reference point. Theorem 4.2 formalizes why this distinction matters: two reward functions in the same equivalence class (differing by an input-specific shift $h(x)$ ) induce the same optimal policy and the same Bradley-Terry preference distribution, but they induce different human value distributions. Maximizing preference likelihood does not uniquely determine the utility that humans experience—KTO directly targets utility instead.

It establishes that preferences are not necessary. By breaking each preference pair $(y_w, y_l)$ into two independent binary examples (desirable and undesirable) and applying KTO, the paper demonstrates that the pairwise structure of preference data is not required to achieve DPO-level alignment. The one-y-per-x experiments (Table 3), where only one output per input is used—removing any trace of pairing—further confirm this. KTO with 72% less data (due to the one-y-per-x restriction) still outperforms DPO on the full dataset when aligning Mistral-7B on OpenAssistant.

It acknowledges that no single loss is universally superior. Despite the paper's advocacy for KTO, it concludes in Section 6 that "the best HALO depends on the inductive biases that are most appropriate for a given setting." This is not false modesty—it follows from the theoretical analysis. Theorem 4.3 shows that when preference data contains contradictions (intransitivity), DPO can produce the minority-preferred output under certain conditions, while KTO with a loss-neutral value function deterministically produces the majority-preferred output. This makes KTO more robust to noisy feedback, which is common in real-world data, but it also means KTO may underfit complex distributions by ignoring hard-to-learn examples (Proposition 4.1). The choice between DPO and KTO—and between any two HALOs—should depend on the noise characteristics of the data and the desired robustness properties.

The Prospect Theory Connection: Why Human Biases Matter for Alignment

A key motivation that deserves emphasis is the paper's insight about why human biases are relevant to alignment objectives. Prospect theory describes systematic deviations from expected utility maximization: humans are loss-averse (a loss of $X hurts more than a gain of$ X pleases), they exhibit diminishing sensitivity (the difference between $0 and$ 100 feels larger than between $1000 and$ 1100), and they evaluate outcomes relative to a reference point rather than in absolute terms. These are biases—they cause humans to make decisions that do not maximize expected value.

But for alignment, these biases may be features, not bugs. When we ask an LLM to produce outputs that humans find helpful, harmless, and ethical, we are asking it to satisfy human preferences. If human preferences are shaped by prospect-theoretic biases, then a loss function that incorporates those same biases may better capture what humans actually want. The paper's Figure 1 illustrates this visually: DPO, PPO-Clip, and the Kahneman-Tversky value function all share qualitative properties (reference dependence, concavity in gains, steeper slope for losses) that distinguish them from the identity function (risk-neutrality) or a purely concave function.

This reframes alignment not as an optimization problem against an objective reward function, but as a modeling problem: how do we model human utility in a way that captures the systematic ways humans evaluate outcomes? The HALO framework provides a language for this modeling, and KTO provides one specific instantiation based on the most influential model of human utility in the behavioral economics literature.

3. Technical Approach

3.1 Reader Orientation

This paper develops a new loss function for aligning LLMs with human feedback. The system is a mathematical objective—Kahneman-Tversky Optimization (KTO)—that takes binary feedback (desirable/undesirable labels on individual outputs) and a reference model, and produces gradient updates that push the model to generate desirable outputs while avoiding undesirable ones. The problem it solves is that existing alignment methods like DPO require expensive pairwise preference data, but binary feedback is far more abundant in practice; KTO bridges this gap by directly maximizing the prospect-theoretic utility of generations—how humans actually perceive value with biases like loss aversion—rather than maximizing the likelihood of observed preferences.

3.2 Big-Picture Architecture (Diagram in Words)

The KTO training pipeline has five major components:

Reference Model ( $\pi_{\text{ref}}$ ): A pretrained and optionally supervised-finetuned LLM that serves as the baseline from which the aligned model should not drift too far. It provides the denominator in all reward calculations.
Implied Reward Function ( $r_\theta$ ): A scalar computed per-output as the log-ratio of probabilities under the current policy versus the reference model, normalized by a factor $\beta$ . This measures how much the aligned model has increased or decreased the likelihood of an output relative to the reference—positive for outputs the model favors more than the reference, negative for outputs it favors less.
Reference Point Estimator ( $\hat{z}_0$ ): A biased but computationally efficient estimate of the expected reward across the output distribution. It uses mismatched input-output pairs within each microbatch to approximate the KL divergence between the policy and reference model, serving as the baseline against which gains and losses are measured.
Kahneman-Tversky Value Function ( $v$ ): A logistic sigmoid applied to the reward deviation from the reference point, with separate scaling for desirable and undesirable outputs. This function is concave in gains (diminishing sensitivity to increasingly good outputs) and convex in losses (diminishing sensitivity to increasingly bad outputs), with loss aversion controlled by separate coefficients $\lambda_D$ and $\lambda_U$ .
KTO Loss ( $\mathcal{L}_{\text{KTO}}$ ): The final scalar objective computed as the sum over examples of $\lambda_y - v(x, y)$ . Gradient descent on this loss increases the probability of desirable outputs and decreases the probability of undesirable ones, with the update magnitude naturally decaying to zero as outputs become extremely likely or unlikely.

Information flows as follows: a batch of $(x, y, \text{label})$ triples enters the system → the current policy $\pi_\theta$ and reference model $\pi_{\text{ref}}$ both score each $y|x$ → the implied reward $r_\theta(x, y)$ is computed as the log-ratio of these probabilities → mismatched pairs within the microbatch produce an estimate $\hat{z}_0$ of the reference point → the value function $v$ maps the deviation $r_\theta(x, y) - \hat{z}_0$ to a utility between 0 and $\lambda_y$ → the loss computes $\lambda_y - v(x, y)$ , and gradients flow back through $\pi_\theta$ only (the reference point is treated as a constant for stability).

3.3 Roadmap for the Deep Dive

First, the formal definition of Human-Aware Losses (HALOs) in Definition 3.4 and the proof that DPO and PPO-Clip are HALOs (Theorem 3.5), because this establishes the theoretical framework that KTO belongs to and explains why the specific mathematical structure—implied reward, reference point, value function—matters.
Second, the construction of the KTO loss from the Kahneman-Tversky value function (Equation 8), walking through each design choice: the logistic sigmoid replacement for numerical stability, the separate $\lambda_D$ and $\lambda_U$ coefficients, the mismatched-pair estimator for the reference point, and why the reference point is not backpropagated through.
Third, the gradient of the KTO loss (Equation 10) and what it reveals about how updates behave: why the gradient vanishes for extreme rewards (Proposition 4.1), how this provides implicit robustness to mislabeled data, and why it may also cause underfitting.
Fourth, the hyperparameter configuration space: learning rate, batch size, $\beta$ (risk aversion), $\lambda_D$ and $\lambda_U$ (loss aversion), the effective ratio guideline in Equation 9, and the settings that work for different model scales and whether SFT is used.
Fifth, the theoretical results that explain KTO's empirical performance: Theorem 4.2 (preference likelihood maximization does not uniquely determine human utility) and Theorem 4.3 (KTO handles intransitive preferences with better worst-case guarantees than DPO).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theory-driven method paper whose core idea is that alignment objectives can be understood as prospect-theoretic value functions applied to an implied reward, and that constructing a loss function directly from the canonical Kahneman-Tversky model of human utility enables effective alignment from binary feedback alone.

The HALO Framework: A Unifying Definition for Alignment Objectives

The paper begins not with KTO itself but with a definitional framework that captures what makes existing alignment methods effective. This framework—Human-Aware Losses (HALOs)—is Definition 3.4, and it serves as both an analytical tool (explaining why DPO works) and a design tool (guiding the construction of KTO). Understanding HALOs is prerequisite to understanding KTO because KTO is deliberately constructed to satisfy the HALO definition.

Definition 3.4 formalizes a HALO as any loss function $f(\pi_\theta, \pi_{\text{ref}})$ that can be expressed in the following form:

$f(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x,y \sim \mathcal{D}}[a_{x,y} \cdot v(r_\theta(x, y) - \mathbb{E}_Q[r_\theta(x, y')])] + C_\mathcal{D}$

where:

$\pi_\theta$ is the policy being optimized (the aligned model), mapping inputs $x \in \mathcal{X}$ to distributions over outputs $y \in \mathcal{Y}$ .
$\pi_{\text{ref}}$ is the reference model, typically the SFT model, treated as fixed during alignment.
$r_\theta(x, y) = l(y) \cdot \log\frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ is the implied reward: the change in conditional surprisal (measured in nats, since $\log$ is natural log) when using $\pi_\theta$ instead of $\pi_{\text{ref}}$ to predict $y|x$ , scaled by a normalizing factor $l(y)$ . A positive $r_\theta$ means the aligned model assigns higher probability to $y$ than the reference does; a negative $r_\theta$ means lower probability.
$\mathbb{E}_Q[r_\theta(x, y')]$ is the reference point: the expected reward under some distribution $Q(Y'|x)$ over outputs. This is what the human uses as a baseline when evaluating whether a particular output is good or bad.
$v: \mathbb{R} \to \mathbb{R}$ is the value function (in the prospect-theoretic sense), satisfying two properties: it is non-decreasing everywhere (better rewards never produce lower perceived value), and it is concave in $(0, \infty)$ (diminishing sensitivity to gains—the jump from reward 0 to reward 1 feels larger than from reward 10 to reward 11).
$a_{x,y} \in \{-1, +1\}$ is a sign that determines whether the loss decreases as the value increases ( $a_{x,y} = -1$ , typical for desirable outputs) or decreases as the value decreases ( $a_{x,y} = +1$ , typical for undesirable outputs).
$C_\mathcal{D} \in \mathbb{R}$ is a data-specific constant that does not depend on $\theta$ and therefore does not affect the gradient.
$\mathcal{D}$ is the feedback dataset, which for HALOs can contain any form of human feedback—preferences, binary labels, ratings—as long as it can be mapped to the $(x, y, a_{x,y})$ structure.

What this definition computes, operationally: For each training example $(x, y)$ , the model computes how much it has changed the probability of $y$ relative to the reference (the implied reward $r_\theta$ ), subtracts the human's expected baseline reward (the reference point), maps this deviation through a value function $v$ that captures prospect-theoretic biases (loss aversion, diminishing sensitivity), and then multiplies by a sign $a_{x,y}$ that determines whether increasing or decreasing the value reduces the loss. The loss is the expectation of this signed value over the dataset, plus a constant.

Why this form: The definition is designed to capture three properties that prospect theory identifies as fundamental to human decision-making, and that the paper argues are critical for alignment:

Reference dependence: Humans evaluate outcomes relative to a reference point, not in absolute terms. The subtraction $\mathbb{E}_Q[r_\theta(x, y')]$ encodes this: an output is perceived as a gain or loss relative to what the human expected, not relative to some absolute scale of goodness. In DPO, the reference point is the reward of the dispreferred output $y_l$ ; in KTO, it is the expected reward over the output distribution.
Diminishing sensitivity: The concavity of $v$ in gains means that as outputs become increasingly good (relative to the reference point), further improvements yield smaller and smaller increases in perceived value. This prevents the model from over-optimizing a few examples at the expense of the distribution—once an output is "good enough," the gradient signal weakens. The requirement that $v$ is non-decreasing (not necessarily convex) in losses accommodates the fact that some humans are risk-averse in losses (concave everywhere) while the canonical Kahneman-Tversky value function is convex in losses (risk-seeking).
Implied reward as the carrier of alignment information: The reward $r_\theta(x, y) = l(y) \log[\pi_\theta(y|x) / \pi_{\text{ref}}(y|x)]$ follows naturally from the next-token prediction objective used in pretraining and SFT. It measures alignment progress in the same units (nats of surprisal reduction) that the model was originally trained to optimize. The connection to RLHF is direct: under the optimal policy $\pi^*$ that maximizes the RLHF objective (Equation 2), the HALO-defined reward becomes $r_{\theta^*}(x, y) = r^*(x, y) - \beta \log Z(x)$ , which is the true reward shifted by an input-specific term. This shift does not change the optimal policy (Lemma 1, Rafailov et al., 2023), meaning that the HALO reward and the true reward are in the same equivalence class.

Theorem 3.5: DPO and PPO-Clip are HALOs (proof in Appendix B). The paper proves that both DPO and the PPO-Clip objective satisfy Definition 3.4, establishing that the HALO framework captures the methods that empirically work best. For DPO, the construction is:

$l(y) = \beta$ (the same $\beta$ from the RLHF KL penalty)
$r_\theta(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ (the DPO implicit reward)
$v(\cdot) = \log \sigma(\cdot)$ (the log-sigmoid, which is increasing and concave everywhere)
The reference point distribution $Q$ places all mass on $(x, y_l)$ , the dispreferred output, so $\mathbb{E}_Q[r_\theta(x, y')] = r_\theta(x, y_l)$
$a_{x,y_w} = -1$ (the loss decreases as the value increases)

Plugging these into the HALO form recovers exactly the DPO loss:

$\mathcal{L}_{\text{DPO}} = \mathbb{E}_{x, y_w, y_l \sim \mathcal{D}}[-\log \sigma(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)})]$

For PPO-Clip, the construction is more involved (see Appendix B), but the key insight is that the per-token advantage $A(x:y_{<t}, y_t) = Q^\pi(x:y_{<t}, y_t) - \mathbb{E}_{y \sim \pi}[Q^\pi(x:y_{<t}, y)]$ naturally provides the reference point (the expected action-value under the policy), and the clipping function $v(q_\theta A) = \min(q_\theta A, A(1 + \text{sign}(q_\theta A)\epsilon))$ serves as the value function. This means PPO-Clip's success can be partially attributed to it implementing a prospect-theoretic value function with a specific shape—clipped linear rather than sigmoidal.

The significance of Theorem 3.5: It provides a unified explanation for why DPO and PPO-Clip outperform non-HALOs like CSFT and SLiC. The inductive biases that make these losses effective—comparing outputs to a reference point, diminishing sensitivity to gains, loss aversion through asymmetric treatment of positive and negative advantages—are the same biases that prospect theory identifies in human decision-making. In other words, DPO works well partly because its loss function "thinks about" the relative goodness of outputs in the same biased way that humans do.

The Kahneman-Tversky Value Function: From Monetary Gambles to Text Generation

Prospect theory, as developed by Kahneman and Tversky across decades of work (1979, 1992), provides a mathematical model of how humans evaluate uncertain outcomes. The central component is the value function, which maps objective outcomes (e.g., dollar amounts) to subjective value (how good or bad the outcome feels). The canonical functional form proposed in Tversky & Kahneman (1992), based on experiments where real humans stated their certainty equivalents for monetary gambles, is:

$v(z; \lambda, \alpha, z_0) = \begin{cases} (z - z_0)^\alpha & \text{if } z \geq z_0 \\ -\lambda(z_0 - z)^\alpha & \text{if } z < z_0 \end{cases}$

where:

$z$ is the objective outcome (e.g., amount of money received)
$z_0$ is the reference point relative to which gains and losses are evaluated
$\alpha \in (0, 1)$ controls the curvature: $\alpha < 1$ produces concavity in gains (diminishing sensitivity—the psychological difference between gaining $0 and$ 100 exceeds that between gaining $1000 and$ 1100) and convexity in losses (diminishing sensitivity to losses—losing $100 hurts almost as much as losing$ 200)
$\lambda > 1$ controls loss aversion: losses are weighted $\lambda$ times more heavily than equivalent-magnitude gains. The median $\lambda = 2.25$ across individuals means a loss of $X feels about 2.25 times as bad as a gain of$ X feels good.

What this function computes, operationally: Given an outcome $z$ and a reference point $z_0$ , it produces a scalar representing the subjective value of that outcome. If $z \geq z_0$ , the value is $(z - z_0)^\alpha$ —positive, concave, reflecting risk aversion in gains. If $z < z_0$ , the value is $-\lambda(z_0 - z)^\alpha$ —negative, steeper by factor $\lambda$ , reflecting loss aversion. The curvature $\alpha$ means that marginal value diminishes with distance from the reference point in both directions.

Why this form: The power-law shape with $\alpha < 1$ captures the empirically observed pattern that humans are insensitive to the absolute magnitude of outcomes far from the reference point but highly sensitive near it. The asymmetry $\lambda > 1$ captures loss aversion, one of the most robust findings in behavioral economics. The reference point $z_0$ captures the fact that humans do not evaluate outcomes in isolation—a $100 gain feels very different to someone expecting to lose$ 100 versus someone expecting to gain $1000.

The challenge in adapting this to LLM alignment: The original Kahneman-Tversky value function cannot be directly plugged into a loss function for gradient-based optimization for two reasons. First, the exponent $\alpha$ (typically around 0.88) is between 0 and 1, which means the gradient $\alpha(z - z_0)^{\alpha-1}$ becomes unbounded as $z \to z_0$ —this causes numerical instability during training when the implied reward is near the reference point. Second, the value function grows unboundedly as $z \to \infty$ , meaning there is no natural saturation point—the model would receive gradient signal pushing it to increase the reward of desirable outputs without bound, leading to over-optimization. The paper's solution is to replace the power-law with a logistic sigmoid, which is bounded, smooth, and has well-behaved gradients everywhere.

Constructing the KTO Loss: From Value Function to Training Objective

The KTO loss (Equation 8) is constructed by instantiating each component of the HALO framework with choices motivated by the Kahneman-Tversky model. The derivation proceeds through five design decisions:

Design Decision 1: Replace the power-law value function with a logistic sigmoid.

The value function used in KTO, applied to the deviation $r_\theta(x, y) - z_0$ from the reference point, is:

$v(x, y) = \begin{cases} \lambda_D \cdot \sigma(\beta(r_\theta(x, y) - z_0)) & \text{if } y \text{ is desirable given } x \\ \lambda_U \cdot \sigma(\beta(z_0 - r_\theta(x, y))) & \text{if } y \text{ is undesirable given } x \end{cases}$

where $\sigma(u) = \frac{1}{1 + e^{-u}}$ is the logistic sigmoid, and $r_\theta(x, y) = \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ is the implied reward with normalizing factor $l(y) = 1$ (absorbed into $\beta$ ).

What this computes: For a desirable output, the value is $\lambda_D$ times the sigmoid of the positive deviation $\beta(r_\theta - z_0)$ . As the model increases the reward $r_\theta$ relative to the reference point, the sigmoid approaches 1 from below, and the value approaches $\lambda_D$ . For an undesirable output, the argument is flipped to $\beta(z_0 - r_\theta)$ , so the sigmoid approaches 1 as the reward decreases relative to the reference point, and the value approaches $\lambda_U$ . In both cases, the value is bounded in $[0, \lambda_y]$ , and the $\beta$ parameter controls how quickly it saturates—higher $\beta$ means the sigmoid transitions from near-0 to near-1 over a narrower range of reward deviations.

Why the logistic form over the power-law:

Boundedness: The sigmoid saturates at 0 and 1 (times $\lambda_y$ ), so the value cannot grow unboundedly. This prevents the over-optimization problem that would occur with the unbounded power-law. Once an output is sufficiently good or sufficiently bad, further changes in probability produce negligible changes in value, and the gradient naturally decays to zero.
Numerical stability: The sigmoid and its derivative $\sigma'(u) = \sigma(u)(1 - \sigma(u))$ are smooth and well-behaved for all real $u$ . There are no singularities near the reference point.
Concavity in gains, convexity in losses: The logistic function is concave for $u > 0$ (its second derivative is negative) and convex for $u < 0$ (its second derivative is positive), matching the qualitative shape of the Kahneman-Tversky value function. For desirable outputs, we operate in the $u > 0$ regime (concave gains); for undesirable outputs, $\beta(z_0 - r_\theta) > 0$ when $r_\theta < z_0$ , which is the convex loss regime.
Symmetry: The use of $\sigma(\beta(r_\theta - z_0))$ for desirable and $\sigma(\beta(z_0 - r_\theta))$ for undesirable creates a symmetric treatment around the reference point (up to the scaling by $\lambda_D$ and $\lambda_U$ ). If $\lambda_D = \lambda_U$ , the value function is perfectly symmetric: a desirable output with reward $z_0 + \delta$ has the same value as an undesirable output with reward $z_0 - \delta$ .

Design Decision 2: Use separate loss aversion coefficients $\lambda_D$ and $\lambda_U$ .

In the original Kahneman-Tversky value function, a single parameter $\lambda > 1$ captures loss aversion—losses are weighted more heavily than gains. KTO replaces this with two hyperparameters:

$\lambda_D$ : the maximum value (and loss contribution) for desirable outputs
$\lambda_U$ : the maximum value (and loss contribution) for undesirable outputs

Why separate coefficients: The single $\lambda$ in the original theory captures the relative weighting of losses versus gains. In the alignment setting, the relative weighting matters, but so does the absolute weighting, because the dataset may have different numbers of desirable and undesirable examples. If there are 10 undesirable examples for every desirable one, and both have $\lambda_D = \lambda_U = 1$ , the total loss contribution from undesirable examples will dominate, and the model will focus almost exclusively on avoiding bad outputs rather than producing good ones. By setting $\lambda_D > \lambda_U$ , the practitioner can upweight the desirable examples to compensate for their scarcity, controlling the effective balance between positive and negative feedback.

The paper provides an empirical guideline (Equation 9) for setting these coefficients based on dataset imbalance:

$\frac{\lambda_D n_D}{\lambda_U n_U} \in \left[1, \frac{4}{3}\right]$

where $n_D$ and $n_U$ are the number of desirable and undesirable examples respectively. This means the total effective weight on desirable examples should be equal to or slightly higher than the total effective weight on undesirable examples. The upper bound of $4/3$ encodes the finding that "gain sensitivity yields better performance than loss sensitivity"—producing good outputs is slightly more important than avoiding bad outputs for success on most benchmarks.

Design Decision 3: Define the reference point as the KL divergence and estimate it from mismatched pairs.

The reference point $z_0$ in a HALO represents the human's baseline expectation—what reward they expect an average output to have. The paper argues that humans judge the quality of $y|x$ in relation to all possible outputs the model could produce, not just a single dispreferred alternative (as in DPO). This means the reference point distribution $Q(Y'|x)$ should be the policy itself, and the reference point should be the expected reward under that policy:

$z_0 = \mathbb{E}_{y' \sim \pi_\theta(\cdot|x)}[r_\theta(x, y')] = \mathbb{E}_{y' \sim \pi_\theta(\cdot|x)}\left[\log \frac{\pi_\theta(y'|x)}{\pi_{\text{ref}}(y'|x)}\right] = \text{KL}(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))$

What this computes: The reference point is exactly the KL divergence from the reference model to the current policy. If $\pi_\theta = \pi_{\text{ref}}$ , the KL is zero—the expected reward is zero, meaning the model is neither better nor worse than the reference on average. As the model diverges from the reference (by increasing probability on some outputs and decreasing on others), the KL grows positive, and the reference point shifts upward. This means that to achieve a positive reward deviation $r_\theta(x, y) - z_0$ for a desirable output, the model must not merely increase $\pi_\theta(y|x)$ relative to $\pi_{\text{ref}}(y|x)$ , but must do so selectively—increasing probability on specific desirable outputs while keeping the overall distribution close enough to the reference that the KL penalty doesn't cancel out the gain.

Why this form: Using the KL divergence as the reference point has two crucial properties:

It forces selective improvement: The model cannot simply increase the probability of all desirable outputs uniformly, because that would increase the KL divergence and shift the reference point upward, negating the gain. Instead, the model must learn what makes an output desirable—the features that distinguish good outputs from the average—and allocate probability mass to those features specifically. This is the key mechanism that prevents reward hacking: if the model bluntly increases probability on all training examples, the KL term rises and no net progress is made.
It provides a natural scaling: As the model improves and the KL divergence grows, the bar for what counts as a "good" output rises. This creates a self-regulating dynamic: early in training, when the model is close to the reference, even small probability increases on desirable outputs produce positive reward deviations. Later in training, when the model has diverged significantly, only outputs that the model strongly prefers over the reference produce gains relative to the elevated baseline.

The practical estimation challenge: Computing the exact KL divergence would require summing over all possible outputs $y'$ (which is intractable for autoregressive models over token sequences) or sampling from $\pi_\theta$ (which is slow, since it requires generating complete sequences at each training step). The paper instead uses a biased but convenient estimator based on mismatched pairs within each microbatch.

The mismatched-pair estimator. For a microbatch of $m$ examples $\{(x_1, y_1), (x_2, y_2), ..., (x_m, y_m)\}$ , the reference point is estimated as:

$\hat{z}_0 = \max\left(0, \frac{1}{m} \sum_{1 \leq i < m} \log \frac{\pi_\theta(y_j|x_i)}{\pi_{\text{ref}}(y_j|x_i)}\right)$

where $j = (i + 1) \bmod m$ , meaning each output $y_j$ is paired with the next input $x_i$ in the microbatch, creating mismatched pairs $(x_1, y_2), (x_2, y_3), ..., (x_m, y_1)$ . The log-ratio is computed for each mismatched pair, averaged, and clamped to be non-negative.

What this computes, operationally: For each example in the microbatch, instead of evaluating the reward of $y_i$ given its own input $x_i$ (which would give the per-example reward, not the reference point), we evaluate the reward of $y_j$ given a different input $x_i$ from the same microbatch. Since $y_j$ was generated for $x_j$ , not $x_i$ , the pair $(x_i, y_j)$ represents an approximately random output for input $x_i$ , and the average of these cross-rewards approximates the expected reward over the output distribution. The max-clamping at zero ensures the reference point is never negative (which would be problematic since the KL divergence is always non-negative).

Why mismatched pairs instead of matched pairs: Using the matched pair $(x_i, y_i)$ would give a biased estimate because $y_i$ is deliberately chosen to be canonically good or bad—it is not a random draw from the output distribution. A desirable $y_i$ would have an unrepresentatively high reward, inflating the reference point estimate and making it harder for desirable outputs to register as gains. Using mismatched pairs breaks this correlation: $y_j$ is not related to $x_i$ , so its reward approximates what a random output would get, which is closer to the true expected reward.

Why the estimator is biased: The clamping $\max(0, \cdot)$ introduces a positive bias (the estimate is always $\geq 0$ and strictly positive whenever the unclamped average is negative). The mismatched pairs themselves are not truly independent draws from the output distribution—they come from the same microbatch and may share statistical properties. However, the paper explicitly acknowledges this bias and argues that it is acceptable because the human-perceived reference point would also be biased: humans do not perceive the full distribution induced by $\pi_\theta$ and would employ an "availability heuristic" (Tversky & Kahneman, 1973) that overweights outputs for which they have recently given feedback, regardless of whether those outputs are good continuations of the current input. The mismatched-pair estimator, by drawing from the current microbatch (which contains recently-seen feedback examples), mimics this availability bias.

When the estimator can be set to zero: If KTO is preceded by SFT on the same data used as desirable feedback, and the SFT model is used as $\pi_{\text{ref}}$ , the KL divergence will quickly approach zero. The policy, having already learned what is desirable during SFT, will tend to scatter the probability mass placed on undesirable examples (since they are not reinforced during SFT), leading to minimal divergence. In this case, $\hat{z}_0$ is effectively zero, and the extra computation of the mismatched-pair forward pass can be skipped. This is noted in Section 4.1 as a practical optimization.

Design Decision 4: Do not backpropagate through the reference point.

The KTO loss treats $z_0$ as a constant during gradient computation. The paper states this explicitly: "For more stable training, we do not backpropagate through $z_0$ ; it exists purely to control the loss saturation."

Why detach the reference point: If gradients were allowed to flow through $z_0$ , the model could reduce the loss by artificially lowering the reference point (e.g., by decreasing probability on all outputs, which would reduce the KL divergence). This would create a pathological incentive: the model could make desirable outputs look better not by actually improving them, but by making the baseline worse. Detaching $z_0$ from the computational graph means the reference point is treated as an external constant that the model must genuinely surpass, not manipulate.

Design Decision 5: The complete KTO loss.

Assembling the components, the full KTO loss for a single example $(x, y)$ with desirability label is:

$\mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x, y \sim \mathcal{D}}[\lambda_y - v(x, y)]$

where $\lambda_y$ is $\lambda_D$ if $y$ is desirable and $\lambda_U$ if $y$ is undesirable, and $v(x, y)$ is as defined in Design Decision 1.

What this computes, operationally: For each training example, subtract the prospect-theoretic value of the output from its maximum possible value $\lambda_y$ . Since the value is bounded in $[0, \lambda_y]$ by the sigmoid, the loss per example is bounded in $[0, \lambda_y]$ —zero when the value is saturated at $\lambda_y$ (the output is so strongly preferred or dispreferred that further changes don't matter), and $\lambda_y$ when the value is zero (the model's probability assignment provides no utility at all). Minimizing this loss pushes desirable outputs toward higher value (increasing their probability relative to the reference, above the KL baseline) and undesirable outputs toward higher value (decreasing their probability relative to the reference, below the KL baseline).

Why this loss form over alternatives:

Non-negativity: The $\lambda_y - v(x, y)$ construction ensures the loss is always non-negative, which is a convenient property for monitoring training (the loss has a natural floor at zero).
Direct utility maximization: Unlike DPO, which maximizes the log-likelihood of preferences under a Bradley-Terry model, KTO directly maximizes the prospect-theoretic utility of generations. The distinction matters because, as Theorem 4.2 shows, maximizing preference likelihood does not uniquely determine the human value distribution—two reward functions in the same equivalence class can have different value distributions but identical preference likelihoods. KTO's objective is aligned with what we actually care about: how much utility humans derive from the model's outputs.
Binary feedback compatibility: The loss depends only on the reward $r_\theta(x, y)$ and the label (desirable/undesirable). It does not require paired comparisons, Bradley-Terry assumptions, or even that desirable and undesirable outputs for the same input be drawn from comparable distributions. This is what enables KTO to work with naturally binary feedback sources.
Implicit robustness to noise: As shown in Proposition 4.1 (analyzed in the next subsection), the gradient decays to zero for extreme rewards. This means that outputs which are hard to learn from (because the model already strongly prefers or disprefers them) are automatically downweighted, providing robustness to mislabeled examples.

The KTO Gradient and the Automatic Curriculum Property

The gradient of the KTO loss with respect to the policy parameters $\theta$ reveals the mechanism by which KTO updates the model. Taking the derivative of Equation 8:

$\nabla_\theta \mathcal{L}_{\text{KTO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x, y \sim \mathcal{D}}\left[d(y) \cdot \lambda_y \cdot \sigma(\beta z) \cdot (1 - \sigma(\beta z)) \cdot \beta \cdot \nabla_\theta \log \pi_\theta(y|x)\right]$

where:

$d(y) = -1$ if $y$ is desirable and $+1$ if $y$ is undesirable
$z = r_\theta(x, y) - z_0$ is the deviation of the implied reward from the reference point
$\sigma(\beta z)$ is the sigmoid of the scaled deviation
$\lambda_y$ is $\lambda_D$ or $\lambda_U$ as appropriate
$\beta$ is the risk aversion parameter
$\nabla_\theta \log \pi_\theta(y|x)$ is the standard score function gradient (the direction in parameter space that increases the log-probability of $y|x$ )

What this computes, operationally: The gradient update is a weighted combination of score function gradients. For desirable outputs ( $d(y) = -1$ ), the gradient points in the direction $-\nabla_\theta \log \pi_\theta(y|x)$ , which increases the log-probability of $y|x$ —the model is pushed to make desirable outputs more likely. For undesirable outputs ( $d(y) = +1$ ), the gradient points in the direction $+\nabla_\theta \log \pi_\theta(y|x)$ , which decreases the log-probability of $y|x$ —the model is pushed to make undesirable outputs less likely. The magnitude of the update is modulated by three multiplicative factors:

$\lambda_y$ , the loss aversion coefficient, which controls the overall weight of this example class.
$\beta$ , which scales the entire gradient—higher $\beta$ means larger steps when the sigmoid is not saturated.
$\sigma(\beta z)(1 - \sigma(\beta z))$ , the derivative of the sigmoid, which is maximized at $z = 0$ (when the reward equals the reference point, $\sigma(0) = 0.5$ , and the product is $0.25$ ) and decays to zero as $|z| \to \infty$ (when the sigmoid saturates at 0 or 1).

Why this gradient form matters—Proposition 4.1 (the automatic curriculum): As the deviation $z = r_\theta(x, y) - z_0$ tends to $\pm \infty$ , the sigmoid derivative $\sigma(\beta z)(1 - \sigma(\beta z))$ tends to zero. This means:

If a desirable output already has a highly positive reward relative to the reference point ( $z \to +\infty$ ), the gradient vanishes—the model stops trying to further increase its probability. The output is already "good enough."
If an undesirable output already has a highly negative reward relative to the reference point ( $z \to -\infty$ ), the gradient also vanishes—the model stops trying to further decrease its probability. The output is already sufficiently suppressed.
If either type of output has a reward very far from the reference point in the wrong direction (a desirable output with very negative reward, or an undesirable output with very positive reward), the gradient also vanishes. These are outputs that are "too hard to learn from"—they are so misaligned with their labels that the model effectively ignores them.

Why this is a desirable property: The vanishing gradient for extreme rewards provides an automatic curriculum: the model focuses its learning on examples where it can make the most progress—those near the reference point, where the sigmoid derivative is large—and automatically downweights examples that are either already solved or too difficult. This provides implicit robustness to label noise: if a desirable output is mislabeled as undesirable (or vice versa), the model will initially try to push its reward in the wrong direction, but as the reward moves further from the reference point against the label, the gradient will decay, and the model will stop wasting capacity on this contradictory example. This is formalized in the paper's discussion of Theorem 4.3 (analyzed below), which shows that KTO handles intransitive preferences with better worst-case guarantees than DPO.

The potential downside—risk of underfitting: The same property that provides noise robustness can also cause the model to ignore genuinely hard-to-learn but correctly labeled examples. If the reference model places very low probability on a desirable output (so $r_\theta$ starts very negative), the gradient signal may be too weak to overcome this initial disadvantage, and the model may never learn to produce that output. The paper acknowledges this risk in Section 4.4: "KTO could end up ignoring data that is hard-to-learn but necessary to recover $r^*$ , potentially resulting in underfitting to complex distributions." The mitigation suggested is to use a lower $\beta$ (which makes the sigmoid less steep, broadening the region of non-zero gradient) and to train for more epochs.

Contrast with DPO gradient: In DPO, the gradient for a preference pair $(y_w, y_l)$ is:

$\nabla_\theta \mathcal{L}_{\text{DPO}} = -\beta \cdot \sigma(\beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} - \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}) \cdot (\nabla_\theta \log \pi_\theta(y_w|x) - \nabla_\theta \log \pi_\theta(y_l|x))$

The DPO gradient scales as $\sigma(-\Delta r)$ where $\Delta r = r_\theta(x, y_w) - r_\theta(x, y_l)$ is the reward gap. When the preferred output already has much higher reward than the dispreferred output, $\Delta r \gg 0$ and $\sigma(-\Delta r) \to 0$ , so the gradient vanishes—similar to KTO's saturation. However, the DPO gradient depends on the relative reward between the two outputs in a pair, while the KTO gradient depends on the absolute deviation of each output from the reference point. This is the key structural difference that allows KTO to work with unpaired binary feedback: each output's gradient is independent of any other output, requiring only the reference point $z_0$ as a shared baseline.

Hyperparameter Configuration: Risk Aversion, Loss Aversion, and Practical Guidelines

KTO introduces three hyperparameters beyond the standard learning rate and batch size: $\beta$ (risk aversion), $\lambda_D$ and $\lambda_U$ (loss aversion for desirable and undesirable outputs). The paper provides extensive empirical guidance for setting these, grounded in the behavioral interpretation of what each parameter controls.

$\beta$ —Risk Aversion (and Risk Seeking): In the original Kahneman-Tversky value function, the curvature parameter $\alpha$ controls risk aversion: $\alpha < 1$ means the value function is concave in gains (risk-averse—preferring a sure $50 to a 50% chance of$ 100) and convex in losses (risk-seeking—preferring a 50% chance of losing $100 to a sure loss of$ 50). In KTO, $\beta$ serves an analogous role by controlling the steepness of the logistic sigmoid.

Higher $\beta$ : The sigmoid transitions from 0 to 1 over a narrower range of reward deviations. This means the value saturates faster—small improvements in reward produce large increases in value when near the reference point, but the value plateaus quickly. Behaviorally, this corresponds to greater risk aversion in gains (the model is satisfied with moderate improvements and doesn't seek large ones) and greater risk-seeking in losses (the model is willing to accept large negative deviations because further decreases don't change the value much).
Lower $\beta$ : The sigmoid is shallower, transitioning gradually over a wider range. The value function is closer to linear near the reference point, meaning the model continues to receive meaningful gradient signal over a broader range of reward values. This corresponds to lower risk aversion—the model keeps pushing for larger improvements rather than saturating early.

Empirical settings from the paper (Section 4.2):

For larger models that have already undergone SFT: lower $\beta$ in the range $[0.01, 0.10]$ works better. The SFT model already produces reasonable outputs, so the alignment stage should make gradual, fine-grained adjustments rather than saturating quickly.
For smaller models undergoing KTO directly without SFT: higher $\beta$ in the range $[0.10, 1.00]$ works better. The model starts further from the reference and needs stronger gradient signal to make meaningful changes.
Default setting used in most experiments: $\beta = 0.1$ (same as the typical DPO $\beta$ ).
Table 1 recommends: $\beta = 0.05$ for Llama-3 8B with SFT+KTO, $\beta = 0.10$ for Llama-3 8B with KTO alone and for Qwen2.5 3B Instruct with SFT+KTO, and $\beta = 0.50$ for Qwen2.5 3B Instruct with KTO alone.

The relationship between KTO's $\beta$ and DPO's $\beta$ : In DPO, $\beta$ originates from the KL penalty coefficient in the RLHF objective (Equation 2) and controls how far $\pi_\theta$ can drift from $\pi_{\text{ref}}$ . In KTO, $\beta$ is explicitly introduced as a risk aversion parameter within the value function. However, in practice they have similar effects: higher $\beta$ in both methods means the model is more strongly anchored to the reference and makes more conservative updates. The paper uses the same $\beta = 0.1$ default for both DPO and KTO to ensure fair comparison.

$\lambda_D$ and $\lambda_U$ —Loss Aversion: These coefficients control the relative and absolute weighting of desirable versus undesirable examples. They are the direct analogue of $\lambda$ in the original Kahneman-Tversky value function, but split into two to handle class imbalance.

Default setting: $\lambda_D = \lambda_U = 1$ , giving equal maximum value to desirable and undesirable outputs. This is the loss-neutral setting—no asymmetry between gains and losses.
The effective ratio guideline (Equation 9): When the dataset is imbalanced, set $\lambda_D$ and $\lambda_U$ such that:

$\frac{\lambda_D n_D}{\lambda_U n_U} \in \left[1, \frac{4}{3}\right]$

where $n_D$ and $n_U$ are the counts of desirable and undesirable examples.

What this means in practice:

If the dataset has 1:1 ratio of desirable to undesirable examples, the default $\lambda_D = \lambda_U = 1$ satisfies the guideline (effective ratio = 1).
If 90% of desirable examples are discarded, leaving a 1:10 ratio ( $n_D = 1, n_U = 10$ with $\lambda_U = 1$ ), then $\lambda_D$ should be set between 10 and 13.33. The paper's experiment at this ratio uses $\lambda_D = 13.33$ .
The upper bound of $4/3$ encodes the empirical finding that giving slightly more total weight to desirable examples yields better performance, "ostensibly because producing good outputs is more important than avoiding bad outputs for success on most benchmarks."

The implicit assumption about convergence: The current configuration of $\lambda_D, \lambda_U$ implicitly assumes that upon convergence, all desirable outputs will yield positive rewards and all undesirable outputs will yield negative rewards. In this regime, $\lambda_D$ and $\lambda_U$ directly correspond to gain and loss sensitivity respectively. However, during training, a desirable output may temporarily have a negative reward (if the model initially assigns it low probability), and the same $\lambda_D$ applies regardless of whether $r_\theta - z_0$ is positive or negative. The paper acknowledges this limitation and suggests that a more sophisticated scheme—using different $\lambda_D$ values depending on the sign of $r_\theta - z_0$ —could further improve performance, but leaves this as future work.

Learning Rate: The paper finds that "the performance of an aligned model is more sensitive to the learning rate than any other hyperparameter" (Section 4.2). Critically, the optimal learning rate for KTO is usually 2× to 10× the optimal learning rate for DPO. The reason: "since the reference-adjusted reward tends to be much smaller in magnitude for KTO, one needs to use a more aggressive learning rate to compensate." In DPO, the gradient depends on the difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ , which can be large even when individual rewards are small. In KTO, each gradient depends only on the deviation from the reference point, which is typically smaller in magnitude because the KL divergence term $z_0$ partially cancels the individual rewards. A larger learning rate compensates for these smaller gradient magnitudes.

Specific settings from the paper:

Default DPO learning rate: 5e-7 with RMSProp (from Rafailov et al., 2023)
Recommended KTO learning rate: start at 5e-6 with AdamW
Table 1 uses learning rate 5e-6 for all KTO configurations
The paper's main experiments use the DPO default learning rate with RMSProp for all methods to ensure apples-to-apples comparison, even though this is suboptimal for KTO

Batch Size: "KTO needs a microbatch size $\geq 2$ to estimate the reference point in a single step" because the mismatched-pair estimator requires at least two examples to form a cross-pair. The paper uses an effective batch size of 32 in all experiments and recommends 8–128 in general.

Memory-Efficient Variant (No Reference Model): The paper explores a variant of KTO that avoids storing the reference model in memory by assuming $\pi_{\text{ref}}$ is uniform. This simplifies the reward to:

$r_\theta(x, y) - z_0 \approx \log \pi_\theta(y|x) - H(\pi_\theta(\cdot|x))$

where $H$ is the entropy of the policy. This variant requires setting $\lambda_D = 1.75$ (higher than default) to compensate for the loss of the reference model's grounding. It is more sensitive to the loss aversion hyperparameters than standard KTO, but strictly outperforms ORPO (Hong et al., 2024), another reference-free method, while using less memory (since a batch of $m$ KTO outputs is smaller than a batch of $m$ preferences, which is $2m$ outputs).

Theoretical Foundations: Why KTO Works Despite Weaker Signal

The paper provides three theoretical results that explain KTO's surprising empirical performance—specifically, why it matches or exceeds DPO despite learning from a binary signal that is strictly weaker than pairwise preferences.

Theorem 4.2: Preference Likelihood Maximization ≠ Human Utility Maximization. This theorem addresses a subtle but important distinction. DPO maximizes the log-likelihood of observed preferences under a Bradley-Terry model. It seems natural to assume that this also maximizes human utility—after all, if the model assigns higher reward to preferred outputs, shouldn't humans be happier? The theorem shows that this intuition is incorrect.

Theorem 4.2 (restated): Assuming the value function is logistic, for a reward function $r^*_a$ that maximizes the RLHF objective (Equation 2), there exists a reward function in its equivalence class (i.e., $r^*_b(x, y) = r^*_a(x, y) + h(x)$ for some function $h(x)$ ) that induces the same optimal policy $\pi^*$ and the same Bradley-Terry preference distribution but a different human value distribution.

What this means, operationally: Two reward functions that differ only by an input-specific shift $h(x)$ —for example, $r^*_b(x, y) = r^*_a(x, y) + 100$ for all outputs of input $x$ —will produce exactly the same preferences (since the shift cancels in the difference $r(x, y_w) - r(x, y_l)$ ), exactly the same optimal policy $\pi^*$ (since the shift is absorbed into the partition function $Z(x)$ ), but different prospect-theoretic values (since the shift changes the deviation from the reference point for every output of input $x$ ). Maximizing preference likelihood leaves the input-specific shift $h(x)$ undetermined, meaning there are infinitely many reward functions—and infinitely many human value distributions—consistent with the same observed preferences and the same optimal policy. DPO converges to one of them (determined by initialization and optimization dynamics), but not necessarily the one that maximizes human utility.

Why this matters: KTO, by directly maximizing a specific parameterization of human utility (the Kahneman-Tversky value function), optimizes a more targeted objective. It doesn't try to infer the "true" reward from preferences and then maximize it—it directly shapes the policy to produce outputs that have high prospect-theoretic value. The paper suggests this explains why "the margin between KTO and DPO is even bigger in human evaluations than it is in automated LLM-as-a-judge evaluations" (Appendix D): KTO is optimizing for what humans actually experience, not for what a preference model predicts.

The proof sketch (from Appendix B): The proof uses Taylor series expansions. The human value of $r^*_a(x, y)$ expanded around 0 is:

$\sigma(0) + \sigma'(0)(r^*_a(x, y) - z_0) + \frac{\sigma''(0)}{2}(r^*_a(x, y) - z_0)^2 + ...$

The human value of $r^*_b(x, y) = r^*_a(x, y) + h(x)$ expanded around $h(x)$ is:

$\sigma(h(x)) + \sigma'(h(x))(r^*_a(x, y) - z_0) + \frac{\sigma''(h(x))}{2}(r^*_a(x, y) - z_0)^2 + ...$

Since $\sigma$ is strictly monotonic, for these series to be equal at all $x$ , we must have $h(x) = 0$ . If $h(x) \neq 0$ , the value distributions differ—the first-order sensitivity $\sigma'(h(x))$ differs from $\sigma'(0)$ , and the baseline value $\sigma(h(x))$ differs from $\sigma(0)$ . Thus two reward functions in the same equivalence class generally induce different human value distributions.

Theorem 4.3: KTO Has Better Worst-Case Behavior Under Intransitive Preferences. Real-world human feedback often contains contradictions—different annotators prefer different outputs, or the same annotator's preferences are inconsistent across time. Such intransitivity poses a challenge for preference-based methods because the Bradley-Terry model assumes a consistent underlying reward function. Theorem 4.3 analyzes a simple case of contradictory feedback and shows that KTO is more robust.

Theorem 4.3 (restated): For input $x$ with outputs $\{y_a, y_b\}$ , let dataset $\mathcal{D}$ comprise contradictory preferences $y_a \succ y_b$ and $y_b \succ y_a$ in proportion $p \in (0.5, 1)$ and $(1-p) \in (0, 0.5)$ respectively. If $p^{1/\beta}\pi_{\text{ref}}(y_a|x) < (1-p)^{1/\beta}\pi_{\text{ref}}(y_b|x)$ , then the optimal DPO policy is more likely to produce the minority-preferred $y_b$ ; the optimal KTO policy will strictly produce the majority-preferred $y_a$ for a loss-neutral value function ( $\lambda_D = \lambda_U$ ).

What this means, operationally: Suppose 60% of annotators prefer $y_a$ and 40% prefer $y_b$ . Under certain conditions—specifically, when the reference model sufficiently favors $y_b$ relative to $y_a$ (weighted by the preference proportions raised to $1/\beta$ )—the DPO-optimal policy will actually make $y_b$ more likely than $y_a$ , siding with the minority. This happens because DPO balances the preference likelihood against the KL penalty from the reference model: if the reference model strongly prefers $y_b$ , moving to $y_a$ is expensive in KL terms, and the 60/40 split doesn't provide enough likelihood improvement to justify the cost. KTO, in contrast, treats each output independently: $y_a$ appears as a desirable output in 60% of examples, $y_b$ in 40%, and since the loss-neutral value function gives equal weight to each example, the model simply learns to produce $y_a$ more often. The majority preference deterministically wins.

The condition for DPO's failure: $p^{1/\beta}\pi_{\text{ref}}(y_a|x) < (1-p)^{1/\beta}\pi_{\text{ref}}(y_b|x)$ . This inequality is more likely to hold when:

$p$ is close to 0.5 (the preference split is nearly even)
$\beta$ is small (the KL penalty is weak, so the reference model's prior is weighted more heavily)
$\pi_{\text{ref}}(y_b|x) \gg \pi_{\text{ref}}(y_a|x)$ (the reference model has a strong prior toward the minority-preferred output)

In such cases, DPO effectively defers to the reference model's prior over the majority preference. KTO does not suffer from this failure mode because it does not optimize a paired objective—each output's gradient depends only on its own label and the shared reference point, not on a comparison with another output.

The proof sketch (from Appendix B): For DPO, the total loss for $x$ with contradictory preferences is:

$\mathcal{L}_{\text{DPO}}(x) = p(-\log \sigma(u)) + (1-p)(-\log \sigma(-u))$

where $u = \beta(r_\theta(x, y_a) - r_\theta(x, y_b))$ . Setting the derivative with respect to $u$ to zero yields the optimum $u^* = \sigma^{-1}(p) = \log\frac{p}{1-p}$ , from which the optimal policy ratio $\frac{\pi_\theta^*(y_a|x)}{\pi_\theta^*(y_b|x)}$ can be derived. When the inequality condition holds, this ratio is less than 1, meaning $y_b$ is more probable.

For KTO, with loss-neutral value function ( $\lambda_D = \lambda_U$ ), the total loss is:

$\mathcal{L}_{\text{KTO}}(x) = \lambda_D + \lambda_D((1-2p)\sigma(u_a) + (2p-1)\sigma(u_b))$

where $u_a = \beta(r_\theta(x, y_a) - \mathbb{E}_Q[r_\theta])$ and $u_b = \beta(r_\theta(x, y_b) - \mathbb{E}_Q[r_\theta])$ . Since $p > 0.5$ , the coefficient on $\sigma(u_a)$ is negative and on $\sigma(u_b)$ is positive, meaning the loss decreases as $r_\theta(x, y_a)$ increases and as $r_\theta(x, y_b)$ decreases. The optimal policy therefore deterministically produces $y_a$ .

Why this matters: Most publicly available preference datasets (SHP, OpenAssistant, Anthropic-HH) contain feedback from many different humans whose preferences are not perfectly consistent—they contradict each other to some extent. Theorem 4.3 suggests that KTO handles this intransitivity more robustly than DPO, which helps explain why KTO matches or exceeds DPO on these datasets despite learning from a weaker signal. The paper explicitly connects this to the empirical results: "Most publicly available preference datasets contain noisy feedback from many different humans whose preferences likely contradict to some extent, which explains why KTO was able to match or exceed DPO performance in our experiments" (Section 4.5).

Data Handling: Converting Preferences to Binary Feedback

Since the canonical academic alignment datasets (Anthropic-HH, OpenAssistant, SHP) are in preference format, the paper needs a procedure to convert them to binary feedback for KTO. The procedure is simple:

Standard conversion (used in most experiments): Given a preference pair $(x, y_w, y_l)$ where $y_w \succ y_l$ , create two examples: $(x, y_w, \text{desirable})$ and $(x, y_l, \text{undesirable})$ . This means each preference pair yields two independent training examples for KTO, doubling the effective dataset size compared to DPO (which uses the pair as a single example). The paper explicitly acknowledges that this assumption—that the preferred output is drawn from the "desirable distribution" and the dispreferred from the "undesirable distribution"—is "a naive assumption, made for the sake of simplicity, and a more complex deconstruction of preferences into binary feedback would likely yield better results."

One-y-per-x conversion (used for ablation): To test whether KTO benefits from the paired structure of preference data (beyond just having more examples), the paper also experiments with discarding one output from each pair, keeping only one $y$ per $x$ . This reduces the amount of training data but ensures that no pairing information remains. In the Mistral-7B experiment on OpenAssistant (Table 3), this reduces training data by 72% but KTO still outperforms DPO on the full dataset, confirming that KTO's success is not an artifact of the preference data structure.

Imbalanced data experiments (Figure 5): The paper further tests robustness by randomly discarding increasingly large fractions of the desirable examples while keeping all undesirable examples. At the extreme, with only 10% of desirable examples remaining (a 1:10 ratio), KTO still matches DPO performance after adjusting $\lambda_D$ to satisfy the effective ratio guideline. This demonstrates that KTO can work with naturally imbalanced binary feedback—for example, in a production system where positive feedback (thumbs up) is much rarer than negative feedback (thumbs down), or vice versa.

Handling scores and ratings: The paper briefly discusses how to extend KTO to richer feedback formats. For score-based feedback, the simplest approach is to construct a weighting function where high-magnitude scores receive higher weight, and examples with scores above or below some threshold are labeled desirable or undesirable. More sophisticated score-based HALOs could be constructed from first principles, but this is left to future work.

Summary of Design Choices and Their Justifications

Logistic sigmoid over power-law value function: Boundedness prevents over-optimization; smooth gradients ensure numerical stability; qualitative shape (concave gains, convex losses) matches prospect theory.
Separate $\lambda_D$ and $\lambda_U$ over single $\lambda$ : Enables handling of class imbalance; allows practitioners to control the relative importance of producing good outputs vs. avoiding bad outputs.
KL divergence as reference point over fixed baseline or single-output baseline: Forces selective improvement—the model must learn distinguishing features of desirable outputs rather than uniformly increasing all probabilities.
Mismatched-pair estimator over exact KL or matched-pair estimator: Computationally efficient (no sampling from $\pi_\theta$ ); avoids bias from using canonically good/bad outputs; mimics human availability heuristic.
Detached reference point (no backpropagation through $z_0$ ): Prevents pathological incentive to manipulate the baseline; stabilizes training.
Binary feedback over preferences: Matches the natural format of most real-world feedback; enables use of abundant, cheap data sources; the theoretical results (Theorems 4.2, 4.3) show that maximizing human utility from binary feedback can be more robust than maximizing preference likelihood.
Automatic curriculum via sigmoid saturation: Provides implicit robustness to label noise; focuses learning on examples where the model can make the most progress; potential underfitting risk can be mitigated by lower $\beta$ and more training epochs.

4. Key Insights and Innovations

Innovation 1: Reframing Alignment Objectives as Prospect-Theoretic Value Functions (the HALO Framework)

The paper's most fundamental contribution is not KTO itself but the conceptual reframing of alignment objectives through the lens of prospect theory. Before this work, the dominant mental model for understanding alignment losses was reward-centric: RLHF learns a reward function from preferences and optimizes against it; DPO cleverly reparameterizes the reward to avoid explicit learning but still inherits the reward-maximization framing via the Bradley-Terry model. Success was attributed to the quality of the reward signal—better reward models produce better alignment.

This paper proposes a different explanation: alignment objectives work because they incorporate the same cognitive biases that prospect theory identifies in human decision-making. Definition 3.4 formalizes this as the Human-Aware Loss (HALO) framework, which decomposes any alignment loss into three components: an implied reward (the change in surprisal relative to a reference model), a reference point (the human's baseline expectation), and a value function that maps reward deviations to subjective value with prospect-theoretic properties (concavity in gains, non-decreasing everywhere). Theorem 3.5 then proves that both DPO and PPO-Clip are HALOs—their loss functions can be expressed as specific choices of these three components.

This is a diagnostic contribution, not just a taxonomic one. It explains why some alignment methods work and others don't, independent of the data they use. CSFT and SLiC are not HALOs—they cannot be expressed with a reference point that depends on the policy—and they empirically underperform at every scale (Figure 2). The offline PPO variant with dummy +1/-1 rewards, despite having no learned reward model, is a HALO and performs competitively with DPO up to 13B parameters. The implication is striking: the inductive bias of the loss function matters more than the quality of the reward signal. A binary signal (+1/-1) fed into a HALO-structured loss can match a learned reward model fed into a non-HALO loss, as long as the loss function has the right shape.

The paper explicitly positions this against the dominant view in the field, citing Casper et al. (2023) as representative of the conventional wisdom that reward learning is critical. The HALO framework says otherwise: reward learning is one way to get good alignment, but it is neither necessary nor sufficient. What matters is that the loss function thinks about outputs the way humans do—evaluating them relative to a reference point, with diminishing sensitivity to extreme goodness or badness, and with potential asymmetry between gains and losses. This is a fundamental reframing of the alignment problem, not an incremental improvement over DPO.

Evidence for this innovation is not a single table but the architecture of the paper itself: Figure 1 visualizes the value functions implied by DPO, PPO-Clip, and the Kahneman-Tversky model, showing their shared qualitative properties; Figure 2 shows that HALOs (DPO, offline PPO) match or outperform non-HALOs (CSFT, SLiC) at every scale; and the proof of Theorem 3.5 in Appendix B establishes that the HALO definition is not ad hoc—it captures exactly the methods that work.

Innovation 2: Binary Feedback Can Match Preferences—When the Loss Function Has the Right Inductive Biases

The second major insight is empirical and contravenes a reasonable default assumption: that preference data is necessary for strong alignment because it provides a richer signal than binary feedback. The paper demonstrates that this assumption is false under the right conditions.

KTO matches or exceeds DPO at scales from 1B to 30B (Figure 3) while using only a binary desirable/undesirable label per output. This is not merely a "weaker signal compensated by more data" story—though that was the original motivation. The paper shows that even when KTO is restricted to one output per input (the one-y-per-x ablation in Table 3), reducing training data by 72% compared to the full DPO dataset, it still outperforms DPO on Mistral-7B aligned on OpenAssistant. And in the data imbalance experiments (Figure 5), KTO matches DPO while using up to 90% fewer desirable examples, with appropriate adjustment of $\lambda_D$ .

What makes this a genuine innovation rather than an obvious result is that prior attempts to use binary feedback for alignment—CSFT (Korbak et al., 2023), unlikelihood training (Welleck et al., 2019)—significantly underperformed preference-based methods. The field had reasonably concluded that binary feedback was insufficient. KTO shows that the failure was not in the feedback format but in the loss functions used with it. CSFT is not a HALO; KTO is. The difference is not the data—it's the prospect-theoretic structure of the objective.

This finding has substantial practical significance beyond the benchmark numbers. Binary feedback (thumbs up/down, accepted/rejected, flagged/clean) is abundant in production systems where pairwise preferences are impractical to collect. The paper's results suggest that these naturally occurring signals, when paired with an appropriate loss function, can drive alignment as effectively as carefully curated preference data. This lowers the barrier to continuous alignment in deployed systems and opens the door to personalization at scale, since binary feedback can be collected per-user without the cognitive overhead of pairwise comparisons.

The distinction between incremental and fundamental here is nuanced. Using binary feedback for alignment is not new—CSFT did it. What is new is the demonstration that binary feedback can reach parity with preferences when the loss function encodes prospect-theoretic inductive biases. This shifts the bottleneck from data collection format to loss function design, which is a conceptual change in how the field should think about the alignment data pipeline.

Innovation 3: Preference Likelihood Maximization ≠ Human Utility Maximization

This is the paper's deepest theoretical contribution, and it is a negative result about DPO rather than a positive result about KTO. Theorem 4.2 proves that two reward functions in the same equivalence class—differing only by an input-specific shift $h(x)$ —induce the same optimal policy and the same Bradley-Terry preference distribution, but different human value distributions. In plain terms: maximizing the likelihood of observed preferences does not uniquely determine what humans actually value.

This matters because DPO's entire framing is built on maximizing preference likelihood. The implicit assumption is that if the model correctly captures which outputs humans prefer, it will produce outputs that maximize human utility. Theorem 4.2 shows this is mathematically false: the preference likelihood objective leaves an input-specific degree of freedom undetermined, and different choices of this degree of freedom (determined by initialization and optimization dynamics, not by the data) lead to different human value distributions. DPO converges to one of them, but not necessarily the one that maximizes utility.

This is a fundamental conceptual advance because it separates two things the field had implicitly conflated: inferring preferences and maximizing utility. The Bradley-Terry model is a model of choice, not of experienced value. Humans might prefer A over B even when both have negative utility, or when the difference in utility is negligible but one option is slightly better. Maximizing the probability of correct pairwise choices does not directly optimize for the utility humans experience from the outputs they receive. KTO, by contrast, directly maximizes a parameterized model of human utility (the Kahneman-Tversky value function), targeting what we actually care about rather than a proxy.

The paper provides suggestive evidence for this distinction in Appendix D: the margin between KTO and DPO is larger in human evaluations (72.9% vs. 62.1% winrate over SFT targets) than in GPT-4 evaluations (65.2% vs. 60.0%), consistent with KTO optimizing for something closer to genuine human utility rather than a preference proxy that an LLM judge can also approximate.

This innovation is theoretical—it does not directly improve metrics—but it changes how we should think about alignment objectives. It suggests that the field should move beyond "can we correctly predict which output humans prefer?" to "can we model how much utility humans derive from outputs?" The HALO framework and KTO are first steps in that direction.

Innovation 4: Automatic Robustness to Noisy and Intransitive Feedback via Gradient Saturation

The paper identifies and formalizes a property of KTO that is not present in DPO: the gradient naturally vanishes for examples that are too hard or too easy to learn from (Proposition 4.1). As the implied reward deviation $r_\theta(x, y) - z_0$ tends to $\pm\infty$ , the sigmoid derivative $\sigma(\beta z)(1 - \sigma(\beta z))$ decays to zero, meaning the model stops updating on examples that are extremely misaligned with their labels.

This is significant as a robustness mechanism, not just a mathematical curiosity. Real-world human feedback is noisy—annotators make mistakes, preferences are inconsistent, and some outputs are genuinely ambiguous. In DPO, a mislabeled preference pair (where the "preferred" output is actually worse) can actively damage the model, because the gradient pushes in the wrong direction with magnitude proportional to how wrong the current policy is. In KTO, a mislabeled example initially pushes the model in the wrong direction, but as the reward moves further from the reference point against the (incorrect) label, the gradient decays, and the model effectively ignores the example. This is an automatic curriculum: the model focuses capacity on examples it can learn from and downweights contradictions.

Theorem 4.3 formalizes the benefit of this property for one important case: intransitive preferences. When different annotators disagree (e.g., 60% prefer $y_a$ , 40% prefer $y_b$ ), DPO can—under certain conditions on the reference model and $\beta$ —side with the minority, producing the output that fewer humans preferred. KTO with loss-neutral $\lambda$ deterministically sides with the majority. The proof shows this is a direct consequence of KTO's independent per-output gradients versus DPO's paired objective that balances preference likelihood against KL cost from the reference model.

This is an incremental theoretical advance with fundamental practical implications. Noise and intransitivity are endemic in real human feedback—they are not edge cases. Most public preference datasets (SHP, OpenAssistant, Anthropic-HH) aggregate feedback from diverse annotators whose preferences contradict. The paper explicitly argues that this explains why KTO matches or exceeds DPO on these datasets (Section 4.5): KTO's robustness to contradiction compensates for the weaker per-example signal. The fact that KTO also outperforms DPO on UltraFeedback (Table 2)—a synthetically generated dataset with its own noise patterns—suggests the robustness advantage is not specific to human annotation noise.

The tradeoff, which the paper is candid about, is potential underfitting: the same mechanism that ignores noisy examples can also ignore genuinely hard-to-learn but correctly labeled examples. This is not a solved problem, but it is a well-characterized one—the paper provides mitigations (lower $\beta$ , more epochs) and frames it as a deliberate design choice rather than an undiscovered bug.

Innovation 5: The Kahneman-Tversky Value Function as a Pluggable Component for Alignment Objective Design

The final innovation is methodological rather than theoretical: the paper demonstrates that value functions from the prospect theory literature can be directly adapted as components of alignment objectives, and that the specific choice of value function matters substantially for downstream performance.

This is more than "we tried a different activation function." The ablation experiments in Table 2 (middle) systematically vary the shape of the value function while keeping the rest of the KTO framework fixed. The results are striking: making the value function concave everywhere (replacing $1 - \sigma(\cdot)$ with $-\log \sigma(\cdot)$ , which is the DPO value function shape) causes a 9.4-point drop on BBH; making it risk-neutral (identity function) causes a collapse to 6.1% on BBH; removing the reference point $z_0$ entirely causes a 3.6-point drop on BBH. These are not small differences—the specific prospect-theoretic shape (concave in gains, convex in losses, reference-dependent) is essential for performance.

This is a design principle contribution: it suggests that the field should treat the value function as a deliberate design choice informed by what we know about human utility, not as an incidental detail of the optimization algorithm. Just as prospect theory identified that different individuals have different value function parameters ( $\alpha$ and $\lambda$ vary across people; Tversky & Kahneman, 1992), and that different domains may call for different value function shapes (Gurevich et al., 2009), the paper argues that different alignment settings may call for different HALOs. The concluding claim—"there is no one HALO that is universally superior; the best HALO depends on the inductive biases appropriate for a given setting"—is not false modesty. It follows directly from the evidence that value function shape matters and that the optimal shape likely depends on the noise characteristics, intransitivity levels, and desirability distributions of the feedback data.

This is a fundamental shift from the status quo, where alignment objectives are designed around mathematical convenience (the Bradley-Terry model) or algorithmic considerations (clipping for stability). The paper opens a design space where the value function is chosen based on what we know about human perception and decision-making, drawing on decades of behavioral economics research. KTO is one point in this space—the Kahneman-Tversky value function for monetary gambles, adapted to text. The paper explicitly calls for future work to identify the best value functions for different domains and individuals (Section 5), suggesting this is the beginning of a research program, not its conclusion.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary experiments use a combination of three preference datasets: Anthropic-HH (Ganguli et al., 2022), OpenAssistant (Köpf et al., 2023), and SHP (Ethayarajh et al., 2022). These are standard academic alignment datasets containing pairwise human preferences. For the Zephyr experiments in Table 2, the paper uses UltraFeedback (Cui et al., 2023), a synthetically generated preference dataset. All datasets are converted to binary feedback for KTO by treating the preferred output $y_w$ as desirable and the dispreferred output $y_l$ as undesirable, yielding two examples per preference pair. The total data volume across the combined [HH, OASST, SHP] corpus is not explicitly stated, but the one-y-per-x ablation in Table 3 reduces training data by 72% (from using both outputs per input to using only one), implying the original dataset contains a substantial number of examples. Open-ended evaluation uses prompts from the OpenAssistant test set, with 214 comparisons for human evaluation and an unspecified number for GPT-4 evaluation.
Base model(s). The paper uses two model families to assess scaling behavior: Pythia-{1.4B, 2.8B, 6.9B, 12B} (Biderman et al., 2023) and Llama-{7B, 13B, 30B} (Touvron et al., 2023). Pythia provides a controlled scaling suite where all models share the same architecture and training data (only model size varies), enabling clean isolation of how alignment effectiveness scales with parameter count. Llama provides stronger base models at larger scales. Later experiments use Mistral-7B (Jiang et al., 2023) and derivatives (specifically Zephyr-β-SFT; Tunstall et al., 2023), along with Llama-3 8B (Dubey et al., 2024) and Qwen2.5 3B Instruct (Yang et al., 2024) for hyperparameter recommendations in Table 1. Llama-2 is notably absent because it lacks a 30B model in the released suite, hence the use of the original Llama family. The choice of Pythia for scaling analysis is deliberate: it enables comparisons within a model family where the only variable is parameter count, avoiding confounds from architecture changes or training data differences.
Metrics. The primary metric is GPT-4-0613-judged winrate against the SFT target output, expressed as a percentage (winrate – 50% is reported as "winrate above chance" in Figure 2). GPT-4 is asked to judge whether the aligned model's response is better than the SFT target with respect to helpfulness, harmlessness, and conciseness—a protocol validated against human judgments in Appendix D (68.7% agreement for KTO, 65.9% for DPO). For the Zephyr-UltraFeedback experiments (Table 2), the paper reports exact match (EM) accuracy on MMLU (0-shot), GSM8K (8-shot with chain-of-thought), HumanEval (0-shot pass@1), BBH (3-shot with chain-of-thought), TydiQA (1-shot), and AlpacaEval 2 winrate (GPT-4 Turbo as judge). For human evaluation in Appendix D, 256 prompts from OpenAssistant are sampled, reduced to 214 after filtering out domain-specific questions (e.g., coding), and binary preferences between aligned model outputs and SFT targets are collected from third-party annotators with 90% binomial confidence intervals reported.
Baselines. The paper compares against five baselines:
1. DPO (Rafailov et al., 2023): The primary preference-based baseline, using $\beta = 0.1$ with the implementation from the original paper.
2. CSFT (Conditional SFT) (Korbak et al., 2023): Prepends control tokens ⟨|good|⟩, ⟨|bad|⟩ during training and uses ⟨|good|⟩ at inference.
3. SLiC (Zhao et al., 2023): Combines a max-margin preference loss with a language modeling regularization term. Instead of sampling from $\pi_{\text{ref}}$ for the regularization (as in the original paper, which the authors note is slow), the cross-entropy loss is applied directly to the SFT data.
4. PPO (offline): An offline variant of PPO-Clip (Schulman et al., 2017) with dummy +1/-1 rewards for preferred and dispreferred outputs respectively. Never updates $\pi_{\text{old}}$ (keeping it as $\pi_{\text{ref}}$ ), clips probability ratios asymmetrically at $[0.25, 4.0]$ , includes a KL penalty estimated as the mean difference in log-probabilities of actual output tokens, and uses a detached 3-layer MLP value head. Described in Section 3.3 and Appendix C.
5. ORPO (Hong et al., 2024): A reference-free preference optimization method, compared against KTO's memory-efficient (reference-free) variant in Table 2. Additional baselines appearing in specific experiments: Mistral-7B-Instruct (the official instruction-tuned model; Jiang et al., 2023), Zephyr-β-SFT (Tunstall et al., 2023), and the SFT model itself (used as the target for winrate comparisons across all experiments).
Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time. Instead, fairness between KTO and DPO is ensured by: (1) using identical hardware and training configurations except for loss-specific hyperparameters; (2) converting each preference pair into two binary examples for KTO, meaning KTO processes twice as many training examples as DPO on the same underlying preference data—this is acknowledged as a data efficiency advantage of the binary format, not a methodological flaw, since the paper's claim is precisely that binary feedback can be collected at higher volume than preferences; (3) the one-y-per-x ablation (Table 3) controls for this by discarding one output per input, giving KTO less data than DPO (72% reduction) while still testing on the same information content. This is the key fairness control: KTO with one-y-per-x processes fewer examples than DPO but still matches its performance. All models are trained for exactly one epoch on UltraFeedback in the Zephyr experiments for an apples-to-apples comparison.
Cross-validation / statistical protocol. For the winrate experiments (Figures 2, 3), GPT-4 judgments are treated as Bernoulli trials, with error bars showing 90% binomial confidence intervals. Statistical significance is assessed using the Holm-Bonferroni correction for multiple comparisons (Holm, 1979). For the Zephyr experiments (Table 2), no cross-validation is reported—all methods are evaluated on standard benchmark test sets. Human evaluation (Appendix D) uses 90% binomial confidence intervals on 214 comparisons per method. The data imbalance experiments (Figure 5) use a fixed random seed for subsampling but do not report multiple random seeds or cross-validation over subsamples. No confidence intervals are reported for the benchmark evaluations in Table 2 beyond the winrate metrics.

Main Quantitative Results

HALOs vs. Non-HALOs: The Inductive Bias Matters (Figure 2)

Headline finding: HALOs (DPO, offline PPO) match or outperform non-HALOs (CSFT, SLiC) at every model scale tested, and only HALO-aligned models can match or exceed the SFT baseline at 13B+ parameters.

Figure 2 reports the GPT-4-0613-judged winrate above chance (winrate – 50%) against SFT targets for all four methods across Pythia-{1.4B, 2.8B, 6.9B, 12B} and Llama-{7B, 13B, 30B}. The SFT targets are subsets of the preferred outputs $\{y_w\}$ from the training data—they represent a strong baseline because many are human-written. Achieving above-chance winrate means the aligned model produces outputs judged better than these human demonstrations.

Scale-dependent effects:

At Pythia-1.4B through 6.9B: All methods cluster near or slightly below chance. The paper states that "up to a scale of 7B parameters, alignment provides virtually no gains over SFT alone," attributing this to the base models not being sufficiently performant for alignment to yield measurable improvements when the SFT data distribution is similar to the preference data.
At Pythia-12B and Llama-7B: HALOs begin to separate from non-HALOs. DPO and offline PPO achieve winrates near chance, while CSFT and SLiC remain below chance.
At Llama-13B and Llama-30B: The gap widens. DPO achieves a winrate above chance (positive values in Figure 2), offline PPO matches DPO at Llama-13B but falls behind at Llama-30B. CSFT and SLiC remain below chance at all scales. The gap between HALOs and non-HALOs is only statistically significant ( $p < 0.05$ ) at 13B+ parameters after Holm-Bonferroni correction.

The offline PPO surprise: Despite using only dummy +1/-1 rewards (no learned reward model, no online sampling), offline PPO performs as well as DPO for all models except Llama-30B. The paper presents this as evidence that "even the simplest rewards can prove useful when used in a loss function that has the right inductive bias," directly challenging the conventional emphasis on reward learning (citing Casper et al., 2023). However, offline PPO "still suffers from hyperparameter sensitivity and training instability, albeit not to the same extent as traditional RLHF," motivating the search for a more principled HALO (KTO).

Critical nuance: The claim that "only HALO-aligned models match the baseline" is true at 13B+ but the absolute gains are modest—winrates just above 50% (i.e., just above chance). This means aligned models are only slightly better than the SFT targets, which are themselves strong baselines (human-written or human-preferred outputs). The paper frames this as a difficult evaluation (Section 3.3: "this is difficult given that many of the target sequences are human-written"), but it also means the practical improvement from alignment over SFT is small in absolute terms on this metric.

KTO Matches or Exceeds DPO at All Scales (Figure 3)

Headline finding: SFT+KTO is competitive with SFT+DPO at scales from 1B to 30B; KTO alone (without SFT) is significantly better than DPO alone for Llama-{7B, 13B, 30B}, with the gap statistically significant ( $p < 0.01$ ) at 7B and 30B after Holm-Bonferroni correction.

Figure 3 re-runs the Figure 2 winrate evaluation with KTO added. The key comparisons:

SFT+KTO vs. SFT+DPO: The two curves largely overlap across all model scales. For Pythia models, there is "no significant difference between the two," while for Llama models, SFT+KTO is marginally ahead at 7B and 13B but the gaps are not flagged as statistically significant. The paper's phrasing—"despite the model learning from a weaker signal"—emphasizes that this is competitive performance, not dominance.
KTO alone vs. DPO alone: Without SFT preprocessing, KTO significantly outperforms DPO at Llama-7B and Llama-30B ( $p < 0.01$ ), and is competitive at Llama-13B. For Pythia models, the two are indistinguishable. This is a striking result: KTO without SFT performs as well as or better than DPO without SFT, even though SFT is generally considered essential for DPO to work well (Rafailov et al., 2023).
The SFT-independence of KTO: At Llama-13B and Llama-30B, KTO alone is competitive with SFT+KTO. This is unique among the tested methods—"KTO alone matches the performance of SFT+DPO and is significantly better than DPO alone" (Section 4.3). The paper attributes this to KTO keeping average response length roughly constant without SFT, whereas "running DPO without SFT prior causes the response length to increase dramatically" (Figure 4), with DPO models tending to "ramble and hallucinate entire conversations."

Sample generations (Table 6): The paper provides qualitative examples for Llama-30B on a show-recommendation prompt. The SFT answer is "not helpful" (generic statement about Breaking Bad being good). The SFT+DPO answer "hallucinates multiple turns of the conversation" and had to be truncated for the paper. The SFT+PPO (offline) answer is helpful but provides only one recommendation. SFT+KTO is described as "succinct and provides multiple options" (recommending Narcos, Sons of Anarchy, and Better Call Saul).

KTO on Generative Benchmarks: Strong Gains on Reasoning Tasks (Table 2)

Headline finding: Aligning Zephyr-β-SFT on UltraFeedback with KTO instead of DPO improves GSM8K accuracy by 13.5 points (40.0 → 53.5), with substantial gains on BBH (44.1 → 52.6) and moderate gains on MMLU (58.2 → 58.6). KTO outperforms all baselines on every benchmark except TydiQA.

Table 2 (top section) reports benchmark performance after exactly one epoch of alignment on UltraFeedback:

Method	MMLU (EM)	GSM8K (EM)	HumanEval (pass@1)	BBH (EM)
SFT (Zephyr-β-SFT)	57.2	39.0	30.1	46.3
DPO	58.2	40.0	30.1	44.1
ORPO ( $\lambda=0.1$ )	57.1	36.5	29.5	47.5
KTO ( $\beta=0.1, \lambda_D=1$ )	58.6	53.5	30.9	52.6
KTO (one-y-per-x)	58.0	50.0	30.7	49.9

The 13.5-point gain on GSM8K (a mathematical reasoning dataset) is the headline number. Notably, this is an 8-shot chain-of-thought evaluation, meaning the model must generate coherent multi-step reasoning to arrive at the correct answer. DPO provides only a 1.0-point improvement over SFT (39.0 → 40.0), while KTO provides 14.5 points (39.0 → 53.5). The paper does not analyze why GSM8K benefits so disproportionately from KTO—this is presented as an empirical finding rather than a theoretically explained one.

On BBH (BigBench-Hard, 3-shot chain-of-thought), KTO achieves 52.6 versus DPO's 44.1 (8.5-point gain). This is notable because BBH is a diverse reasoning benchmark; the gain suggests KTO's benefits generalize across reasoning types.

On MMLU and HumanEval, gains are modest (0.4 and 0.8 points respectively over DPO), suggesting KTO's advantage is concentrated in tasks requiring multi-step reasoning rather than factual recall or code generation.

The one-y-per-x variant—which discards one output per input, reducing data volume but removing any trace of preference pairing—still achieves 50.0 on GSM8K (10 points above DPO on the full dataset) and 49.9 on BBH (5.8 points above DPO). This is a critical robustness check: KTO's gains are not dependent on having twice as many training examples as DPO, since the one-y-per-x variant processes fewer examples than DPO (one per input vs. a pair per input) yet still outperforms it.

Full benchmark suite (Table 5): The extended results in Appendix Table 5 add TydiQA (1-shot, F1) and AlpacaEval 2 (win%). KTO underperforms DPO on TydiQA (31.2 vs. 36.5 F1) but substantially outperforms on AlpacaEval 2 (12.5% vs. 7.8% winrate). The average across all six benchmarks is KTO 39.9 vs. DPO 36.1, a 3.8-point advantage.

Table 1 (recommended hyperparameters): For practitioners, the paper provides tuned configurations for Llama-3 8B and Qwen2.5 3B Instruct, reporting AlpacaEval (LC), BBH, and GSM8K scores. KTO (without SFT) on Llama-3 8B achieves 11.25 AlpacaEval winrate vs. 10.59 for SFT+KTO—notably, skipping SFT actually improves AlpacaEval performance for this configuration. On Qwen2.5 3B Instruct, SFT+KTO dramatically outperforms KTO alone on BBH (32.39 vs. 20.41), showing that the SFT-independence observed at 13B+ for Llama does not generalize to smaller instruction-tuned models.

KTO Does Not Need SFT at Sufficient Scale (Figures 3, 4; Table 6)

Headline finding: For Llama-{13B, 30B}, KTO without SFT matches SFT+KTO in generation quality, while DPO without SFT produces degraded outputs (rambling, hallucination). This is not observed for Pythia models or Llama-7B.

Figure 3 shows KTO alone matching SFT+KTO for Llama-13B and Llama-30B. Figure 4 provides the mechanistic explanation: DPO without SFT causes response length to dramatically increase, with the model generating excessively long and hallucinatory outputs. KTO without SFT maintains response lengths comparable to the SFT baseline. The paper does not provide a theoretical explanation for why KTO avoids the length explosion—it is presented as an empirical property.

Table 6 provides qualitative evidence: the SFT+DPO Llama-30B output for a show-recommendation prompt hallucinates multiple conversation turns (the assistant asks and answers its own follow-up questions, generating an entire multi-turn dialogue). The KTO output is concise and stays on-task. This suggests that KTO's reference-point mechanism (penalizing divergence from $\pi_{\text{ref}}$ via the KL term in $z_0$ ) provides implicit regularization against the distributional collapse that DPO without SFT experiences, even when $\pi_{\text{ref}}$ is the pretrained rather than SFT model.

Caveat: The SFT-independence claim is qualified—it holds "at sufficient scale" (13B+ for Llama), but not for Pythia models (which are weaker base models) or Llama-7B, and not for the instruction-tuned Qwen2.5 3B (Table 1, where SFT+KTO substantially outperforms KTO alone on BBH). The property appears to require both a strong base model and sufficient parameter count.

KTO Data Need Not Come from Preferences (Figure 5, Table 3)

Headline finding: KTO maintains DPO-competitive performance when up to 90% of desirable examples are discarded, and when only one output per input is used (removing all pairing structure), demonstrating that KTO's success is not an artifact of the preference data it was trained on.

Figure 5 shows the GPT-4-judged winrate of KTO-aligned Llama-7B as increasingly large fractions of desirable data are randomly discarded. With the default 1:1 ratio (no discarding), KTO matches DPO. As the desirable fraction decreases to 0.1 (a 1:10 ratio of desirable to undesirable examples), KTO continues to match or exceed DPO performance when $\lambda_D$ and $\lambda_U$ are adjusted according to Equation 9 ( $\lambda_U = 1, \lambda_D = 13.33$ at 90% discarding). The paper does not report performance without this adjustment—the loss aversion hyperparameters are essential for handling imbalance, not optional.

Table 3 addresses a different concern: what if KTO is "secretly benefiting from its 2n examples coming from n preference pairs"? The experiment aligns Mistral-7B on OpenAssistant using three setups: DPO on $n$ preference pairs, KTO on all $2n$ outputs (standard conversion), and KTO on only one output per $x$ (discarding one output from each pair). The one-y-per-x setup reduces training data by 72% but still outperforms DPO:

Method	Winrate vs. SFT Target
Mistral-7B (unaligned)	0.525 ± 0.037
Mistral-7B + DPO	0.600 ± 0.037
Mistral-7B + KTO (all y per x)	0.652 ± 0.036
Mistral-7B + KTO (one y per x)	0.631 ± 0.036
Mistral-7B-Instruct	0.621 ± 0.031

KTO with one-y-per-x achieves 0.631 winrate versus DPO's 0.600 and the official Mistral-7B-Instruct's 0.621. The confidence intervals (90% binomial) show KTO (one-y-per-x) significantly outperforming DPO ( $p < 0.05$ with non-overlapping intervals), confirming that KTO's advantage is not due to the paired structure or larger effective dataset size.

What these experiments collectively demonstrate: KTO can work with binary feedback that does not originate from preference pairs, can handle severe class imbalance, and does not require the information contained in knowing which specific outputs were compared. This is the empirical foundation for the paper's claim that KTO enables alignment from naturally-occurring binary feedback sources.

Memory-Efficient KTO Outperforms ORPO (Table 2)

Headline finding: KTO without a reference model (assuming $\pi_{\text{ref}}$ is uniform, simplifying the reward to $\log \pi_\theta(y|x) - H(\pi_\theta)$ ) strictly outperforms ORPO, another reference-free method, while using less memory (since a batch of $m$ KTO outputs is half the size of a batch of $m$ preference pairs).

Table 2 (middle section) compares the reference-free KTO variant ( $\lambda_D = 1.75$ ) against ORPO ( $\lambda = 0.1$ ) on the Zephyr benchmarks:

Method	MMLU	GSM8K	HumanEval	BBH
ORPO	57.1	36.5	29.5	47.5
KTO (no $\pi_{\text{ref}}, \lambda_D=1.75$ )	57.5	47.5	29.5	51.6

The reference-free KTO outperforms ORPO by 11.0 points on GSM8K and 4.1 points on BBH, while matching on MMLU and HumanEval. However, it trails standard KTO (with $\pi_{\text{ref}}$ ) by 6.0 points on GSM8K (47.5 vs. 53.5) and 1.0 point on BBH, confirming that the reference model provides valuable grounding. The paper notes this variant is "more sensitive to the loss aversion hyperparameters" and that "setting $\lambda_D \in \{1.5, 2.0\}$ reduces performance on GSM8K and BBH by several points."

Ablation Studies and Robustness Checks

Value function shape matters substantially (Table 2, middle): Four variants of KTO are tested that modify the value function while keeping the rest of the framework fixed. Removing the reference point ( $z_0 = 0$ constant, not a HALO) causes drops of 3.6 on BBH (52.6 → 49.0) and 4.0 on GSM8K (53.5 → 49.5) but only minor changes on MMLU and HumanEval, showing that the reference point is most important for reasoning-intensive tasks. Making the value function concave everywhere (replacing $1 - \sigma(\cdot)$ with $-\log \sigma(\cdot)$ , matching DPO's value function shape) causes a 9.4-point drop on BBH (52.6 → 43.2) and 11.0-point drop on GSM8K (53.5 → 42.5)—the convexity in losses is essential, not incidental. Making the value function risk-neutral (identity function) causes a catastrophic collapse on BBH (52.6 → 6.1) and substantial drops elsewhere, confirming that the prospect-theoretic shape (sigmoidal, saturating) is critical. Removing the reference model entirely (reference-free variant with $\lambda_D = 1.75$ ) preserves much of BBH performance (51.6) but loses 6.0 points on GSM8K (47.5), showing the reference model matters most for mathematical reasoning.
$\beta$ (risk aversion) affects performance significantly (Table 2, bottom): Three values are tested for Zephyr-β-SFT: $\beta = 0.01, 0.10, 0.50$ . The default $\beta = 0.10$ achieves the best GSM8K (53.5) but $\beta = 0.01$ achieves the best BBH (49.8) and substantially better HumanEval (33.2 vs. 30.9). $\beta = 0.50$ degrades GSM8K (45.0) but is best on MMLU (58.3) and comparable on HumanEval (32.7) and BBH (51.8). There is no universally optimal $\beta$ —the best value depends on the downstream task, supporting the paper's claim that "there is no one HALO that is universally superior."
$\lambda_D$ (loss aversion for desirable outputs) modulates the gain-loss balance (Table 2, bottom): Five values of $\lambda_D$ are tested with $\lambda_U = 1$ fixed: 0.50, 0.75, 1.00, 1.33, 2.00. The default $\lambda_D = 1.00$ gives the best GSM8K (53.5) and second-best BBH (52.6). Lower $\lambda_D = 0.50$ substantially hurts BBH (52.6 → 30.5) but preserves GSM8K reasonably (47.0). Higher $\lambda_D = 2.00$ lowers GSM8K to 43.0. The pattern confirms that the effective ratio between desirable and undesirable example weights matters—deviations from the optimal balance degrade performance, and the degradation is task-specific.
KTO can align on desirable-only data but underperforms SFT (Table 4): When trained only on desirable examples (the same data used for SFT), KTO with $\beta = 0.1$ achieves results between SFT and full KTO—better than SFT on GSM8K (12.5 vs. 1.0) but worse on BBH (29.5 vs. 33.2). KTO with $\beta = 0.01$ produces results nearly identical to SFT across all benchmarks, which is intuitive: with very low risk aversion, the model is less bound to the reference and essentially recovers SFT-like behavior. This ablation confirms that undesirable examples are important for KTO's gains—the method is not simply performing SFT with a different objective.
One-y-per-x matches full KTO on most metrics (Tables 2, 3): The one-y-per-x variant, which discards one output per input (removing all pairing structure), achieves only slightly lower performance than full KTO: 50.0 vs. 53.5 on GSM8K, 49.9 vs. 52.6 on BBH, 58.0 vs. 58.6 on MMLU. In the Mistral-7B winrate experiment (Table 3), one-y-per-x achieves 0.631 vs. 0.652 for full KTO—a 2.1 percentage point drop, but still significantly above DPO's 0.600. This demonstrates robustness to data volume and confirms that the preference pairing information is not load-bearing for KTO's performance.
GPT-4 judgments concur with human judgments at ~67% rate (Appendix D): For KTO, 68.7% of individual GPT-4 judgments agree with human judgments; for DPO, this drops to 65.9%. The human evaluation gives KTO a larger margin over DPO (72.9% vs. 62.1% winrate) than GPT-4 evaluation does (65.2% vs. 60.0%), and the human evaluation gap is statistically significant ( $p < 0.05$ ) while the GPT-4 gap is not. This suggests that KTO's advantage may be partially obscured by automated evaluation, consistent with Theorem 4.2's implication that KTO optimizes directly for human utility rather than a preference proxy.
Offline PPO with dummy rewards is a strong but brittle baseline (Figure 2): A notable negative result: the offline PPO variant matches DPO for all models except Llama-30B, demonstrating that binary signals can work with a HALO-structured loss even without careful reward modeling. However, it "suffers from hyperparameter sensitivity and training instability," limiting its practical utility. This baseline serves as a motivating existence proof for KTO—binary feedback can work, but it needs a more principled loss function than dummy rewards fed into PPO-Clip.
KTO alignment scales with model size but not uniformly (Figures 2, 3): The gap between KTO and baselines widens at larger scales, but only for the Llama family. For Pythia models, all methods are clustered near chance with no significant differences. This suggests a threshold effect: the base model must be sufficiently capable before alignment (of any type) provides measurable benefits on the winrate metric. The paper attributes this to the Pythia models being less performant and the SFT data being drawn from a similar distribution to the alignment data, meaning there is less room for alignment to improve over SFT.

Critical Assessment

Claim 1: KTO matches or exceeds DPO performance at scales from 1B to 30B. Supported with important nuance. The claim holds on the winrate metric (Figure 3) across both model families, but "matches" is more accurate than "exceeds" for most configurations—SFT+KTO is competitive with SFT+DPO, not clearly superior. The "exceeds" part is specifically supported for KTO without SFT vs. DPO without SFT (significant at 7B and 30B) and on generative benchmarks (Table 2), where KTO shows substantial gains on GSM8K and BBH. However, the benchmark evaluation is only reported for one model (Zephyr-β-SFT on UltraFeedback), not across the full Pythia/Llama scaling sweep. Without benchmark results across model scales, we cannot assess whether the GSM8K gain (13.5 points) is a general property or specific to the Zephyr-UltraFeedback combination. A missing experiment: running DPO and KTO on the same benchmarks across multiple model scales.

Claim 2: KTO works from binary feedback alone. Supported, but with a structural caveat. All experiments in the main paper (except one-y-per-x) convert preference pairs into binary feedback. This means the binary data inherits the property that for each input, one output is strictly better than the other—the "desirable" and "undesirable" labels are not independent judgments but come from a pairwise comparison. The one-y-per-x experiments (Tables 2, 3) partially address this by removing the pairing, but the data still originates from preferences. A stronger test would use naturally binary feedback (e.g., thumbs up/down from a production system, or toxicity flags) rather than deconstructed preferences. The paper acknowledges this limitation: "a more complex deconstruction of preferences into binary feedback would likely yield better results."

Claim 3: KTO can handle extreme data imbalances, matching DPO with up to 90% fewer desirable examples. Supported with reservations. Figure 5 shows this for Llama-7B on the winrate metric, but only at one scale and with careful hyperparameter tuning of $\lambda_D$ and $\lambda_U$ . The paper does not report whether the same imbalance robustness holds for larger models or on generative benchmarks. The experiment also discards desirable examples while keeping all undesirable examples, creating a specific type of imbalance—it does not test the reverse (few undesirable examples) or random imbalance patterns. The tuning of $\lambda_D$ according to Equation 9 is critical; without it, performance likely degrades, but this ablation is not reported.

Claim 4: When the pretrained model is sufficiently good, KTO can skip SFT without loss in generation quality. Supported with strong evidence for Llama-13B and Llama-30B (Figure 3, confirmed by Figure 4 showing DPO's length explosion without SFT). However, the claim does not generalize: it fails for Pythia models (all scales), Llama-7B, and Qwen2.5 3B Instruct (Table 1, where SFT+KTO substantially outperforms KTO alone). The "sufficiently good" condition is empirically a threshold around 13B parameters for the Llama family, but the paper does not characterize this threshold in terms of base model metrics (e.g., MMLU score, few-shot accuracy) that would predict when SFT can be skipped for other model families. A missing experiment: ablating $\beta$ for KTO without SFT across model scales to determine whether the length explosion is a $\beta$ -dependent phenomenon rather than an inherent property of the loss function.

Claim 5: The best HALO depends on the inductive biases appropriate for a given setting, and no single HALO is universally superior. Supported by the ablation results in Table 2 but not by comparative HALO experiments. The paper shows that different value function shapes and hyperparameters produce different performance profiles across tasks—risk-neutral KTO collapses on BBH but not on GSM8K, low $\beta$ helps HumanEval but hurts GSM8K, reference-free KTO trails standard KTO on GSM8K but is competitive on BBH. However, the paper only compares KTO variants, not different HALOs (e.g., KTO vs. DPO vs. some new HALO with a different value function) on the same benchmarks at the same scale. The claim is a conceptual framing rather than a demonstrated empirical fact—it is plausible and consistent with the results, but the paper does not systematically vary the HALO while holding other factors constant.

Genuine weaknesses in the experimental design:

Single-turn, single-domain evaluation. All winrate evaluations are on OpenAssistant test prompts (multi-turn conversation), and all benchmark evaluations are on standard academic NLP benchmarks. The paper does not evaluate on tasks where binary feedback is most natural in production—toxicity detection, content moderation, refusal of harmful requests—where the distinction between "desirable" and "undesirable" has clear safety implications. A missing experiment: evaluating KTO-aligned models on harmfulness benchmarks (e.g., Anthropic's red-teaming data, TruthfulQA).
No online or iterative alignment experiments. KTO is evaluated only in the offline setting (one epoch over fixed data). The paper speculates about online alignment in Section 5 but provides no evidence. Given that one of KTO's selling points is compatibility with abundant binary feedback (which could be collected online during deployment), the absence of any online experiment is a significant gap.
Hyperparameter sensitivity is acknowledged but not quantified. The paper states that learning rate is the most sensitive hyperparameter and that KTO needs 2× to 10× the DPO learning rate, but does not provide learning rate sensitivity curves. The loss aversion ratio guideline (Equation 9) is presented as empirical but without the experiments that generated it—how much does performance degrade if $\lambda_D n_D / \lambda_U n_U$ is set to 0.5 or 2.0? The recommended $\beta$ ranges ( $[0.01, 0.10]$ for large SFT models, $[0.10, 1.00]$ for small non-SFT models) are based on unspecified sweeps; a grid of results across $\beta$ and model scale would substantially strengthen the practical guidance.
The $z_0$ estimator is biased and its impact is not isolated. The mismatched-pair estimator introduces a positive bias (due to max-clamping at zero) and uses an availability-heuristic-inspired pairing. The "no $z_0$ " ablation (Table 2) drops $z_0$ entirely (sets it to constant zero), but does not compare the mismatched-pair estimator against an unbiased (but more expensive) estimator, such as sampling from the policy. We cannot distinguish whether the gains from including $z_0$ come from having any reference point or from the specific properties of the mismatched-pair estimator.
No direct comparison between KTO and RLHF with a learned reward model. The paper compares KTO to DPO and an offline PPO variant with dummy rewards, but not to standard online RLHF with a trained reward model. This is an understandable omission (RLHF is expensive and unstable), but it means we cannot assess how KTO compares to the method that has historically produced the strongest aligned models (e.g., InstructGPT, Claude). The offline PPO baseline with dummy rewards is a weak proxy for full RLHF.
The winrate metric's practical significance is unclear. A winrate of 52% against SFT targets means the aligned model is judged better 52% of the time—barely above chance. The paper frames this as a difficult evaluation (since SFT targets are human-written), but it also means the absolute improvement from alignment is small on this metric. The benchmark evaluations (Table 2) show larger absolute improvements, but these are only reported for one model configuration. A more comprehensive picture would report both types of evaluation across all scales.
The $4/3$ upper bound in Equation 9 lacks theoretical or extensive empirical justification. The paper states this interval "was determined empirically" but does not provide the experiments supporting it. This is a practical guideline that practitioners will use, but its robustness across datasets, model families, and tasks is unverified.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Efficiency Claims

The assumption or constraint. The compute-optimal framework rests on the ability to estimate prompt difficulty before allocating the inference budget. The paper's method for doing so requires generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or PRM final-answer scores (predicted bins). The authors explicitly acknowledge this cost in Section 3.2: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. The 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is already known, without amortizing the cost of learning it. At 2048 samples per question, difficulty estimation alone can consume more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter on problems where the optimal strategy uses a small budget. The head-to-head comparisons between predicted and oracle bins (Figures 4 and 8) show that the PRM-based difficulty estimate works almost as well as the oracle—but both require the same expensive 2048-sample generation step. The predicted bins are only "practical" in the sense that they don't need ground-truth labels; they are not cheaper to obtain.

What evidence exists in the paper. The authors flag this issue in Section 3.2 and frame it as "an important avenue for future work." Figures 4 and 8 show compute-optimal scaling curves that start at low generation budgets (e.g., 4 generations), but the difficulty estimation cost is never added to the x-axis, meaning the curves represent an upper bound on achievable efficiency rather than a realized deployment gain. No experiment measures how performance degrades if difficulty is estimated from fewer than 2048 samples—for instance, 16, 32, or 64 samples would be far cheaper but might produce noisier difficulty bins that erode the gains.

Mitigation status. The paper does not attempt to mitigate this limitation experimentally. It suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) and mentions the possibility of using the PRM's score distribution on a small number of initial samples as a difficulty signal—but neither approach is implemented or evaluated. Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency, not a realized deployment gain.

Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Compensate for Missing Base Capability

The assumption or constraint. The entire compute-optimal framework assumes that the base model can produce correct solutions at some non-trivial rate. This is captured in the paper's central finding about difficulty-dependence: on the hardest questions (difficulty bin 5), no method—search, revisions, or their compute-optimal combinations—makes meaningful progress regardless of budget. The base model's pass@1 on these problems is near zero, meaning there are no correct solutions in the proposal distribution to find or refine. The paper explicitly states this in Section 7: test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address."

The consequence. For any problem distribution that includes a substantial fraction of genuinely hard questions (outside the base model's capability range), the compute-optimal framework offers no improvement over baseline SFT or greedy decoding. The FLOPs-matched comparison in Figure 9 shows this starkly: on the hardest difficulty bin (bin 5, blue line), compute-optimal scaling curves are flat near 0–5% accuracy, well below the 14× larger model's performance at all values of the inference-to-pretraining token ratio R. The entire framework's value proposition rests on the problem distribution skewing toward easy-to-medium difficulty—a property that is dataset-specific and deployment-specific, not guaranteed.

What evidence exists in the paper. Across all methods and budgets, bin 5 accuracy hovers at 1–3% in the search experiments (Figure 3, right), roughly 2–3% in the revision experiments regardless of the sequential-to-parallel ratio (Figure 7, right), and flat near 0–5% in the FLOPs-matched comparison (Figure 9, bin 5 panels). The FLOPs-matched bar charts (Figure 1) show relative disadvantages of −37.2% for revisions and −52.9% for PRM search on hard questions at R ≫ 1, meaning test-time compute is actively worse than just training a larger model on these problems.

Mitigation status. The paper is transparent about this limitation (Section 7 takeaway box), but offers no mitigation within the framework itself. The authors frame it as a boundary condition: "test-time compute amplifies existing capability but does not create it." The implication is that pretraining remains the only viable path for genuinely novel or out-of-distribution reasoning, but the paper does not discuss how to detect which regime a given deployment falls into before investing in test-time compute infrastructure versus pretraining.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)—Generality Is Unestablished

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* (Codey) as the base model. MATH consists of high-school competition-level math problems requiring symbolic multi-step reasoning with ground-truth answers that can be checked via string matching. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The PRM's quality and over-optimization behavior, the revision model's ability to learn from incorrect in-context examples, and the difficulty-dependent scaling curves could all be specific to this model's output distribution and this task's structure.

The consequence. Several aspects of the findings may not transfer to other settings:

The PRM's Monte Carlo rollout training procedure depends on having clean correctness signals (exact answer matching) for labeling—many important real-world tasks lack such signals.
Mathematical reasoning has a particular structure (step-by-step derivability, verifiable intermediate steps) that makes PRM-guided search viable. Code generation has similar properties (unit tests), but open-ended generation, dialogue, and creative writing do not.
The difficulty-dependent optimal strategies—beam search on medium problems, sequential revisions on easy problems—are derived from the interaction between PaLM 2-S*'s specific error patterns and the MATH dataset's difficulty distribution. A model with different calibration properties or a task with different reasoning patterns might exhibit different optimal policies.
The 14× larger model in the FLOPs-matched comparison is also from the PaLM 2 family, meaning the pretraining-inference tradeoff is assessed within one model architecture. Different model families have different scaling properties.

What evidence exists in the paper. No experiments use any benchmark other than MATH, any task other than closed-form math reasoning, or any base model other than PaLM 2-S*. The revision model's detrimental behavior under ReST^EM training (Appendix K, Figure 16) already suggests sensitivity to training methodology within the same model family—behavior could differ further across architectures. The test set consists of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample, and the selected strategies may not be robust.

Mitigation status. The paper explicitly acknowledges this limitation in Section 4 and frames it as scope rather than a flaw, but makes no attempt to address it experimentally. The authors' belief that PaLM 2-S* is "representative" is stated as an opinion, not a claim supported by cross-model comparisons. A single replication on a different model family (e.g., Llama, Mistral) or a different reasoning benchmark (e.g., GSM8K, HumanEval for code) would substantially strengthen the generality claims, but no such replication is provided.

Revisions and Search Are Studied Independently, Never Combined—The Full Potential of the Framework Is Unexplored

The assumption or constraint. The paper studies two complementary mechanisms—PRM-guided search and iterative revision—as separate, independent scaling axes. The revision model is never used as the proposal distribution within beam search, and the PRM is never used to guide which revisions to pursue or when to restart a revision chain. Section 8 explicitly acknowledges: "we did not experiment with PRM tree-search techniques in combination with revisions."

The consequence. The reported results represent a lower bound on what an integrated system could achieve. The two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates by conditioning on previous attempts), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs—or using the PRM's per-step scores to decide when a revision is on track versus when to restart—could yield gains beyond either method alone, potentially breaking through the performance ceiling that each individual method hits at high budgets (Figures 3 and 6 show both methods plateauing). The difficulty-dependent optimal strategies might also change if both mechanisms are available: a problem that is "medium-hard" under search alone might become "medium-easy" when a revision model provides higher-quality beam candidates.

What evidence exists in the paper. The paper provides no evidence for or against the combination. The search experiments (Section 5) use the base few-shot prompted model as the proposal distribution. The revision experiments (Section 6) use majority voting or an ORM for answer selection, not PRM-guided search. The two pipelines are never connected. The revision model's ability to condition on previous incorrect answers (Figure 6, left) suggests it could potentially serve as a stronger proposal distribution for beam search—each beam could be conditioned on the history of rejected branches—but this hypothesis is untested.

Mitigation status. The paper explicitly flags this as future work in Section 8: "we did not experiment with PRM tree-search techniques in combination with revisions." No mitigation is attempted, and the paper does not discuss the technical challenges that combining the two approaches would entail (e.g., how to define the PRM's step-level scores when the proposal distribution is itself conditioning on revision history, or how to allocate the generation budget between search width and revision depth within a tree).

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining vs. Test-Time Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the Llama paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both data and parameters are scaled (Hoffmann et al., 2022). The authors acknowledge this: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." Additionally, the 14× larger model uses only greedy decoding—no test-time compute of any kind (no majority voting, no best-of-N, no search).

The consequence. The pretraining baseline is weaker than it could be in two ways. First, a Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model trained on the same data, making the pretraining side of the tradeoff stronger. Second, even the parameter-scaled model could benefit from some test-time compute—giving it best-of-8 or majority voting would create a fairer comparison (matching total FLOPs by adjusting the smaller model's inference budget downward accordingly). The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1) may shrink or reverse against improved baselines.

What evidence exists in the paper. The paper reports the specific FLOPs accounting in Section 7 with the formulas X = 6ND_pretrain and Y = 2ND_inference, and explicitly states the parameter-only scaling design choice. The 14× larger model's performance is shown as stars in Figure 9 and as the baseline in the bar charts of Figure 1. No experiments compare against a Chinchilla-optimal larger model or a larger model with any test-time compute budget. The paper acknowledges this as a limitation in Section 7 but does not quantify how much it might change the results.

Mitigation status. The paper leaves the compute-optimal pretraining comparison to future work and frames the current comparison as "representative of a canonical approach." This is a reasonable scoping decision given the difficulty of training multiple Chinchilla-optimal models at different scales, but it means the headline finding—that test-time compute can substitute for pretraining—has not been tested against the strongest possible pretraining baseline. A practitioner deciding whether to invest in test-time compute infrastructure versus training a larger model should treat the reported advantages as upper bounds.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate—A Fundamental Training Data Mismatch

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct target. It never sees examples where the current answer is already correct and should be preserved. At inference time, the model generates a chain of revisions, and approximately 38% of correct answers produced during this chain get "revised" back to incorrect answers in the subsequent step (Section 6.1). This is a direct consequence of the training data construction: the model learns that its role is to produce a different (correct) answer from the previous (incorrect) one, and when the previous answer happens to be correct, the model still tries to produce something different.

The consequence. The revision chain is inherently unstable—even when the model finds a correct solution, it may discard it on the next revision step. The paper's mitigation (selecting the best answer across the entire chain via majority voting or verifier-based selection) is a post-hoc patch that does not fix the underlying behavior. This means the effective length of useful revision chains is limited: after a certain number of steps, the probability of inadvertently reversing a correct answer outweighs the probability of correcting an incorrect one. The paper's Figure 6 (left) shows pass@1 gradually improving through ~20 steps and then remaining in the 23–25% range out to 64 steps, suggesting the marginal benefit of additional revisions diminishes and may eventually become negative if the selection mechanism fails.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The within-chain selection mechanisms (majority voting and verifier-based) are described as mitigations, and the sequential revision curves in Figure 6 (right) show that sequential + selection outperforms parallel sampling, confirming that the benefit of revisions survives despite the reversion problem. However, the paper does not report how many correct answers are lost at each revision depth, how the reversion rate changes with chain length, or whether training the model with some "preserve correct answer" examples would reduce the reversion rate.

Mitigation status. The paper acknowledges this as a "significant practical issue" (Section 6.1) and mitigates it with within-chain answer selection, but does not attempt to fix the underlying training data mismatch. A more principled solution—such as including training examples where the correct answer appears in context and the target is to reproduce it (teaching the model when not to revise)—is not explored. The ReST^EM experiment (Appendix K, Figure 16) shows that attempting to optimize the revision model with RL-style training further degrades performance, suggesting the revision training procedure is fragile and the reversion problem is not easily solved by more training.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes how the field should think about alignment objectives. Before this work, the dominant mental model was reward-centric: alignment means inferring what humans want (via preference data and reward models) and then optimizing against that inferred signal. DPO elegantly collapsed this into a single stage but preserved the core logic—maximize the likelihood of observed preferences under a Bradley-Terry model. Success was attributed to better reward inference. KTO and the HALO framework challenge this entire framing by proposing a different answer to why alignment methods work: not because they accurately recover a latent reward function, but because their loss functions encode the same cognitive biases—reference dependence, diminishing sensitivity, loss aversion—that prospect theory identifies in human decision-making.

This is a reframing, not a paradigm shift. It does not invalidate RLHF or DPO; it reinterprets their success and opens a new design space. The proof that DPO and PPO-Clip are HALOs (Theorem 3.5) shows that the framework retrospectively captures what was already working. The novelty is making this structure explicit and showing that it can be deliberately instantiated with different value functions—specifically, the Kahneman-Tversky value function—to achieve strong alignment from weaker feedback. The magnitude of the shift is moderate: it does not change what alignment practitioners do (train models on human feedback), but it changes how they should choose their loss function—moving from "use DPO because it works" to "select a HALO whose value function shape matches the inductive biases of your data and task."

The paper resolves a simmering contradiction in the literature about whether binary feedback is sufficient for alignment. Prior work using binary signals (CSFT, unlikelihood training) consistently underperformed preference-based methods, leading to the reasonable conclusion that preferences were necessary. KTO demonstrates that this conclusion was about loss function design, not feedback format. CSFT is not a HALO; KTO is. The failure was not in the data but in the objective's inductive biases. This reconciliation is practically important: it means the field can now confidently use abundantly available binary feedback (thumbs up/down, accept/reject signals) for alignment, provided the loss function has the right structure—specifically, a reference-dependent value function that is concave in gains and convex in losses.

The paper redirects research attention in several ways. More attractive directions: (1) Designing new HALOs with value functions tailored to specific domains, feedback types, or individual differences, drawing on the rich behavioral economics literature on human utility. (2) Collecting large-scale binary feedback datasets, since KTO removes the data format bottleneck that made preference collection the rate-limiting step. (3) Studying the interaction between value function shape and data characteristics (noise levels, intransitivity rates, class imbalance), since Table 2 shows that even small changes to the value function (concave vs. sigmoidal, risk-neutral vs. risk-averse) produce large performance differences. (4) Online and iterative alignment with binary feedback, since the lower cost of binary data collection makes continuous alignment loops more practical. Less attractive directions: (1) Incremental improvements to Bradley-Terry-based preference methods, since Theorem 4.2 shows that preference likelihood maximization does not uniquely determine human utility—there may be fundamental limits to what better preference modeling can achieve for utility. (2) Treating reward model quality as the primary bottleneck for alignment, since the offline PPO variant with dummy +1/-1 rewards (Figure 2) demonstrates that even the simplest reward signal can work when the loss function has the right inductive biases.

Follow-Up Research This Work Enables

Training a difficulty predictor to replace the expensive 2048-sample estimation. The paper's compute-optimal framework requires estimating question difficulty to allocate test-time compute, but the current method—generating 2048 samples per question and averaging PRM scores—is far too expensive for deployment. A natural follow-up trains a lightweight classifier (possibly distilled from the PRM) that takes only the question text as input and predicts the difficulty quintile. The experiment would compare accuracy of a PRM-distilled classifier against the 2048-sample estimate on the MATH test set, measure the downstream impact on compute-optimal scaling curves (does the 4× efficiency gain survive when difficulty is estimated from text alone?), and test whether the classifier transfers to out-of-distribution math problems (e.g., GSM8K) without retraining. Success would make compute-optimal test-time scaling deployable without the prohibitive difficulty estimation cost.

Combining PRM-guided tree search with the KTO-aligned model as the proposal distribution. The paper studies search against PRMs and KTO alignment as independent mechanisms, but never uses a KTO-aligned model within beam search. Since KTO directly maximizes the prospect-theoretic utility of generations—and Theorem 4.2 shows this targets a different objective than preference likelihood—a KTO-aligned proposal distribution might produce candidates that are qualitatively different from those of a base or SFT model, potentially changing which search strategies are optimal at each difficulty level. The experiment would replicate the Section 5.2 search algorithms (best-of-N, beam search, lookahead) using a KTO-aligned model as the generator instead of the few-shot prompted base model, measure whether the difficulty-dependent optimal strategies shift (e.g., does beam search still over-optimize on easy problems when the proposal distribution already encodes Kahneman-Tversky utility?), and compare against the original PaLM 2-S* search results on the same MATH test set. A negative result—KTO alignment provides no benefit over SFT for downstream search—would clarify whether the value function shape matters only for the alignment stage or also for the proposal distribution quality in search.

Measuring KTO's robustness to naturally noisy binary feedback (not deconstructed preferences). All of KTO's experiments convert preference pairs into binary labels, which inherits the property that for each input, one output is strictly better than the other. A stronger test of the paper's central claim—that KTO works with binary feedback from non-preference sources—would train on naturally binary data: for example, toxicity flags from civil comments datasets, helpfulness ratings binarized at a threshold, or production thumbs-up/thumbs-down signals. The experiment would compare KTO against DPO (which would require artificially pairing the binary examples) and against CSFT on metrics that are directly relevant to the binary signal (e.g., toxicity reduction, helpfulness winrate). The key measurement is whether KTO's advantage over DPO persists when the binary labels are independent judgments rather than deconstructed preferences, and whether Theorem 4.3's prediction—that KTO handles contradictory feedback better than DPO—manifests as improved robustness to annotator disagreement in naturally noisy data. This directly tests the paper's motivating claim that binary feedback is "more abundant, cheaper, and faster to collect in the real world."

Identifying the optimal value function shape for different alignment domains. The paper's ablation in Table 2 shows that value function shape dramatically affects performance: making the value function concave everywhere (like DPO) drops BBH by 9.4 points compared to KTO's concave-in-gains, convex-in-losses shape; making it risk-neutral collapses BBH to 6.1%. But these results are for a single model (Zephyr-β-SFT) on a single dataset (UltraFeedback). A systematic study would vary the value function shape parametrically—sweeping the power-law exponent α in the original Kahneman-Tversky form (Equation 4), comparing logistic vs. power-law vs. piecewise-linear shapes, varying the asymmetry between gains and losses—across multiple alignment domains: helpfulness, harmlessness, factuality, code generation, and creative writing. The experiment would measure whether the optimal shape is domain-specific (does harmlessness alignment benefit from higher loss aversion λ because avoiding bad outputs matters more?) and individual-specific (do different annotator populations have systematically different value function parameters?). This directly builds on the paper's closing claim that "the best HALO depends on the inductive biases most appropriate for a given setting" and would transform it from a conceptual framing into an empirically validated design principle.

KTO with online binary feedback in an iterative self-improvement loop. The paper evaluates KTO only in the offline setting (one epoch over fixed data), but one of its selling points is compatibility with abundant binary feedback that could be collected online during deployment. A natural extension implements an iterative alignment loop: deploy a KTO-aligned model, collect binary feedback (thumbs up/down) on its generations from users, retrain with KTO on the new feedback, and repeat. The experiment would measure whether performance improves across iterations or plateaus/deteriorates (as the ReST^EM revision model did in Appendix K), whether the effective ratio guideline (Equation 9) needs adjustment when feedback is collected from the model's own evolving distribution, and whether KTO's automatic curriculum property (Proposition 4.1—gradients vanish for examples that are too hard or too easy) causes the model to ignore systematically difficult outputs that are important to learn. A negative result—performance degrades with online iterations due to distribution shift or underfitting—would identify an important boundary condition on KTO's applicability and motivate the dynamic λ_D, λ_U schemes mentioned in Section 4.2.

Practical Applications and Downstream Use Cases

Continuous alignment of production chatbots from user feedback. In deployed chatbot systems, every user interaction generates implicit binary feedback: a thumbs-up/thumbs-down rating, whether the user copied a code snippet, whether they asked a clarifying follow-up question, or whether they ended the conversation. These signals are abundant and naturally binary but cannot be directly used by DPO or RLHF without artificial pairing. KTO enables training directly on this feedback stream without any preprocessing. The practical benefit is a continuous alignment loop: the model can be retrained weekly or daily on fresh feedback, adapting to shifting user expectations and catching failure modes that were not represented in the original alignment dataset. The paper's results suggest this should match the performance of preference-based retraining (Figure 3: KTO matches DPO at all scales) while using feedback that is effectively free. The data imbalance experiments (Figure 5) are directly relevant here—production feedback is often heavily imbalanced (many more positive than negative interactions, or vice versa), and KTO's ability to handle 1:10 ratios with appropriate λ_D, λ_U tuning makes it practical for real-world feedback distributions.

Scaling alignment data collection for underrepresented user populations. Because KTO works with binary feedback—which is cognitively simpler to provide than pairwise preferences—it significantly lowers the barrier to collecting alignment data from diverse user populations. Instead of asking annotators to compare two outputs (which requires reading both carefully and rendering a comparative judgment), they can simply mark individual outputs as acceptable or unacceptable for their use case. A deployment serving multiple user demographics could collect binary feedback from each group, train separate KTO-aligned models (or a single model with demographic-conditioned generation), and serve personalized aligned models rather than a one-size-fits-all solution. The paper's one-y-per-x results (Table 3: KTO with 72% less data still outperforms DPO) suggest that even modest amounts of per-group binary feedback could produce meaningful alignment improvements. The memory-efficient KTO variant (no π_ref, Table 2) further reduces deployment costs since the reference model does not need to be stored, making per-user model variants more feasible.

Toxicity and safety filtering with explicit undesirable examples. In content moderation settings, the binary signal is naturally asymmetric: a large volume of "safe" outputs and a smaller but critical set of "unsafe" outputs that must be avoided. KTO's loss aversion parameters (λ_D, λ_U) provide direct control over this asymmetry. For safety-critical applications where false negatives (generating harmful content) are much worse than false positives (over-refusing), practitioners can set λ_U significantly higher than λ_D—for instance, λ_U = 5, λ_D = 1—making the model highly loss-averse to undesirable outputs. The paper's theoretical analysis (Theorem 4.3) suggests KTO will deterministically avoid outputs consistently flagged as undesirable by the majority of annotators, which is the desired behavior for safety filtering where annotator disagreement often reflects genuine ambiguity about edge cases. A concrete deployment would train KTO on a dataset of prompts with human-labeled safe/unsafe outputs, sweep λ_U/λ_D to find the precision-recall tradeoff point that meets the application's safety requirements, and deploy with the reference-free variant if memory constraints preclude storing a reference model on-device.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Alignment Methods Demand Preference Data, but Preferences Are Scarce and Expensive

Why This Matters: Scaling Alignment in Production

Prior Approaches and Where They Fall Short

Where Prior Work Falls Short: A Unified Gap

How This Paper Positions Itself

The Prospect Theory Connection: Why Human Biases Matter for Alignment

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The HALO Framework: A Unifying Definition for Alignment Objectives

The Kahneman-Tversky Value Function: From Monetary Gambles to Text Generation

Constructing the KTO Loss: From Value Function to Training Objective

The KTO Gradient and the Automatic Curriculum Property

Hyperparameter Configuration: Risk Aversion, Loss Aversion, and Practical Guidelines

Theoretical Foundations: Why KTO Works Despite Weaker Signal

Data Handling: Converting Preferences to Binary Feedback

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Reframing Alignment Objectives as Prospect-Theoretic Value Functions (the HALO Framework)

Innovation 2: Binary Feedback Can Match Preferences—When the Loss Function Has the Right Inductive Biases

Innovation 3: Preference Likelihood Maximization ≠ Human Utility Maximization

Innovation 4: Automatic Robustness to Noisy and Intransitive Feedback via Gradient Saturation

Innovation 5: The Kahneman-Tversky Value Function as a Pluggable Component for Alignment Objective Design

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

HALOs vs. Non-HALOs: The Inductive Bias Matters (Figure 2)

KTO Matches or Exceeds DPO at All Scales (Figure 3)

KTO on Generative Benchmarks: Strong Gains on Reasoning Tasks (Table 2)

KTO Does Not Need SFT at Sufficient Scale (Figures 3, 4; Table 6)

KTO Data Need Not Come from Preferences (Figure 5, Table 3)

Memory-Efficient KTO Outperforms ORPO (Table 2)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Efficiency Claims

Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Compensate for Missing Base Capability

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)—Generality Is Unestablished

Revisions and Search Are Studied Independently, Never Combined—The Full Potential of the Framework Is Unexplored

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining vs. Test-Time Comparison

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate—A Fundamental Training Data Mismatch

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

AtP*: An efficient and scalable method for localizing LLM behaviour to components

ODIN: Disentangled Reward Mitigates Hacking in RLHF