Towards a Unified View of Preference Learning for Large Language Models: A Survey

🎯 Pitch

Reinforcement learning and supervised fine-tuning for aligning LLMs with human preferences actually share the same mathematical core, differing mainly in how feedback is gathered and how many samples guide each update, not in their fundamental optimization objectives. This survey dismantles over 100 existing methods into a single four-component taxonomy, erasing the artificial RL-vs-SFT divide and revealing that no single algorithm dominates—performance hinges on data quality, feedback reliability, and task context.

1. Executive Summary

This survey decomposes existing preference learning strategies for LLMs into four components—model, data, feedback, and algorithm—to establish a unified view that bridges the traditional divide between reinforcement learning-based methods (like PPO) and supervised finetuning-based methods (like DPO). By formalizing preference optimization through a shared gradient coefficient framework and decoupling algorithms from on-policy/off-policy settings, the paper reveals that RL-based and SFT-based approaches share fundamentally similar objectives, with the critical distinctions lying in how feedback is obtained (direct rules vs. trained reward models vs. LLM-as-a-judge) and how many samples inform each gradient update (point-wise methods like RFT and KTO, pair-wise contrasts like DPO and SimPO, list-wise contrasts like RRHF and GRPO, and training-free approaches like logits manipulation). The survey organizes over 100 existing methods into this taxonomy while providing detailed running examples of representative algorithms, ultimately arguing that no single algorithm dominates across all settings—performance varies significantly depending on data quality, feedback reliability, and task characteristics—and identifying scalable oversight, better preference data diversity, and more robust training algorithms as the key bottlenecks for future alignment research.

2. Context and Motivation

The Core Problem: Preference Learning Research Is Fragmented and Disconnected

The central gap this survey addresses is not a missing capability — preference alignment demonstrably works, as evidenced by the success of models like ChatGPT and GPT-4. Rather, the problem is conceptual fragmentation: the field has accumulated dozens of alignment methods across multiple research communities without a shared vocabulary or organizing framework for understanding how they relate.

The authors diagnose this explicitly in the introduction:

"While many works have extensively discussed the concept of alignment, the relationships among the various algorithms of preference learning remain fragmented, lacking a cohesive framework to unify them."

This fragmentation manifests in three specific ways that the paper identifies:

1. The artificial RL-vs-SFT divide. The prevailing taxonomy in prior surveys splits alignment methods into "reinforcement learning-based" methods (PPO, ReMax) and "supervised finetuning-based" methods (DPO, KTO). This split, while historically motivated, creates what the authors call a "barrier between the two groups of works" that obscures their shared mathematical structure. As they argue:

"This split can unconsciously result in a barrier between the two groups of works, which is not conducive to further understanding of researchers for the common core of preference alignment."

The consequence is that innovations in one camp — say, online data collection techniques from the RL literature — are not naturally transferred to the other camp, even when the underlying optimization problem is similar. Researchers working on DPO variants rarely ask whether their algorithm could benefit from on-policy sampling, and PPO practitioners rarely consider whether their critic model is truly necessary, simply because the two traditions use different jargon and reference different baselines.

2. The on-policy/off-policy confusion cascades into algorithm design. A second source of confusion is the conflation of data collection strategy (on-policy vs. off-policy) with the optimization algorithm itself. Traditional RLHF is associated with on-policy sampling — the policy model generates responses in real-time, the reward model scores them, and the policy updates immediately. DPO is associated with off-policy data — a pre-collected preference dataset is used for training without further interaction. But as recent work like Guo et al. (2024) demonstrates, DPO can be deployed online, and PPO can be run on offline datasets. The algorithm and the data collection strategy are orthogonal design choices, yet prior surveys and much of the literature treat them as inseparable.

The paper formalizes this decoupling explicitly:

"What determines whether the setting is online or offline is merely whether the preference signal is generated in real-time or pre-stored... Therefore, unlike the categorization in other survey papers, we do not use online/offline nor RL/SFT as criteria for classifying algorithms."

By failing to decouple these axes, prior work has made it difficult to ask systematic questions like: "Does online data collection benefit pair-wise contrast methods the same way it benefits point-wise RL methods?" — a question the survey's framework enables but does not yet answer.

3. The optimization objectives are unified, but no one has shown this. The paper's most technical contribution to diagnosing the fragmentation is the observation that the gradient of essentially all preference learning methods can be written in a shared form:

$\nabla_\theta = \mathbb{E}_{(q, o) \sim \mathcal{D}}\left[ \frac{1}{|o|} \sum_{t=1}^{|o|} \delta_{\mathcal{A}}(r, q, o, t) \nabla_\theta \log \pi_\theta(o_t | q, o_{<t}) \right]$

In this formulation, the difference between PPO and DPO is not architectural — it reduces to how the gradient coefficient $\delta$ is computed. In PPO, $\delta$ is the advantage estimate from generalized advantage estimation (GAE), involving a critic model and a reward signal. In DPO, $\delta$ is a function of the log-ratio between the policy and reference model for chosen vs. rejected responses. Both methods ultimately update the policy by scaling the log-probability gradient of generated tokens — but prior work rarely presents them this way, instead emphasizing their surface-level differences (actor-critic architecture vs. contrastive loss).

The authors draw on Shao et al. (2024) for this formulation, but their contribution is to make it the organizing principle of the entire taxonomy, showing that algorithm categories (point-wise, pair-wise, list-wise) correspond directly to how many samples are needed to compute $\delta$ , rather than whether a critic model or KL penalty is involved.

Why This Problem Matters: Practical and Scientific Stakes

The fragmentation is not merely an aesthetic concern — it has real consequences for both research progress and deployment decisions.

For researchers: Without a unified framework, each new method is proposed and evaluated in isolation, making it difficult to identify which components are responsible for improvements. When DPO beats PPO on a benchmark, is it because the contrastive loss is inherently better, or because the offline dataset happened to be higher-quality, or because the KL penalty was tuned differently? The survey's decomposition into data, feedback, algorithm, and model makes it possible to ask these questions systematically. The authors highlight this directly:

"The performance of these algorithms may be inconsistent across different models and task settings. Finding the most effective variant from a theoretical perspective is also a very practical topic."

This inconsistency is a significant drag on the field. New PhD students and engineers entering alignment research face a bewildering landscape of DPO variants (IPO, f-DPO, SimPO, ORPO, sDPO, DPO-positive, etc.) without a clear understanding of which differences are fundamental and which are cosmetic. The survey aims to provide exactly this understanding — not by advocating for any single method, but by revealing the design space itself.

For practitioners: The decoupling of data, feedback, and algorithm has direct implications for resource allocation. If an organization has a large budget for human annotation but limited GPU hours, they should invest in high-quality offline preference datasets and pair them with simple, sample-efficient algorithms (like point-wise methods). If they have abundant compute but concerns about distribution shift, online methods with real-time feedback may be preferable. Prior surveys that conflate these choices — presenting RL as "online" and DPO as "offline" as though these are intrinsic properties rather than deployment decisions — obscure these tradeoffs.

For safety and alignment research: The feedback component of the framework directly connects to the problem of scalable oversight — the challenge of providing reliable feedback for tasks where humans cannot reliably judge correctness (e.g., advanced mathematics, code verification, scientific reasoning). The survey's taxonomy makes explicit that feedback can come from direct rules (unit tests, proof verifiers), trained reward models, or LLM-as-a-judge systems, each with different failure modes and scalability properties. Understanding these failure modes systematically is essential for building alignment pipelines that remain reliable as models exceed human evaluation capabilities.

Prior Surveys and Where They Fall Short

The paper situates itself against three main prior surveys:

Shen et al. (2023) provides a broad overview of LLM alignment but categorizes methods primarily along the RL-vs-SFT axis, replicating the very divide the authors argue is counterproductive.

Wang et al. (2024a) (an earlier arXiv version from Jan 2024) similarly uses the RL-vs-SFT taxonomy and, critically, does not decouple on-policy/off-policy from algorithm choice.

Jiang et al. (2024) surveys human preference learning but, according to the authors' implicit critique, maintains the traditional categorization that treats online/offline as an algorithmic distinction rather than a data collection choice.

What distinguishes this survey is two conceptual innovations:

Innovation 1: The unified gradient coefficient formulation. By showing that all methods optimize the same underlying gradient structure and differ only in $\delta$ , the survey provides a mathematical foundation for the taxonomy rather than relying on surface-level descriptions (e.g., "uses a reward model" vs. "doesn't use a reward model"). This is not a new mathematical result — the authors credit Shao et al. (2024) — but its use as a taxonomic organizing principle is novel.

Innovation 2: Decoupling the four components. Prior surveys tend to present methods as monolithic pipelines: "RLHF works by training a reward model and then running PPO." This survey decomposes that pipeline into four independent choices: Where does the data come from? What produces the feedback signal? How many samples inform each update? Is the model fine-tuned at all? By presenting these as orthogonal axes, the framework reveals combinations that the literature has underexplored — for example, list-wise contrast algorithms (like GRPO) with direct rule-based feedback (like unit tests) on online data, a combination that could combine the sample efficiency of list-wise methods with the reliability of rule-based feedback and the distributional correctness of on-policy data.

How This Paper Positions Itself

The paper is explicitly a taxonomy and synthesis work, not a proposal of a new method. The authors state their goal in the abstract:

"We decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies."

This positions the paper as filling a gap that is increasingly costly as the field accelerates: the number of alignment papers has exploded (the survey covers over 130 citations), and without a shared framework, the community risks rediscovering the same insights under different names or failing to transfer advances across subcommunities.

The paper also positions itself as pragmatic rather than prescriptive. It does not declare DPO superior to PPO, or point-wise methods superior to list-wise methods. Instead, it emphasizes that:

"the core objectives of these alignment algorithms are essentially similar, their performance can vary significantly across different application scenarios."

This is a crucial rhetorical choice. By refusing to pick winners, the survey avoids the trap of becoming outdated as new algorithms emerge — instead, it provides a framework into which new methods can be slotted by identifying what choices they make along each of the four axes.

The paper's scope is deliberately narrowed to textual preference alignment, explicitly excluding multi-modal alignment, hallucination mitigation, and instruction tuning (which are covered in other surveys). This scope constraint is strategic: by focusing on the preference optimization step specifically, the authors can provide depth in the component decomposition without being pulled into adjacent but distinct research areas.

Finally, the paper positions its framework as enabling future research rather than closing the book. The future directions section explicitly calls out combination strategies that the taxonomy reveals as underexplored — for instance, combining advanced sampling techniques (on-policy data) with list-wise contrastive losses and LLM-as-a-judge feedback, a configuration that no existing method implements but that the framework suggests should be possible and potentially high-performing.

The Intellectual Lineage: From RLHF to a Unified Design Space

To understand the paper's contribution, it helps to see the historical progression it responds to:

2017-2022: The RLHF paradigm solidifies. PPO with a trained reward model becomes the dominant alignment method, exemplified by InstructGPT (Ouyang et al., 2022) and subsequently ChatGPT. The paradigm is effective but computationally expensive (requires maintaining four models: policy, reference, reward, critic), sample-inefficient, and sensitive to hyperparameters.
2023: The SFT-based alternative emerges. DPO (Rafailov et al., 2023) demonstrates that the reward model can be eliminated entirely by reparameterizing the RLHF objective as a contrastive loss. This is hailed as a breakthrough for simplicity, but it comes with its own limitations — sensitivity to dataset quality, tendency to overfit, and the implicit assumption that the offline preference data is representative of the deployment distribution.
2023-2024: Proliferation of variants. The success of DPO sparks a wave of modifications (IPO to prevent overfitting, f-DPO to generalize the divergence constraint, SimPO to remove the reference model, ORPO to combine SFT and alignment into one stage). Simultaneously, the RL camp produces PPO simplifications (ReMax, GRPO) and the feedback camp explores alternatives to trained reward models (LLM-as-a-judge, direct rule-based feedback, generative verifiers). Each paper proposes a new loss function or training procedure, but the relationships between these proposals are rarely discussed.
This survey (2024): The authors survey this rapidly expanding design space and identify the four-axis decomposition as the minimal structure needed to organize it. By showing that all methods optimize the same gradient form and differ only in how $\delta$ is computed (which in turn depends on data source, feedback type, and number of samples per update), they provide a unified vocabulary for discussing tradeoffs that were previously implicit.

The survey's significance is thus primarily architectural: it provides the conceptual infrastructure for the field to transition from proposing isolated methods to reasoning systematically about design choices, much as the transformer architecture provided a shared vocabulary for discussing innovations in attention mechanisms, positional encodings, and normalization strategies.

3. Technical Approach

3.1 Reader Orientation

This paper is fundamentally a taxonomy and synthesis work — not a proposal of a new algorithm or training procedure, but a conceptual framework that reorganizes the existing landscape of preference learning methods for large language models (LLMs) into four orthogonal, composable components. The core insight is that all preference alignment algorithms, despite their surface-level diversity (PPO vs. DPO, online vs. offline, RL vs. SFT), share a common mathematical structure and can be understood as different choices along four independent axes: model (the LLM being aligned), data (where training samples come from), feedback (how preference signals are generated), and algorithm (how many samples inform each gradient update and what loss is computed). By revealing this shared structure, the survey enables systematic reasoning about tradeoffs, identifies underexplored combinations, and provides a shared vocabulary for a field that the authors argue has become unnecessarily fragmented.

3.2 Big-Picture Architecture (Diagram in Words)

The survey's framework decomposes preference learning into a pipeline with four interacting components:

Model — the LLM $\pi_\theta$ that we wish to align with human preferences. This is the object being optimized throughout the process. A reference model $\pi_{ref}$ (typically a frozen copy of the initial policy) may optionally be maintained for KL-divergence regularization, depending on the algorithm chosen.
Data — the source of training samples $(x, y, r)$ , where $x$ is a prompt, $y$ is a candidate response, and $r$ is a preference label or reward signal. Data can be collected on-policy (generated in real-time by the current policy $\pi_\theta$ as training progresses) or off-policy (pre-collected from external sources such as human annotators, other LLMs, or an earlier checkpoint of the policy itself).
Feedback — the mechanism that produces the preference signal $r$ for a given $(x, y)$ pair. Feedback can be direct (derived from hand-designed rules like mathematical correctness, unit test pass/fail, or pre-existing human preference labels in a dataset) or model-based (produced by a trained reward model, a pairwise scoring model, or an LLM-as-a-judge prompted to evaluate quality).
Algorithm — the optimization procedure that consumes $(x, y, r)$ tuples (and optionally a reference model) and updates the policy parameters $\theta$ . Algorithms are categorized by how many samples inform the gradient coefficient $\delta$ in the unified gradient formula (Equation 1): point-wise methods use a single sample, pair-wise contrast methods compare two samples (chosen vs. rejected), and list-wise contrast methods evaluate an entire ranked list of samples. A fourth category, training-free alignment, optimizes outputs without updating model parameters at all (via prompt engineering, logits manipulation, or output rewriting).

Information flows sequentially: the data source produces candidate responses → the feedback mechanism scores or labels them → the algorithm computes a gradient update from these scored samples and applies it to the model. The key architectural claim is that these four components are independent design axes: any choice of data source can be combined with any feedback type and any algorithm, and the survey is organized as a tour through each axis separately.

3.3 Roadmap for the Deep Dive

First, the unified gradient coefficient formulation (Equation 1), since it is the mathematical linchpin that shows all methods optimize the same underlying objective and differ only in how $\delta$ is computed. Understanding this formula reveals why the data/feedback/algorithm decomposition is natural rather than arbitrary.
Second, the data axis (Section 4 of the paper), separated into on-policy and off-policy collection strategies, because the quality and distribution of training data fundamentally determine what any algorithm can achieve — no optimization procedure can compensate for systematically biased or low-diversity data.
Third, the feedback axis (Section 5), which determines the reward signal $r$ that feeds into the gradient coefficient computation. We examine direct feedback (rule-based signals requiring no trained model) and model-based feedback (reward models, pairwise scorers, LLM-as-a-judge), each with distinct tradeoffs in cost, reliability, and domain generality.
Fourth, the algorithm axis (Section 6), where we walk through the four categories — point-wise, pair-wise contrast, list-wise contrast, and training-free — explaining the loss functions, underlying assumptions, and representative methods within each. This section includes the detailed running examples from Figures 3 and 4.
Fifth, the evaluation axis (Section 7), which the framework treats as a necessary complement to training, covering rule-based benchmarks and LLM-based evaluation with their respective biases and limitations.
Sixth, we'll synthesize how the framework enables cross-category reasoning, illustrating with examples of how the taxonomy reveals underexplored algorithm-feedback-data combinations.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a taxonomy paper whose central contribution is a four-axis decomposition of preference learning, unified by the observation that all alignment methods optimize a shared gradient form (Equation 1) and differ only in how the gradient coefficient $\delta$ is computed from data, feedback, and algorithm choices.

The Unified Gradient Coefficient Formulation

The paper introduces a shared mathematical structure that underlies both reinforcement-learning-based methods (PPO and its variants) and supervised-finetuning-based methods (DPO and its variants). The authors state this explicitly:

"the optimization objectives of RL and SFT-based methods can be described within the same framework."

The gradient of the optimization objective with respect to the policy parameters $\theta$ is written as:

$\nabla_\theta = \mathbb{E}_{(q, o) \sim \mathcal{D}}\left[ \frac{1}{|o|} \sum_{t=1}^{|o|} \delta_{\mathcal{A}}(r, q, o, t) \nabla_\theta \log \pi_\theta(o_t | q, o_{<t}) \right]$

where $\mathcal{D}$ denotes the data source containing input questions $q$ and candidate outputs $o$ , $|o|$ is the length (number of tokens) of the output sequence, $t$ indexes over tokens in the output, $\pi_\theta(o_t | q, o_{<t})$ is the policy's predicted probability of token $o_t$ given the question and preceding tokens, $\nabla_\theta \log \pi_\theta(o_t | q, o_{<t})$ is the standard policy gradient term — the direction in parameter space that increases the log-probability of token $o_t$ — and $\delta_{\mathcal{A}}(r, q, o, t)$ is the gradient coefficient determined by the specific algorithm $\mathcal{A}$ , the reward or preference signal $r$ , and the current position $t$ in the sequence.

What it computes: this equation describes the parameter update for any preference learning method. For each token $o_t$ in each generated output $o$ from the data distribution $\mathcal{D}$ , the algorithm computes a scalar coefficient $\delta$ that captures how "good" or "bad" that token is (relative to some baseline or comparison). This coefficient then scales the policy gradient $\nabla_\theta \log \pi_\theta(o_t)$ , which points in the direction that most steeply increases the probability of that token. Multiplying by $\delta$ determines both the magnitude and sign of the update: a positive $\delta$ means "make this token more likely," a negative $\delta$ means "make this token less likely," and the absolute value $|\delta|$ controls the step size. The expectation over the data distribution $\mathcal{D}$ ensures the gradient is estimated from representative samples, and the sum over tokens applies this token-level reinforcement to the entire generated sequence.

Why this form: the expression decomposes preference learning into two cleanly separated concerns. The policy gradient term $\nabla_\theta \log \pi_\theta(o_t)$ is universal across all methods — it is the direction in which the model's parameters must move to increase the assigned probability to token $o_t$ at position $t$ . The gradient coefficient $\delta$ is where all the algorithmic variation lives. In PPO, $\delta$ is the advantage estimate (reward minus value baseline), computed using a separate critic network and generalized advantage estimation. In DPO, $\delta$ is a function of the log-ratio between the policy and reference model for chosen versus rejected responses, derived from the Bradley-Terry preference model. In Rejection Sampling Fine-Tuning (RFT), $\delta$ is simply 1 for correct samples and 0 (no update) for incorrect ones. This unified form explains why the four-axis decomposition (data, feedback, algorithm, model) is natural: $\mathcal{D}$ corresponds to the data choice, $r$ corresponds to the feedback choice, $\mathcal{A}$ (which determines how $\delta$ is computed from $r$ ) corresponds to the algorithm choice, and $\pi_\theta$ is the model being updated.

Critically, the authors emphasize that the feedback $r$ in this framework is broadly defined — it is not limited to scalar rewards from a trained reward model. As they state:

"the feedback in our paper as the preference given by the environment that can affect the gradient coefficient."

This means $r$ can be a binary correctness label (in RFT), a scalar from a reward model (in PPO), a preference ordering between two responses (in DPO), or any other signal that influences $\delta$ . This breadth is what allows the framework to encompass methods as diverse as rule-based filtering and LLM-as-a-judge.

A second critical point: the authors explicitly decouple the algorithm from the online/offline setting:

"the algorithm can be decoupled from online/offline settings... what determines whether the setting is online or offline is merely whether the preference signal is generated in real-time or pre-stored."

In this framework, "online" means the feedback $r$ is obtained in real-time from the environment during training (e.g., scoring freshly sampled responses with a reward model), while "offline" means the feedback is pre-stored in a dataset. Both PPO and DPO can operate in either regime — the algorithm choice is orthogonal to the data collection strategy. This is a significant departure from prior surveys that categorized PPO as "online RL" and DPO as "offline SFT," which the authors argue obscures the underlying design space.

The Preference Learning Pipeline (Algorithm 1)

The paper formalizes the overall preference learning process as a simple algorithm (Algorithm 1 in the text), which we can walk through step by step:

Input: an LLM $\pi_\theta$ to be aligned, an environment $\mathcal{E}$ that embodies human preference (this could be a dataset with labels, a reward model, a set of rules, or a human annotator), unlabeled queries $\mathcal{Q}$ (for online settings) or a pre-prepared offline dataset $\mathcal{D}$ , and an algorithm $\mathcal{A}$ .

Step 1 (optional): if the algorithm requires a reference model (as in PPO and DPO, where a KL-divergence penalty prevents the policy from drifting too far from its initial behavior), the reference model $\pi_{ref}$ is initialized as a frozen copy of $\pi_\theta$ .

Step 2 (loop): training proceeds for a fixed number of steps. In each step:

If online: a batch $\mathcal{B}$ of candidate responses is sampled from the current policy $\pi_\theta$ using the unlabeled queries $\mathcal{Q}$ . The environment $\mathcal{E}$ provides real-time feedback $\mathcal{R}$ (scores, labels, or preference orderings) for these freshly generated responses.
If offline: a batch $\mathcal{B}$ of data with pre-stored preference feedback is drawn from the pre-prepared dataset $\mathcal{D}$ .
Update: the tuple $(\mathcal{B}_{(x,y)}, \mathcal{R}, \pi_\theta, \pi_{ref})$ is fed into the algorithm $\mathcal{A}$ , which computes the gradient (via Equation 1 with algorithm-specific $\delta$ ) and updates the policy parameters to produce $\pi_{\theta'}$ .
Advance: the updated policy $\pi_{\theta'}$ becomes $\pi_\theta$ for the next iteration.

Output: the aligned LLM $\pi_{\theta'}$ .

This pipeline is deliberately generic — it does not commit to a specific data source, feedback mechanism, or algorithm. The entire taxonomy in Sections 4–6 is essentially a catalog of choices one can make at each step of this pipeline.

Preference Data: On-Policy vs. Off-Policy Collection

The data component of the framework concerns where training samples come from. The paper introduces a precise definition:

"we use the simplest notation to represent preference data as $(x, y, r)$ . Here $x, y$ are the literal information input and the candidate output. $r$ is a preference label given by certain feedback systems."

The critical distinction is between on-policy and off-policy data collection. The authors define this crisply:

"the on-policy data collection means we collect the data directly from our policy LLM $\pi_{\theta_t}$ at each training step $t$ . The off-policy data collection could be done outside the box, independent from the LLM to conduct preference learning and result in a dataset consisting of data that is not generated by the policy model itself."

A subtle but important point: using data sampled from an earlier checkpoint $\pi_{\theta_0}$ to train a later checkpoint $\pi_{\theta_t}$ for $t > 0$ counts as off-policy, even though it came from the same model family. The distinction is temporal: if the data was generated by a policy that is no longer the current policy, it is off-policy with respect to the current optimization step.

On-Policy Data Collection

On-policy collection is analogous to the standard setting in on-policy reinforcement learning. The process is: sample a batch of responses from the current policy, obtain rewards by querying the environment (which could be a reward model, a set of rules, or any other feedback mechanism), and immediately use those rewards to update the policy before sampling a new batch from the updated policy.

Sampling strategies. The paper surveys several decoding strategies used during on-policy generation to ensure diversity in the sampled responses. Top-K/Nucleus Sampling (Holtzman et al., 2020) truncates the token distribution to the top-K most likely tokens or to the smallest set whose cumulative probability exceeds a threshold, preventing degenerate repetition while maintaining diversity. Beam Search (Graves, 2012) maintains multiple candidate hypotheses in parallel and selects the overall highest-scoring sequence, trading diversity for quality.

For problems requiring multi-step reasoning (mathematics, code generation, theorem proving), the paper highlights Monte Carlo Tree Search (MCTS) (Kocsis and Szepesvári, 2006) as an emerging sampling strategy. MCTS operates through four phases: selection (traversing the existing search tree to find a promising leaf node), expansion (adding one or more child nodes representing possible next steps), simulation (completing the trajectory from the expanded node using a rollout policy), and backpropagation (updating the value estimates of all nodes along the traversed path based on the simulation outcome). The paper notes that MCTS has been employed both for training reward models — because the tree search provides natural step-level preference labels as a byproduct — and for directly improving decoding quality. Recent works cited include applications to mathematical reasoning, where MCTS-generated data improves both verifier training and policy fine-tuning.

Why MCTS for preference learning data generation? The paper's argument is that MCTS produces higher-quality and more diverse data than independent sampling because it systematically explores the space of possible reasoning paths, evaluating promising branches more deeply rather than relying on random exploration. This is particularly important for tasks where the correct solution requires a specific sequence of reasoning steps and random sampling would rarely stumble upon it — MCTS can discover these rare correct trajectories and add them to the training data.

Off-Policy Data Collection

Off-policy data collection is decoupled from the training process: the dataset is assembled in advance, and training samples are drawn from it without requiring the policy to generate fresh responses.

The paper identifies two main sources of off-policy preference data:

Data from humans. Three representative datasets are described:

WebGPT (Nakano et al., 2021): contains 20K comparisons, each consisting of a question, a pair of model-generated answers, and human-rated preference scores for each answer. The task is web-assisted question answering, where the model retrieves and synthesizes information from web pages.
OpenAI's Human Preferences (Ouyang et al., 2022): derived from a filtered subset of the Reddit TL;DR corpus (Völske et al., 2017). Each example contains a Reddit post, two candidate summaries generated by earlier model checkpoints, and a human annotator's preference label indicating which summary is better.
HH-RLHF (Bai et al., 2022): approximately 170K dialogues between humans and AI assistants, where the AI provides two different responses to each human query and a human annotator selects the preferred one. The dataset is designed to train models that are both helpful (provide useful information) and harmless (refuse to engage with harmful requests).
SHP (Ethayarajh, Choi, and Swayamdipta, 2022): 385K human preference annotations across 18 subject areas, derived entirely from naturally occurring human-written data (Reddit). Unlike HH-RLHF, which mixes model-generated and human data, SHP's preferences come from Reddit users' upvotes and community judgments, providing a different distributional signal.

Data from LLMs. Since human annotation is expensive and slow to scale, the paper highlights a growing trend of using strong LLMs to generate preference labels:

RLAIF (Lee et al., 2023): uses PaLM 2 to annotate preferences on a merged dataset combining Reddit TL;DR, OpenAI's Human Preferences, and HH-RLHF. The key finding is that scaling up with AI feedback significantly improves training performance, sometimes matching or exceeding human-annotated data.
Open-Hermes-Preferences (Huang et al., 2024): roughly 1 million AI-generated preference pairs, integrating outputs from multiple models and using PairRM (a lightweight pairwise scoring model) as the preference oracle.
ULTRAFEEDBACK (Cui et al., 2023): GPT-4 is used as the annotator to produce a large-scale, diverse preference dataset. Unlike earlier datasets that focus on specific tasks (summarization, dialogue), ULTRAFEEDBACK aims for broad coverage across many types of user instructions.
UltraChat (Ding et al., 2023): a million-scale multi-turn instructional conversation dataset generated by two separate ChatGPT Turbo API instances that simulate a user and an assistant engaging in diverse, information-seeking dialogues.

Why the distinction matters for the unified framework. The paper's key point is that any algorithm can be paired with either data collection strategy. DPO, typically presented as an offline method, can be run online if a real-time feedback source is available — sample responses from the current policy, obtain preference labels (e.g., via an LLM-as-a-judge), and compute the DPO loss on the freshly labeled pairs. Conversely, PPO, typically presented as online, can be run on offline data if the dataset includes scalar rewards for each response. This decoupling is a central conceptual move of the survey: it transforms "method X is online/offline" from an intrinsic property into a deployment choice.

Feedback: Direct vs. Model-Based Preference Signals

The feedback component addresses the question: given a candidate response $\hat{y}$ for a prompt $x$ , what signal tells us how well it aligns with human preferences? The paper defines feedback broadly:

"the feedback in this paper refers broadly to the preference indicators that can influence the gradient of the model during the training process."

This feedback can be a scalar reward (for RL-based methods like PPO), a binary preference label (for pair-wise contrast methods like DPO), or any other signal that modifies the gradient coefficient $\delta$ in Equation 1.

The paper categorizes feedback into two types based on whether a separate model must be trained:

Direct Feedback (No Trained Model Required)

Labeled datasets. The simplest feedback source: human annotators have already labeled preferences in a dataset, and these labels are used directly during training. This is the standard setting for offline DPO, where the dataset contains $(x, y^+, y^-)$ triples — a prompt, a chosen (preferred) response, and a rejected (dispreferred) response. No additional model or computation is needed to produce the preference signal; the labels are simply read from the dataset.

Hand-designed rules. For tasks with objective correctness criteria, rule-based feedback provides a clean, cheap, and reliable alternative to human annotation or trained reward models. The paper gives examples across several domains:

Mathematical reasoning (Yuan et al., 2023): the reward is $r = \mathbb{I}(c)$ , where $c$ indicates whether the chain-of-thought reasoning path leads to the correct final answer. This is a binary signal — 1 for correct, 0 for incorrect — and requires only a ground-truth answer for comparison. The simplicity of this reward belies its effectiveness: Yuan et al. showed that filtering training data to keep only correct reasoning paths (rejection sampling fine-tuning, or RFT) significantly improves math reasoning performance.
Theorem proving (Xin et al., 2024): the Lean proof assistant automatically verifies whether a generated proof is correct. The feedback is deterministic and objective — the proof either type-checks in Lean's kernel or it does not — eliminating the ambiguity that plagues human or LLM judgment of proof quality.
Machine translation (Xu et al., 2024): a reference-free Quality Estimation (QE) model scores translation candidates without requiring human reference translations, and these scores provide the preference signal. CPO (Contrastive Preference Optimization) adapts the DPO algorithm to use these QE scores as the preference oracle.
Code generation (Shen et al., 2023; Dou et al., 2024): unit tests and heuristic preferences (e.g., compilation success, runtime behavior) are converted into scalar rewards. Shen et al. assign different scores based on how completely the generated code passes test cases, while Dou et al. design rules that convert different test outcomes (compile error, runtime error, incorrect output, correct output) into a discrete reward scale for RL training.
Summarization (Gao et al., 2024): human edits on model-generated summaries serve as a direct learning signal. When a user edits the model's summary to correct an error or improve clarity, the edit path provides natural preference data — the edited version is "chosen," the original is "rejected." The paper frames this as a form of interactive learning from implicit feedback.

Why direct feedback matters for the framework. Rule-based feedback is domain-specific but extremely reliable within its domain — a unit test either passes or fails, a proof assistant either accepts or rejects. This reliability avoids the reward hacking and over-optimization problems that plague trained reward models. The taxonomy makes explicit that feedback type is an independent choice from algorithm: you can use rule-based feedback with PPO (as in code generation RL), with DPO (by constructing preference pairs from test outcomes), or with list-wise methods (ranking responses by test pass rates). This cross-compatibility is a design insight that the framework reveals but individual method papers rarely discuss.

Model-Based Feedback (Requires Training or Prompting a Separate Evaluator)

When tasks lack objective correctness criteria (e.g., dialogue helpfulness, summary quality, creative writing), a trained model must supply the preference signal. The paper identifies three subcategories:

Reward models. The dominant paradigm for model-based feedback involves training a separate model that predicts human preference. The paper describes two formulations:

Bradley-Terry based reward models. The most common approach, grounded in the Bradley-Terry model of paired comparisons. The model estimates the probability that one response is preferred over another:

$p^*(y_1 \succ y_2 | x) = \frac{\exp(r^*(x, y_1))}{\exp(r^*(x, y_1)) + \exp(r^*(x, y_2))}$

where $r^*(x, y)$ is the latent reward score that the model assigns to response $y$ given prompt $x$ , and $y_1 \succ y_2$ denotes that $y_1$ is preferred over $y_2$ .

What it computes: given a prompt $x$ and two candidate responses $y_1$ and $y_2$ , this formula computes the probability that $y_1$ is the human-preferred response, under the assumption that preferences follow a logistic distribution over latent reward differences. The exponential transformation ensures probabilities are positive, and the denominator normalizes them to sum to 1.

Why this form: the Bradley-Terry model is the standard statistical model for paired comparison data. It assumes that each response has an underlying "quality" score $r^*$ , and the probability of preferring $y_1$ over $y_2$ depends only on the difference $r^*(x, y_1) - r^*(x, y_2)$ . When this difference is large and positive, the probability approaches 1; when negative, it approaches 0; when zero, both responses are equally likely to be preferred. The logistic sigmoid $\sigma(z) = 1/(1 + e^{-z})$ maps real-valued score differences to the [0,1] interval. This formulation has the crucial property that it is transitive: if $y_1 \succ y_2$ and $y_2 \succ y_3$ , then $y_1 \succ y_3$ , which is a reasonable assumption for preference data (unlike, say, tournament outcomes where intransitive cycles can occur).

The reward model is trained to maximize the likelihood of observed human preferences using a negative log-likelihood loss:

$\mathcal{L}_r = -\log \sigma(r^*(y_c, x) - r^*(y_r, x))$

where $y_c$ is the chosen (preferred) response and $y_r$ is the rejected (dispreferred) response. Minimizing this loss increases the gap between the reward assigned to chosen responses and the reward assigned to rejected responses.

Binary classifier based reward models. For tasks with clear outcome signals, a reward model can be trained as a simple binary classifier:

$\mathcal{L}_r = -\left[r \log(\hat{r}) + (1 - r) \log(1 - \hat{r})\right]$

where $r \in \{0, 1\}$ is the binary label (1 for correct/successful, 0 for incorrect/failed) and $\hat{r} \in [0, 1]$ is the model's predicted probability of success.

What it computes: standard binary cross-entropy loss. The model is trained to predict whether a given $(x, y)$ pair is successful, and its predicted probability $\hat{r}$ serves as the reward signal for downstream RL or filtering.

Why this form is preferable to Bradley-Terry for some tasks: for tasks like math reasoning and code generation, correctness is objective — we can label samples as correct or incorrect directly. The binary classifier formulation is simpler than Bradley-Terry (no need to construct pairs) and avoids the assumption that preferences follow a logistic distribution, which may not hold when the quality gap between correct and incorrect responses is essentially binary rather than continuous.

Reward model training optimizations. The paper surveys several research directions for improving reward models:

Better preference data: RLAIF (Lee et al., 2023) scales up using AI-generated labels. Jinnai et al. (2024) regularize Best-of-N sampling using KL divergence and Wasserstein distance to mitigate reward hacking — the phenomenon where the policy learns to exploit quirks in the reward model rather than genuinely improving quality. Pace et al. (2024) introduces West-of-N, a technique for generating synthetic preference data that extends Best-of-N sampling from policy training to reward model training.
Model ensembling: Coste et al. (2024) and Zhang et al. (2024) use ensembles of reward models to reduce over-optimization — the tendency of RL fine-tuning to find adversarial examples that score highly under the reward model but are actually low-quality. Quan (2024) proposes a Mixture-of-Experts reward model that decomposes evaluation into different capability dimensions. Zhai et al. (2024) use LoRA-based ensembles for efficient uncertainty estimation. Ramé et al. (2024) average reward model weights (rather than predictions) for improved robustness.
Fine-grained rewards: Wu et al. (2023) introduce Fine-Grained RLHF, which provides rewards at multiple granularities (e.g., per sentence or per segment) rather than a single scalar for the full response. Yang et al. (2024b) regularize reward model training through text-generation objectives to improve generalization.
Process supervision vs. outcome supervision: Uesato et al. (2022), Lightman et al. (2023), and Yu, Gao, and Wang (2024) explore process reward models (PRMs) that provide rewards for each intermediate reasoning step, rather than only at the final answer. This enables more granular credit assignment — a solution that reaches the right answer through flawed reasoning can be penalized, while a partially correct approach can be rewarded even if the final answer is wrong. Wang et al. (2024c) proposes Math-Shepherd, which constructs process supervision data in an unsupervised manner by using Monte Carlo rollouts to estimate the probability of reaching the correct answer from each intermediate step, eliminating the need for expensive human step-level annotations.
Training process optimization: Dong et al. (2024) and Zhou et al. (2024) use prior constraints to prevent reward scores from growing unboundedly during training — a common failure mode where the reward model's output scale drifts, making KL-divergence penalties ineffective. Gao et al. (2024b) proposes a two-stage training paradigm for mathematical reward models that uses natural language feedback to improve evaluation capability.

Pair-wise scoring models. A lighter-weight alternative to full reward models: specialized pairwise comparison models that directly predict which of two responses is better without producing a global quality score. The paper highlights PairRanker from LLM-Blender (Jiang, Ren, and Lin, 2023), which with only 0.4B parameters achieves high correlation with ChatGPT-based rankings and is used as a preference oracle in works like SPIN and SimPO. The advantage is simplicity and consistency — discriminating between two candidates is often easier than assigning an absolute score, especially for smaller models. The disadvantage is that pairwise models cannot provide a global score (making list-wise ranking expensive) and are limited in the number of candidates they can compare at once.

LLM-as-a-judge. The most direct model-based approach: prompt a powerful LLM (typically GPT-4) to evaluate the quality of generated responses according to specified criteria. The paper describes this as:

"we can specify scoring rules directly in the prompts, allowing the model to score generated responses."

Self-rewarding mechanisms (Yuan et al., 2024) extend this further: the LLM evaluates its own responses and uses those evaluations to improve, creating a self-improvement loop without external annotators. Meta-Rewarding (Wu et al., 2024) adds a second-order evaluation step where the model assesses the quality of its own judgments, using that meta-feedback to refine its evaluation capability.

The paper notes that LLM-as-a-Judge underperforms trained verifiers for complex reasoning tasks (mathematics, code verification), motivating specialized approaches: CriticGPT (McAleese et al., 2024) trains a critic model that outputs plain-text critiques identifying potential problems in code, and generative verifiers (Zhang et al., 2024b) leverage the text-generation capabilities of LLMs to verify mathematical reasoning by predicting correctness as a next-token prediction task.

Cost-reliability tradeoffs across feedback types. The taxonomy makes implicit tradeoffs explicit. Rule-based feedback is the cheapest and most reliable but only applies to domains with objective correctness criteria. Trained reward models are more general but require substantial investment in data collection and training, and they suffer from over-optimization and distribution shift. LLM-as-a-judge is the most flexible (can evaluate any text according to any criteria) but is expensive per evaluation, may introduce biases (verbosity preference, self-preference), and is unreliable for complex reasoning. The framework treats these as independent choices from algorithm selection, enabling systematic comparison of, for example, "DPO with rule-based feedback" vs. "DPO with LLM-as-a-judge feedback" on tasks where both are applicable.

Training-Free Alignment: Optimizing Without Updating Parameters

The paper identifies a fourth algorithm category that operates differently from the gradient-based methods: training-free alignment, which modifies outputs without updating the model's parameters. This category is conceptually distinct because $\delta = 0$ for all tokens — the gradient in Equation 1 is never computed.

The paper divides training-free methods into two subcategories based on where the intervention occurs:

Input Optimization (Modify the Prompt)

URIAL (Lin et al., 2023) analyzes the token distribution differences between base (unaligned) and aligned models, finding that most shifts occur in stylistic tokens rather than content tokens. The method aligns a base model by providing carefully designed system prompts and restyling in-context learning examples — essentially, the alignment is "prompted" rather than trained.

OPO (Xu et al., 2023) retrieves social norms relevant to the given prompt from a pre-built norm database and includes them in the context, guiding the model toward norm-compliant behavior without modifying its weights.

BPO (Cheng et al., 2023) trains a separate sequence-to-sequence model that acts as a prompt rewriter — it takes a potentially suboptimal prompt and rewrites it into a form that elicits better-aligned responses from the frozen LLM. The rewriting model is trained on low-quality/high-quality prompt pairs generated by ChatGPT.

Output Optimization (Modify the Response Distribution or Post-Process Outputs)

Paraphrasing. Aligner (Ji et al., 2024) trains a separate "alignment module" that operates as a post-processing step. During inference, the original model generates an initial (potentially unaligned) response; this response, along with the original instruction, is fed into the alignment module, which produces a rewritten, better-aligned response.

Logits manipulation. FUDGE (Yang and Klein, 2021), DeAL (Huang et al., 2024), and related methods modify the model's output probability distribution during decoding. A separate discriminator or reward model scores candidate tokens or sequences, and these scores are combined with the model's original log-probabilities to bias generation toward aligned outputs. Formally, the sampling distribution becomes $p(y_t | x, y_{<t}) \propto p_{\text{LM}}(y_t | x, y_{<t}) \cdot p_{\text{align}}(y_t | x, y_{<t})^\alpha$ , where $\alpha$ controls the strength of the alignment bias.

Searching. RAIN (Li et al., 2023a) implements "rewindable decoding" — the model generates tokens step by step, evaluating each token for harmfulness using the LLM itself as the evaluator. When a harmful token is encountered, the model backtracks and tries an alternative token. This creates a self-correcting generation process without any parameter updates. DeAL (Huang et al., 2024) replaces self-evaluation with a custom reward model for more fine-grained preference control.

ICDPO (Song et al., 2024b) designs a two-stage Best-of-N-like process at inference time. Multiple candidate responses are sampled from a local (non-aligned) LLM, and the system selects the best-aligned response using a scoring mechanism that aggregates the model's own internal states before and after in-context learning, without requiring an external verifier. The key innovation is that the selection criterion is derived from the model itself — comparing how the model's representations change when exposed to high-quality demonstrations — rather than relying on a trained reward model.

Why training-free methods are included in the framework. The unified gradient formulation (Equation 1) technically assumes gradient-based optimization, which training-free methods do not perform. The paper includes them because they are part of the broader alignment landscape and share the same ultimate goal — producing LLM outputs that better align with human preferences. By placing them within the same four-axis framework, the survey enables apples-to-apples comparisons: a training-free method with rule-based output filtering might be compared against a DPO variant with rule-based preference data, revealing tradeoffs between compute cost (training-free methods require no gradient updates but may require more inference-time computation) and alignment quality.

Summary of the Unified Framework's Design Choices

The paper's framework is not a new method but a conceptual reorganization of existing methods along four axes. The key design choices that make this reorganization valuable are:

The unified gradient formulation (Equation 1) reveals that all gradient-based methods are instances of the same template, differing only in the scalar coefficient $\delta$ . This demystifies the RL-vs-SFT boundary: PPO and DPO are not fundamentally different families of algorithms but different choices for computing $\delta$ from the available feedback.
Decoupling algorithm from data collection strategy (online vs. offline) transforms what prior surveys treated as algorithmic properties into independent deployment decisions. This opens up combination strategies that the field has underexplored — online DPO (as in Guo et al., 2024), offline PPO on pre-collected reward-labeled data, or list-wise contrast methods applied to on-policy data.
Treating feedback as a first-class, independently variable component enables systematic reasoning about cost-reliability-domain coverage tradeoffs. The framework makes it natural to ask: "For this task, should I invest in training a reward model, use an LLM-as-a-judge, or rely on rule-based signals?" — a question that individual method papers rarely address because they typically commit to one feedback source.
Categorizing algorithms by the number of samples per update (point-wise, pair-wise, list-wise) provides a more informative taxonomy than RL-vs-SFT because it directly captures a key computational and data requirement: point-wise methods need only per-sample scores, pair-wise methods need comparisons between two samples, and list-wise methods need a ranked list. This affects both data collection costs and optimization stability.
Including training-free methods within the same framework, despite their different mechanism, enables holistic comparison across the full alignment design space rather than treating prompt-based and gradient-based alignment as separate research areas.

4. Key Insights and Innovations

Innovation 1: The RL-vs-SFT Divide Is an Artifact, Not a Fundamental Distinction

The paper's most significant conceptual move is to demonstrate that the taxonomy the field has used to organize alignment research — reinforcement learning methods (PPO, ReMax) versus supervised finetuning methods (DPO, KTO) — is not a reflection of any deep mathematical difference between these approaches, but rather a historical accident of terminology and community convention that actively impedes cross-pollination of ideas.

Prior state of the field. Before this survey, the dominant organizational framework in alignment papers and surveys (Shen et al., 2023; Wang et al., 2024a; Jiang et al., 2024) split methods into two buckets: RL-based methods that train a reward model and use it for online optimization, and SFT-based methods that directly optimize on pre-collected preference data. This taxonomy had surface plausibility — PPO involves a reward model, a critic, and advantage estimation, while DPO involves a contrastive loss with a reference model — and it mapped cleanly onto the chronological development of the field (RLHF emerged first, DPO was proposed as a simplification). However, it created the impression that these are fundamentally different optimization paradigms requiring different conceptual toolkits, different infrastructure, and different research traditions.

What the paper reveals. The unified gradient coefficient formulation (Equation 1) exposes this divide as superficial. Under this formulation, PPO and DPO are instances of exactly the same optimization template: for each token in each generated output, compute a scalar coefficient δ and multiply it into the policy gradient. In PPO, δ is the advantage estimate from GAE; in DPO, δ is a function of the log-ratio between the policy and reference model for chosen versus rejected responses. Both methods update model parameters by scaling the log-probability gradient of output tokens — the only difference is how δ is computed, which in turn depends on the available feedback (scalar reward vs. binary preference label) and the design choice of how many samples inform each update.

This is a fundamental reframing, not an incremental observation. It transforms the question "should I use RL or SFT?" from a choice between competing paradigms into a design decision within a shared framework: given my task, my available feedback, and my compute budget, what is the best way to compute the gradient coefficient δ? The frame shift matters because it enables systematic comparison along the dimensions that genuinely differ — feedback type, data collection strategy, number of samples per update — rather than fighting phantom paradigm wars.

Evidence and significance. The paper demonstrates this unification not through a new theorem (the gradient formulation is credited to Shao et al., 2024) but through architectural synthesis: by showing that every major alignment algorithm — RFT, PPO, ReMax, KTO, DPO, IPO, SimPO, ORPO, GRPO, RRHF, PRO — can be expressed in this form and differentiated only by δ, the paper converts an implicit intellectual structure (some researchers already sensed these methods were related) into an explicit framework that can guide future research. The practical consequence is that innovations in one "camp" become immediately transferable: online data collection techniques from the RL literature can be applied to pair-wise contrast methods; the reference-model-free insight from SimPO can inform PPO variants; the list-wise normalization trick from GRPO can be evaluated in the context of DPO-like losses.

The paper's key evidence for the unification is not empirical but taxonomic: the framework accommodates over 100 methods from across the RL-SFT spectrum without requiring special cases or exceptions, suggesting it captures something genuine about the optimization structure rather than forcing an artificial consistency. This is a form of theoretical contribution common in survey papers — not proving a new theorem, but revealing that existing theorems share a common structure that the literature had overlooked.

Innovation 2: Decoupling Algorithm, Data, and Feedback as Independent Design Axes

The paper's second major conceptual contribution complements the first: having shown that all methods share a common gradient form, it identifies four orthogonal axes along which any preference learning approach makes independent design choices. This decoupling is not merely a taxonomic convenience — it directly enables research questions that the prior RL-vs-SFT taxonomy made invisible.

Prior state of the field. In the prevailing RL-vs-SFT taxonomy, certain design choices were treated as intrinsic properties of algorithm families: PPO is online (because it samples from the current policy during training), DPO is offline (because it uses pre-collected preference datasets), PPO requires a trained reward model, DPO does not. These associations were empirically common but logically contingent — PPO was typically deployed online with a reward model, DPO was typically deployed offline with pre-labeled data, but there was no mathematical reason that PPO could not be run on offline data with stored rewards, nor that DPO could not be run online with real-time preference labeling. The taxonomy thus conflated algorithm choice with data collection strategy and feedback mechanism, obscuring the fact that these are separable design decisions.

What the paper reveals. The four-axis decomposition — model, data, feedback, algorithm — makes explicit that:

Data collection strategy (on-policy vs. off-policy) is a choice about where training samples come from, independent of what algorithm processes them. A method using on-policy data (sampled from the current policy with real-time feedback) can deploy any algorithm category: PPO (point-wise), online DPO (pair-wise contrast), or GRPO (list-wise contrast). Conversely, a pre-collected offline dataset can feed point-wise methods (RFT, KTO on stored labels), pair-wise methods (standard DPO), or list-wise methods (RRHF on pre-ranked outputs).
Feedback mechanism (direct rules, reward models, LLM-as-a-judge) is a choice about how the preference signal r is generated, independent of both the data source and the algorithm. Rule-based feedback (unit tests, proof verifiers) can drive PPO (as in code generation RL), DPO (by constructing preference pairs from test outcomes), or list-wise ranking (by ordering responses by test pass rates). Reward models can provide scalar rewards for point-wise methods, preference differences for pair-wise methods, or global rankings for list-wise methods — the RM architecture is largely orthogonal to the optimization algorithm.
Algorithm category (point-wise, pair-wise, list-wise) is a choice about how many samples inform each gradient coefficient computation, independent of how those samples were collected or how they were scored.

Why this is a fundamental shift, not incremental taxonomy. Prior surveys could have listed data collection methods, feedback types, and algorithms as separate sections (many did). The innovation here is the claim that these are logically independent axes — that any point in the {on-policy, off-policy} × {direct, RM, judge} × {point-wise, pair-wise, list-wise} space is a coherent and potentially useful alignment method, even if the literature has not explored that combination yet. This claim has direct scientific consequences: it identifies underexplored regions of the design space that the prior taxonomy made invisible.

For example, the combination of on-policy data + rule-based feedback + list-wise contrast is natural under the four-axis framework (sample responses from the current policy, score them with unit tests, compute a list-wise contrastive loss from the ranked outputs) but appears nowhere in the recent literature. Under the old RL-vs-SFT taxonomy, this combination would have been difficult to even describe: is it "RL" (because it uses on-policy data and rule-based rewards) or "SFT" (because it uses a contrastive loss)? The framework resolves this ambiguity by refusing to force methods into categories that conflate independent choices.

Evidence and significance. The paper's evidence for this decoupling is primarily demonstration by example: it walks through PPO and DPO in both online and offline settings (Figure 3), showing that the algorithmic core is unchanged while only the data collection and feedback timing differ. The In-Context DPO example (Figure 3) is particularly illustrative — it shows DPO deployed with online feedback from in-context learning demonstrations rather than pre-stored preference labels, a configuration that the old taxonomy would struggle to classify but that the four-axis framework handles naturally as {online, LLM-as-a-judge, pair-wise contrast}.

The practical upshot is that the paper enables a new kind of research question: comparative ablation of design axes. Instead of asking "does DPO beat PPO on benchmark X?", which confounds algorithm, data, and feedback choices, researchers can ask "does switching from off-policy to on-policy data improve pair-wise contrast methods?" or "does rule-based feedback reduce over-optimization compared to reward models across all algorithm categories?" This systematic decomposition is what distinguishes a genuinely useful taxonomy from a mere list of categories.

Innovation 3: The Gradient Coefficient as the Universal "Difference-Maker"

The paper's third distinctive contribution is more subtle and more technically grounded than the first two: it identifies the gradient coefficient δ as the single scalar that captures all algorithmic variation in preference learning, and uses this observation to define the algorithm categories (point-wise, pair-wise, list-wise) in a principled way that directly reflects the computational and data requirements of each category.

Prior state of the field. Before this survey, algorithm categorizations were based on surface features of the loss function or training procedure: "PPO uses a KL penalty and advantage estimation," "DPO uses a contrastive loss with a reference model," "KTO doesn't need paired data." These descriptions are accurate but provide no principled basis for comparing across categories — how does one evaluate the tradeoff between "uses a reference model" and "uses a critic network"? The absence of a shared vocabulary for comparing algorithm mechanisms meant that researchers proposing new variants often focused on loss-function engineering without articulating what fundamental lever they were adjusting.

What the paper reveals. The unified gradient formula shows that every gradient-based alignment algorithm performs the same atomic operation: for each token t in each output o, multiply the policy gradient ∇θ log πθ(ot) by a coefficient δ, and average over the data distribution. The coefficient δ is the sole locus of algorithmic variation — it encodes everything the algorithm "thinks" about the quality of token ot at position t, incorporating the reward signal r, any baselines or comparisons, and any normalization or regularization.

This observation has three important consequences that elevate it beyond a mathematical curiosity:

1. It provides a principled basis for algorithm categorization. The number of samples needed to compute δ for a single update defines the algorithm category: point-wise methods need one sample (δ depends only on the reward of that sample), pair-wise methods need two (δ depends on the contrast between chosen and rejected responses), list-wise methods need N > 2 (δ depends on the relative position in a ranked list). This categorization directly reflects data requirements (how many scored samples must be collected per prompt), computational cost (how many forward passes through the model or reward function are needed per update), and optimization granularity (how fine-grained the relative quality signal is). In contrast, the prior RL-vs-SFT taxonomy conflated these considerations with unrelated architectural choices (critic networks, reference models).

2. It reveals that "online vs. offline" is a property of when r is computed, not of the algorithm. Under this formulation, "online" simply means that r is freshly computed from the environment for each batch of on-policy samples; "offline" means r is stored in a pre-collected dataset. The gradient computation after r is obtained is identical in both cases. This clarifies a persistent confusion in the literature, where "online RL" and "offline RL" were sometimes treated as different algorithm classes rather than different data collection regimes for the same algorithms.

3. It explains why certain combinations of algorithm and feedback are unstable or inefficient. When δ is computed from a noisy reward signal (e.g., a single reward model score with high variance), point-wise methods with no baseline subtraction (like vanilla REINFORCE) suffer from high gradient variance. Pair-wise methods inherently provide a baseline (the rejected response's score) that reduces variance, and list-wise methods provide an even richer comparison set — but at the cost of requiring more samples per update. This variance-reduction property is implicit in the loss functions but becomes explicit when viewed through the δ lens: the comparison between chosen and rejected responses in DPO's δ term subtracts a baseline, similar to how PPO's advantage estimate subtracts the value function, but without requiring a separate critic network.

Why this is a conceptual contribution rather than a technical one. The gradient coefficient formulation is not new mathematics — Shao et al. (2024) introduced the specific form, and the idea that RL and SFT methods share gradient structure is implicit in DPO's derivation (Rafailov et al., 2023). The paper's contribution is to make δ the central organizing concept of a taxonomy, and to show that doing so yields a cleaner, more predictive categorization than the alternatives. This is analogous to how the transformer architecture's attention formulation (Queries, Keys, Values) became a shared vocabulary for describing innovations — not because the math was novel, but because the abstraction revealed which design choices were independent and which were coupled.

Evidence. The paper supports this contribution primarily through the running examples in Figures 3 and 4, which walk through PPO (online and offline), DPO (online and offline), In-Context DPO, ReMax, RFT, and RAFT — showing for each how the data feedback mechanism, algorithm, and resulting δ computation map onto the unified framework. Table 1 further supports the categorization by showing that diverse loss functions (DPO, IPO, f-DPO, EXO, SimPO, ORPO, RRHF, PRO) differ only in how they compute the pair-wise or list-wise contrast that feeds into δ. The key evidence is coverage: if the formulation failed to accommodate important classes of methods, it would lose its organizing power, but the paper demonstrates that it spans from simple RFT (δ = 1 for correct samples, 0 otherwise) to complex methods like GRPO (δ is the normalized reward within a group).

Innovation 4: Explicitly Identifying Underexplored Regions of the Design Space as a Research Contribution

The paper's final distinctive contribution is unusual for a survey: rather than merely cataloging what exists, it uses its framework to identify specific, promising combinations that the literature has not yet explored, and frames these gaps as actionable research directions. This transforms the survey from a passive reference into an active research agenda-setting document.

Prior state of the field. Most surveys in this area (Shen et al., 2023; Wang et al., 2024a) conclude with broad future directions — "better reward models," "more diverse preference data," "more comprehensive evaluation" — that are sensible but generic. They do not derive their research agendas from their taxonomic structure, and consequently the future directions feel disconnected from the preceding analysis. A reader of these surveys might agree that "better reward models" is important but have no clearer idea of which specific architectural innovation to try next.

What the paper reveals. Because the four-axis framework makes explicit the logical independence of data, feedback, and algorithm choices, it naturally generates hypotheses about unexplored combinations. The paper's future directions section identifies several specific gaps that emerge directly from the taxonomy:

Rule-based feedback applied beyond code and math. The framework makes clear that direct feedback (unit tests, proof verifiers, QE models) is the cheapest and most reliable feedback type but is currently limited to domains with objective correctness criteria. The taxonomy reveals this as a coverage gap — the {rule-based, any algorithm, any data} region of the design space is densely populated for code generation and math but nearly empty for dialogue, summarization, and creative writing. This suggests a concrete research direction: can we design rule-based feedback mechanisms for open-ended tasks? (The paper does not propose a solution but identifies the gap precisely.)
Online + pair-wise contrast + LLM-as-a-judge. The DPO literature almost entirely uses offline pre-collected preference data, but the framework shows that the DPO loss makes no assumption about when the preference labels were generated. Online DPO — where the policy generates responses, an LLM-as-a-judge evaluates them in real-time, and the pair-wise contrastive loss is computed immediately — is a coherent combination that very recent work (Guo et al., 2024, cited in the survey) has begun to explore but which remains underexplored relative to its potential. The survey identifies this not as a proven winner but as a logical combination that the taxonomy reveals as natural and that the literature has only begun to investigate.
Combining feedback types within a single training run. The framework treats feedback as a single choice, but the decoupled structure suggests the possibility of hybrid feedback — e.g., using rule-based feedback where available (math, code) and LLM-as-a-judge feedback elsewhere (open-ended generation), with the same algorithm operating on both signal types. No existing method implements this, but the taxonomy makes its feasibility obvious.
Process reward models with pair-wise contrast algorithms. The paper surveys PRMs (process reward models that provide step-level feedback) as a feedback innovation and DPO-like losses as an algorithm innovation, but notes that these literatures have developed largely independently. The taxonomy reveals no barrier to combining them — a pair-wise contrast loss where δ is computed from per-step PRM scores rather than outcome-level preference labels — yet this combination appears nowhere in the surveyed literature. The gap is particularly striking for mathematical reasoning, where process supervision has proven effective but has been almost exclusively paired with point-wise RL methods (PPO) or best-of-N selection.

Why this is a genuine contribution. The scientific value of a taxonomy is measured not only by how well it organizes existing knowledge, but by how effectively it generates new questions. The four-axis framework passes this test: it makes specific, falsifiable predictions about which combinations should work (because all individual axes have been validated in isolation), and it identifies gaps that are non-obvious under the old RL-vs-SFT taxonomy. Prior surveys, which conflated axes, could not have identified the underexploration of "online + pair-wise + rule-based" because they encoded "pair-wise = offline" as an implicit assumption. The paper's contribution here is not to prove that these combinations work — it explicitly leaves that to future work — but to provide a structured search strategy for the design space that is more principled than random trial-and-error.

Evidence. The survey's future directions section (Section 8) explicitly lists these combinations as research opportunities, and the paper's organization — devoting separate sections to data (Section 4), feedback (Section 5), and algorithms (Section 6) before discussing possible synergies — ensures that readers understand each axis before being shown how they compose. The Figure 3 running examples reinforce this by showing PPO and DPO in both online and offline configurations, demonstrating that the decoupling is not just asserted but illustrated. The strength of this contribution is not empirical (no experiments are run) but architectural: the framework itself is the evidence that these combinations exist, because each axis has been validated independently and nothing in the gradient coefficient formulation prevents their joint deployment.

Innovation 5: Reframing "Alignment" as a Composition of Four Independent Supply Chains

The paper's final conceptual contribution is less explicitly argued but emerges from the overall structure: it reframes alignment not as a monolithic optimization problem (the RLHF perspective) or a simple supervised learning task (the DPO perspective), but as the composition of four independent "supply chains" — data, feedback, algorithm, and model — each with its own quality constraints, cost profiles, and failure modes. This reframing has implications for how alignment research should be organized, funded, and evaluated.

Prior state of the field. In the RLHF paradigm, alignment is a pipeline: collect preference data → train reward model → run PPO. The pipeline is treated as a single integrated system, and improvements are typically evaluated end-to-end — does the final model produce better outputs? This end-to-end evaluation makes it difficult to attribute improvements to specific components, and it encourages researchers to propose new pipelines rather than improve individual components. The DPO literature partially addressed this by eliminating the reward model, but it retained the end-to-end evaluation paradigm — DPO papers typically compare final model quality against PPO baselines, not the quality of individual intermediate steps.

What the paper reveals. By separating alignment into four independently variable axes, the paper implicitly argues that component-level evaluation and improvement is more productive than end-to-end pipeline comparison. The framework makes it natural to ask: "Given a fixed algorithm (say, DPO), how does feedback quality (human labels vs. LLM-as-a-judge vs. reward model) affect final performance?" or "Given fixed data and feedback, how does algorithm choice (point-wise vs. pair-wise vs. list-wise) affect sample efficiency?" These questions are difficult to formulate under the old taxonomy because the components were not cleanly separated — PPO came bundled with a reward model, DPO came bundled with offline data, and comparing them meant changing multiple variables simultaneously.

The "supply chain" analogy is useful: in manufacturing, one does not evaluate a factory by the quality of its final product alone — one evaluates the quality control of each component supplier, because a defect in any link propagates downstream and masks the quality of the other links. Similarly, an alignment method that fails might have excellent feedback but poor data, or excellent data and feedback but an inappropriate algorithm, and end-to-end evaluation cannot distinguish these cases. The four-axis framework enables component-level ablation, which is the first step toward a systematic understanding of failure modes and improvement strategies.

Why this is a significant shift. This reframing changes what counts as a "contribution" in alignment research. Under the end-to-end paradigm, only new pipelines or new loss functions that improve final benchmark scores are publishable. Under the supply-chain paradigm, improvements to individual components — a better method for generating diverse on-policy data, a more reliable feedback mechanism for a specific domain, a variance-reduction technique for gradient coefficient estimation — become independently valuable contributions, because they can be evaluated in isolation and then composed with other components. This has the potential to accelerate progress by lowering the barrier to entry (researchers can specialize in one component rather than building entire pipelines) and by enabling modular innovation (a new feedback mechanism can immediately improve dozens of existing algorithms without any change to those algorithms' code).

Evidence. The paper's own structure embodies this reframing: Sections 4, 5, and 6 are devoted to data, feedback, and algorithm, respectively, each surveying dozens of methods within its axis, and the connections between axes are discussed only after each axis has been treated independently. This is a rhetorical choice that reflects a substantive claim — that these axes are sufficiently decoupled to warrant independent survey sections, and that the field would benefit from treating them as independent research sub-areas rather than inseparable parts of monolithic pipelines.

The paper does not provide empirical evidence for the claim that component-level research accelerates progress (this would require a longitudinal study of the field that is beyond the scope of a survey). However, the rapid proliferation of DPO variants since 2023 — each modifying only the loss function while keeping data and feedback fixed — provides circumstantial evidence that the field is already moving toward component-level research, and the survey provides the conceptual vocabulary to make this trend explicit and intentional rather than implicit and accidental.

5. Experimental Analysis

Evaluation Methodology

Important prefatory note: Unlike the example paper analyzed in Section 3 (which reports original empirical results with specific figures, tables, and quantitative comparisons), this paper is a survey and taxonomy. It does not conduct new experiments, does not report benchmark results, and does not compare methods against baselines under controlled conditions. The paper's contribution is conceptual and organizational — establishing a four-axis decomposition framework and using it to classify over 100 existing methods. Any "experimental analysis" in the traditional sense (training runs, ablation studies, metric-driven comparisons) does not exist in this paper.

However, the paper does engage with empirical claims from the literature it surveys, and it does make evaluative statements about method performance that implicitly rely on experimental evidence reported elsewhere. This section therefore analyzes the paper's relationship to experimental evidence — how it uses (or does not use) quantitative results to support its taxonomic framework, and what kind of empirical validation would be needed to substantiate the framework's claims.

Because the paper performs no experiments, the standard sections (Main Quantitative Results, Ablation Studies, Critical Assessment) must be adapted to reflect what the survey does provide: evidence from the literature, running examples, and comparative analysis. Where the paper makes evaluative claims about methods, I will examine whether those claims are supported by the cited studies. Where the paper's taxonomic framework makes testable predictions, I will identify what experiments would confirm or disconfirm its key organizational claims.

What the Paper Does Provide (in lieu of experiments)

Running examples of representative algorithms (Figures 3 and 4). The paper includes detailed, diagrammed walkthroughs of eight algorithm configurations — PPO (online), PPO (offline), DPO (online), DPO (offline), In-Context DPO, ReMax, RFT, and RAFT — each mapped onto a specific task (code generation, summarization, dialogue, mathematical reasoning). These examples are pedagogical tools rather than experimental comparisons: they show how each method operates within the four-axis framework but do not report quantitative performance of one method vs. another. For instance, Figure 3's DPO (Online) for Summarization panel shows the data flow (prompt → candidate generation → real-time GPT-4 or RM evaluation → pair-wise contrast → update) but contains no accuracy numbers, win rates, or loss curves.

Taxonomic coverage analysis (Figures 2, Table 1). Figure 2 is a taxonomy tree that maps methods onto the data/feedback/algorithm axes. Table 1 shows loss functions for DPO, IPO, f-DPO, EXO, SimPO, ORPO, RRHF, and PRO. The implicit empirical claim in both is coverage: the framework accommodates all listed methods without forcing exceptions. This is a form of validation (does the taxonomy fit the evidence of what methods actually exist?) but it is qualitative rather than quantitative.

Literature-derived comparative statements. The paper occasionally makes comparative claims about method performance, always referencing prior work rather than conducting new comparisons. These claims must be examined for whether the cited studies actually support them, and whether the survey accurately represents the strength and limitations of the evidence.

Main Comparative Claims from the Literature

The paper makes several evaluative assertions about method performance that implicitly require experimental support. Below, I examine each significant claim, identify its source in the surveyed literature, and assess what the cited experiments actually demonstrate.

Claim 1: DPO can achieve comparable or superior performance to PPO while being computationally simpler

Source: Section 6.2, referencing Rafailov et al. (2023).

What the cited paper demonstrated: DPO was evaluated on controlled sentiment generation, summarization, and single-turn dialogue tasks against PPO baselines. On TL;DR summarization, DPO achieved higher win rates against the reference model than PPO at equivalent KL-divergence budgets. On Anthropic's helpfulness/harmlessness benchmarks, DPO matched or exceeded PPO performance.

Limitations of this evidence for the survey's framing: The DPO paper's comparison was between a specific PPO configuration and a specific DPO configuration on specific tasks using specific reward models. The survey's framework claims that DPO and PPO are points in a continuous design space, differing only in how δ is computed. Whether this equivalence holds in practice depends on implementation details (hyperparameter tuning, data quality, reward model calibration) that the original experiments controlled but did not systematically vary. The survey's assertion that PPO and DPO are fundamentally similar algorithms is a mathematical claim supported by Equation 1; the assertion that they achieve comparable performance is an empirical claim supported by the original DPO paper's experiments on a limited set of benchmarks. The survey does not systematically review whether this comparability holds across the dozens of tasks and model families where both methods have since been tested.

Claim 2: Online methods generally outperform offline methods due to reduced distribution shift

Source: Section 4 (on-policy vs. off-policy discussion), referencing Guo et al. (2024) and other online DPO variants.

What the cited papers demonstrate: Guo et al. (2024) shows that online DPO — where the policy generates responses and receives real-time AI feedback — improves over standard offline DPO on several benchmarks. The mechanism is reduced distribution shift: the policy is trained on its own output distribution rather than on a fixed dataset that may not represent its current behavior.

Limitations: The paper presents "online vs. offline" as a clean data collection choice orthogonal to algorithm selection, but the empirical evidence for online superiority comes from a small number of recent studies on specific tasks. Whether the online advantage generalizes across all algorithm categories (point-wise, pair-wise, list-wise), all feedback types (rule-based, RM, LLM-as-a-judge), and all domains (code, math, dialogue) is an open empirical question that the survey's framework makes testable but does not test. The framework's value here is that it enables formulating this question — "does switching from off-policy to on-policy data improve pair-wise contrast methods?" — but it does not answer it.

Claim 3: SimPO (removing the reference model) outperforms DPO on certain benchmarks

Source: Section 6.2, referencing Meng, Xia, and Chen (2024).

What the cited paper demonstrates: SimPO eliminates the reference model from the DPO loss and adds a length-normalized reward margin. On AlpacaEval 2 and Arena-Hard benchmarks, SimPO achieves higher win rates than DPO, particularly when combined with a length penalty.

Limitations: The SimPO paper's experiments were conducted with Llama-3 and Mistral model families on instruction-following benchmarks. Whether the reference model can be safely removed across all model scales, task types, and training data regimes is unknown. The survey presents SimPO as a "concise DPO-like algorithm" without caveats about its limited empirical validation scope.

Claim 4: KTO (point-wise method requiring only binary preference labels) matches DPO performance with simpler data requirements

Source: Section 6.1, referencing Ethayarajh et al. (2024).

What the cited paper demonstrates: KTO uses Kahneman-Tversky prospect theory to define a loss function on individual samples (each labeled as "desirable" or "undesirable") rather than on pairs. On several benchmarks, KTO achieves comparable or better performance than DPO, and it maintains performance under extreme data imbalance where DPO degrades.

Limitations: The same generalizability caveats apply — KTO was tested on a specific set of models and tasks, and the survey cites the original paper's results without qualifying the scope of empirical evidence.

What the Paper's Framework Claims vs. What Is Empirically Validated

The survey makes several structural claims that go beyond summarizing individual paper results. These claims are the core of the survey's contribution, but they require a different kind of validation than standard benchmark comparisons:

Claim: The four axes (data, feedback, algorithm, model) are genuinely independent design choices. This is a mathematical/combinatorial claim about what combinations are possible, not an empirical claim about what combinations are effective. The paper demonstrates possibility through existence — for each combination it claims is feasible, it can point to at least one method in the literature that implements it. For example, the combination {offline, reward model, point-wise} is implemented by RAFT; {online, direct feedback, pair-wise} is implemented by online DPO with rule-based preference labeling. However, the paper does not systematically enumerate all 2×3×4 = 24 combinations of {on-policy/off-policy} × {direct/RM/judge} × {point-wise/pair-wise/list-wise/training-free} and verify that each appears in the literature. Many combinations are indeed missing or have only very recent instantiations (e.g., {online, LLM-as-a-judge, list-wise} appears in very recent work like SPIN but was not widely explored at the time of the survey's writing). The survey's claim of independence is thus a claim about the design space's structure rather than about demonstrated coverage of that space.

Claim: All gradient-based methods optimize the same objective (Equation 1) and differ only in δ. This is a mathematical claim derived from Shao et al. (2024). The paper provides algebraic derivations showing that PPO, DPO, RFT, and GRPO all reduce to this form. The empirical validation question is whether practice matches theory — do actual training runs of these different algorithms produce models that converge to similar solutions when δ is computed from comparable feedback? The survey does not address this empirical convergence question, nor would we expect it to as a taxonomy paper. However, a reader should understand that Equation 1 is a theoretical unification that may not hold under the approximations used in practice (e.g., PPO's clipped surrogate objective, DPO's implicit reward that may not converge to the true Bradley-Terry reward without sufficient data).

Claim: Algorithm performance varies significantly across different models and task settings, and no single algorithm dominates. This is asserted in the introduction and future directions sections but is never systematically demonstrated. The paper does not provide a cross-method comparison table showing that Method A beats Method B on Task 1 but loses on Task 2, while Method C is best on Task 3 at low budget but worse at high budget. Such a table would empirically validate the paper's core motivation — that a framework for reasoning about design tradeoffs is necessary because there is no universal winner. Instead, the paper relies on the reader's general awareness that this claim is true (which, to be fair, is widely acknowledged in the alignment community). This is a significant gap: a survey that argues for taxonomic pluralism should ideally demonstrate that empirical pluralism exists.

Coverage Analysis: What Methods Are Included and What Is Missing

The survey's main "empirical" contribution is the scope of its literature coverage. The taxonomy encompasses:

~130+ citations spanning from foundational RLHF papers (Ouyang et al., 2022) to cutting-edge 2024 work (generative verifiers, Meta-Rewarding, SimPO).
All major algorithm families: PPO and its variants (ReMax, GRPO), DPO and its variants (IPO, f-DPO, SimPO, ORPO, sDPO, DPO-positive, EXO, CPO, MAPO), list-wise methods (RRHF, PRO, LiPO, LIRE), training-free methods (URIAL, BPO, Aligner, RAIN, DeAL, ICDPO).
All major feedback types: direct rules (RFT for math, unit tests for code, QE for translation), trained reward models (Bradley-Terry, binary classifier, PRMs, reward model ensembles), pair-wise scoring models (PairRM, LLM-Blender), LLM-as-a-judge (self-rewarding, Meta-Rewarding, CriticGPT, generative verifiers).
Both data collection paradigms: on-policy (Top-K/Nucleus sampling, beam search, MCTS) and off-policy (human-annotated datasets: WebGPT, HH-RLHF, SHP; LLM-annotated datasets: RLAIF, ULTRAFEEDBACK, UltraChat).

Notable omissions: The paper does not deeply cover constitutional AI (Bai et al., 2022b), which uses model-generated critiques and revisions rather than scalar preference labels — an approach that blurs the feedback/algorithm boundary in ways the taxonomy might struggle to accommodate. The paper also does not address multi-turn preference learning (where preferences accumulate over conversation turns rather than per-response), which introduces temporal credit assignment challenges not captured by Equation 1's per-token gradient formulation. Multi-modal preference alignment is explicitly scoped out (Section 1), which is a defensible scope constraint but limits the framework's applicability to the rapidly growing vision-language model alignment literature.

The Framework's Predictive Power: What Experiments Would Test It

A good taxonomy should generate testable predictions. The paper's framework makes several predictions that, while not tested in the survey itself, could be empirically evaluated:

Prediction 1: On-policy data should improve any algorithm category, not just point-wise RL. If the framework is correct that data collection strategy is orthogonal to algorithm choice, then the benefits of on-policy data demonstrated for PPO (reduced distribution shift) should also appear when the same on-policy data is used with pair-wise contrast losses (online DPO) or list-wise contrast losses (online GRPO). This prediction is partially confirmed by online DPO results (Guo et al., 2024) but has not been systematically tested across all algorithm categories on the same tasks with controlled feedback.

Prediction 2: The "best" algorithm for a given task should depend on feedback quality, not just task type. If noisy feedback increases gradient variance, then methods with implicit baselines (pair-wise contrast, which subtracts the rejected response's score) should outperform point-wise methods when feedback is noisy, even if point-wise methods are simpler. Conversely, when feedback is clean and deterministic (rule-based), the simpler point-wise methods should suffice. This prediction is intuitively plausible but has not been rigorously tested: the literature tends to treat algorithm choice and feedback choice as separate decisions rather than interacting ones.

Prediction 3: List-wise methods should be most beneficial when the ranking within a sample group carries genuine preference information beyond "best vs. rest." If the list-wise contrast in PRO or GRPO is simply weighting all non-best responses equally (as in pair-wise methods repeated across multiple pairs), the additional complexity shouldn't help. If there is genuine graded preference information (e.g., response A is clearly better than B, which is slightly better than C, which is clearly better than D), list-wise methods should exploit this signal and outperform methods that only use binary comparisons. This prediction could be tested by constructing datasets with graded human preferences and comparing list-wise vs. pair-wise methods on those datasets.

Prediction 4: The gradient coefficient formulation implies that methods with similar δ computations should exhibit similar optimization dynamics. For instance, DPO's δ and PPO's δ should scale similarly with the quality gap between chosen and rejected responses (or between a response and the value baseline), leading to similar convergence rates, sensitivity to learning rate, and vulnerability to reward hacking. This is a strong prediction that would be difficult to test without careful controlled experiments, but it is arguably the deepest implication of the unified framework — if PPO and DPO really are "the same algorithm with different δ," then all empirical differences in their performance should be attributable to differences in the quality of the δ estimates they compute, not to fundamental differences in optimization behavior.

Critical Assessment

Does the framework actually unify, or does it merely relabel?

The paper's central claim is that the unified gradient formulation reveals a shared structure underlying RL-based and SFT-based methods. This is mathematically true under the formulation in Equation 1: if you write down the gradient of PPO's clipped surrogate objective and the gradient of DPO's contrastive loss, both reduce to a sum over tokens of (coefficient × policy gradient). However, mathematical unification does not automatically imply practical unification. The differences between PPO and DPO that matter in practice — PPO requires a critic network and generalized advantage estimation; DPO requires a reference model and careful tuning of β; PPO updates are typically small (clipped) and require multiple epochs on the same batch; DPO updates can be large and are typically single-pass — are all implementation details that the gradient coefficient formulation abstracts away. These details may be consequences of the different δ computation strategies (PPO's clipping is necessary because advantage estimates are high-variance; DPO's β tuning is necessary because the implicit reward can diverge), but they are not captured by the formulation itself. A reader who understands Equation 1 does not automatically understand why PPO training is notoriously unstable or why DPO is sensitive to the quality of the chosen-rejected pairs in the dataset. The framework explains what is being optimized but not how optimization proceeds in practice, and for practitioners, the latter is often more important.

The survey does not empirically validate its own motivation.

The paper argues that the RL-vs-SFT taxonomy is harmful because it obscures connections between methods and prevents systematic comparison. But the survey never demonstrates this harm empirically — it does not show that researchers working under the old taxonomy made suboptimal design choices, or that the new taxonomy enables better decisions. The argument is entirely conceptual: "if we reorganize the literature this way, we can see connections that were hidden before." This is a valid contribution for a survey paper, but it means the paper's impact depends on whether the field adopts the framework and finds it productive, not on whether the framework is "correct" in any empirical sense.

The strongest evidence for the framework's value would be a case study: take a specific alignment challenge (e.g., improving mathematical reasoning in a 7B model), use the framework to identify an underexplored combination of data/feedback/algorithm (e.g., on-policy MCTS data + PRM feedback + list-wise contrast loss), implement it, and show it outperforms standard approaches. The paper does not do this, and its future directions section explicitly calls for such work. Until such studies exist, the framework's utility is promising but unproven.

The "point-wise / pair-wise / list-wise" categorization is clean but may not capture the most important algorithmic differences.

The number of samples per update is a coherent organizing principle, but it may not be the most predictive variable for understanding method behavior. For instance, within the "pair-wise contrast" category, DPO and SimPO and ORPO differ in whether they use a reference model, whether they include an SFT loss, and how they handle length normalization — and these differences may matter more for practical performance than the fact that they all use two samples per update. The taxonomy places DPO, SimPO, and ORPO in the same bucket, but a practitioner choosing between them needs to understand their reference model and length normalization differences, not just their sample-per-update count. The framework's categorization is principled but may not align with the distinctions that drive practical decision-making.

Missing: a systematic analysis of failure modes.

The survey catalogs methods and their mechanisms but does not systematically analyze when they fail. A more actionable taxonomy would map methods not just to their computational mechanisms (point-wise vs. pair-wise) but to their failure modes: reward hacking (over-optimization of the reward model), distribution shift (off-policy data mismatch), length exploitation (models learning that longer responses receive higher scores), sycophancy (agreeing with user errors), and benchmark contamination. These failure modes cut across the four-axis categories — reward hacking affects PPO and DPO and list-wise methods whenever a learned reward model is used — and a framework organized around failure modes might be more directly useful for practitioners deciding which risks to prioritize. The survey's future directions section mentions "reliable feedback and scalable oversight" (Section 8) as a challenge but does not integrate failure mode analysis into the taxonomy itself.

The paper's strength is also its weakness: it is a map, not a guide.

The four-axis framework is excellent for understanding the alignment design space — it tells you what choices exist and how they relate. It is less useful for navigating that space — it does not tell you which choices are likely to work well for a given task, budget, and safety requirement. The paper explicitly declines to pick winners ("We leave the exploration of which variants perform better in specific contexts as our future work"), which is intellectually honest but limits the survey's immediate practical value. A practitioner facing a concrete alignment problem will learn from this survey that many combinations are possible but will not learn which combination to try first. Future work that uses the framework as the basis for an empirical meta-analysis (systematically comparing methods along each axis on standardized benchmarks) would complement this survey by adding the navigational layer that it currently lacks.

6. Limitations and Trade-offs

6.1 The Framework Is a Conceptual Reorganization, Not an Empirically Validated Theory

The assumption or constraint. The paper's central claim is that all gradient-based preference learning methods share a unified mathematical structure (Equation 1) and differ only in how the gradient coefficient δ is computed from the available data, feedback, and algorithm choices. This is presented as a unifying insight that reveals hidden connections between apparently disparate methods. However, the paper never empirically validates that this unification holds in practice — that is, that methods with similar δ computation exhibit similar optimization dynamics, converge to comparable solutions given equivalent feedback, or share failure modes. The authors acknowledge this indirectly:

"the core objectives of these alignment algorithms are essentially similar, their performance can vary significantly across different application scenarios. We leave the exploration of which variants perform better in specific contexts as our future work."

The consequence. The mathematical unification in Equation 1 abstracts away implementation details that may dominate practical behavior. For instance, PPO's clipped surrogate objective, its reliance on generalized advantage estimation with a separate critic network, and its typical multi-epoch training on the same batch produce optimization dynamics that differ substantially from DPO's single-pass contrastive update with a reference model, even if both reduce to (coefficient × policy gradient) in the limit. The framework explains what is being optimized but not how optimization proceeds, and for practitioners, the latter often determines success or failure. A method's training stability, sensitivity to hyperparameters, sample efficiency, and vulnerability to reward hacking may be driven more by these implementation details than by the number of samples used to compute δ. By abstracting these details away, the framework risks giving the false impression that switching between, say, PPO and DPO is a straightforward substitution of δ-computation strategies, when in practice it involves fundamentally different infrastructure (critic networks, reference models, clipping thresholds, KL penalties) that interact with δ in complex ways.

What evidence exists in the paper. The paper provides no empirical evidence for the practical validity of the unification. No experiments are conducted, no training runs are compared, and no convergence analyses are presented. The running examples in Figures 3 and 4 are pedagogical walkthroughs that illustrate the data flow of each method within the four-axis framework, but they contain no quantitative results — no accuracy numbers, loss curves, or ablation comparisons that would demonstrate equivalence or divergence. The framework's validity is asserted on mathematical grounds (by reference to Shao et al., 2024) and demonstrated through taxonomic coverage (over 100 methods can be classified within it), but these are arguments for the framework's completeness, not its predictive power.

Mitigation status. The paper does not attempt to address this limitation. It explicitly defers empirical validation to future work. This is a defensible choice for a survey paper — the contribution is the framework itself, not experimental proof of its consequences — but it means the framework should be treated as a hypothesis about the structure of the design space rather than an established theory. A practitioner reading this survey learns how methods relate conceptually but receives no guidance on which relationships drive practical performance differences.

The assumption or constraint. The four-axis decomposition is deliberately descriptive rather than prescriptive. The paper refuses to identify which combinations of data, feedback, and algorithm are likely to work best for specific tasks, budgets, or safety requirements:

"We leave the exploration of which variants perform better in specific contexts as our future work."

The taxonomy tells you what choices exist — on-policy vs. off-policy data, direct vs. model-based feedback, point-wise vs. pair-wise vs. list-wise algorithms — but provides no guidance on which choice to make for a given problem. There is no decision tree, no flow chart, no set of heuristics, and no empirical comparison table ranking methods by performance on standard benchmarks.

The consequence. This severely limits the survey's practical utility for practitioners. A researcher or engineer facing a concrete alignment problem — say, improving the mathematical reasoning of a 7B model with limited GPU budget and no access to human annotators — learns from this survey that many options are available: they could use rule-based feedback (correctness checking) with on-policy sampling and a point-wise method like RFT, or off-policy data with DPO, or MCTS-generated data with a list-wise contrast like GRPO. But the survey provides no basis for choosing among these options. The practitioner must still read dozens of individual method papers to determine which approach is appropriate for their constraints, which is precisely the fragmentation problem the survey aims to address.

The absence of prescriptive guidance also limits the framework's ability to prevent bad choices. The taxonomy treats all combinations of the four axes as logically possible, but many combinations are likely dominated — for instance, using an expensive trained reward model for feedback when simple rule-based checking is available and objective, or deploying a list-wise method when only point-wise labels are affordable. By presenting all choices as equally valid design decisions, the framework risks encouraging exploration of combinations that are a priori unlikely to be cost-effective.

What evidence exists in the paper. The paper's own structure reveals this limitation. Section 4 (Preference Data), Section 5 (Feedbacks), and Section 6 (Algorithms) each survey methods within their respective axis independently, without cross-referencing which combinations have been empirically validated or comparing their costs. Section 8 (Future Directions) identifies specific underexplored combinations but does not prioritize them by expected impact or difficulty. The running examples in Figures 3 and 4 show eight specific configurations but do not compare them head-to-head. The absence of a "method selection guide" or "practitioner recommendations" section is conspicuous — it reflects a deliberate choice to remain descriptive, but it leaves a gap that a practitioner-focused supplement would need to fill.

Mitigation status. Not addressed. The authors explicitly identify this as future work ("Finding the most effective variant from a theoretical perspective is also a very practical topic, which we leave to our future work"). The paper's contribution is to provide the vocabulary and conceptual scaffolding that makes such comparative analysis possible — by separating the axes, it enables questions like "does on-policy data improve pair-wise methods?" that the old RL-vs-SFT taxonomy obscured. But the paper stops at enabling the questions without attempting to answer them.

6.3 The Framework Does Not Account for Computational Cost, Training Stability, or Implementation Complexity

The assumption or constraint. The four-axis decomposition treats data, feedback, and algorithm as logically independent design choices, implying that any combination is feasible. However, the framework makes no provision for the practical costs and risks that constrain these choices in real deployments:

Computational cost: Training a reward model requires substantial GPU hours and curated preference data. Running PPO requires maintaining four models simultaneously (policy, reference, reward, critic). Online data collection requires generating fresh samples from the current policy at each training step, which can dominate total compute. LLM-as-a-Judge feedback (e.g., GPT-4 evaluation) incurs API costs that scale linearly with the number of training samples. The framework treats these as equally valid "feedback" choices without acknowledging their vastly different resource profiles.
Training stability: PPO is notoriously sensitive to hyperparameters (KL penalty coefficient, clipping threshold, learning rate, number of epochs per batch) and can easily diverge or collapse to degenerate outputs. DPO is more stable but can overfit the preference dataset, reducing output diversity and generalization. The framework's unified gradient formulation (Equation 1) abstracts away these stability concerns — it shows what gradient is computed but not whether that gradient reliably improves the policy or causes catastrophic forgetting.
Implementation complexity: Deploying PPO requires engineering an entire RL infrastructure (environment interaction loop, advantage estimation, value function training). Deploying DPO requires only a supervised fine-tuning pipeline. Training-free methods require no gradient updates at all but may require complex decoding-time interventions. The framework places these in separate algorithm categories but does not rank them by engineering difficulty.

The consequence. By abstracting away implementation cost, the framework creates an implicit bias toward combinatorial exploration — it encourages thinking about "what combinations are possible" rather than "what combinations are practical." A researcher inspired by the taxonomy might attempt to combine MCTS-based on-policy data with a Bradley-Terry reward model and a list-wise contrast algorithm, only to discover that MCTS sampling at each training step, simultaneous reward model training, and list-wise loss computation on large groups is computationally infeasible at the scale needed for meaningful results. The framework provides no mechanism for reasoning about such practical constraints.

This limitation is particularly consequential for resource allocation decisions. Organizations deciding how to invest their alignment budget need to understand not just what methods exist but what their total cost of ownership is: data collection costs, training compute, inference overhead, engineering effort, and ongoing maintenance. The survey provides none of this information. A method like RFT (rejection sampling fine-tuning) with rule-based feedback may be orders of magnitude cheaper than PPO with a trained reward model while achieving comparable performance on tasks with objective correctness criteria, but the framework gives them equal billing without acknowledging this cost differential.

What evidence exists in the paper. The paper does not discuss computational cost, training stability, or implementation complexity for any surveyed method. No FLOP counts, GPU-hour estimates, or training time comparisons are provided. The running examples describe algorithmic procedures but not resource requirements. The future directions section calls for "higher training efficiency" as a goal but does not analyze which existing methods are already efficient or why.

Mitigation status. Not addressed. The paper's scope is explicitly conceptual and taxonomic; cost analysis is outside that scope. However, the omission means that the survey is more useful for researchers designing new methods than for engineers deploying existing ones, which limits its audience and impact. A companion analysis that overlays cost estimates onto the four-axis framework would significantly increase the survey's practical value.

6.4 The Framework May Not Accommodate Methods That Blur Component Boundaries

The assumption or constraint. The four-axis decomposition assumes that data, feedback, and algorithm are cleanly separable components with well-defined interfaces: data provides (x, y) pairs; feedback assigns r to those pairs; the algorithm computes δ from (x, y, r) and updates the model. This assumption works well for the standard RLHF and DPO pipelines that dominate the surveyed literature. However, several important alignment methods blur these boundaries in ways that strain the framework:

Constitutional AI (Bai et al., 2022b) uses the LLM itself to generate critiques and revisions of its own outputs according to a "constitution" of principles, then trains on the revised outputs. In this pipeline, the model is simultaneously the data generator, the feedback provider (through self-critique), and the policy being optimized. Is the feedback "LLM-as-a-judge" or "direct" (since the constitution provides rules)? Is the algorithm point-wise (the revised output is treated as positive training data) or does the critique-revision pair constitute a form of implicit contrast? The framework's clean separation between feedback and algorithm, and between the model being optimized and the mechanism providing feedback, is challenged when both are the same LLM.
SPIN and self-play methods iteratively generate data from the current policy, treat previous policy versions as "opponents," and construct preference pairs from the policy's own outputs. The data is on-policy (generated by the current model), the feedback comes from the model's own scoring of its outputs, and the algorithm updates the model to prefer its own improved outputs over previous versions. This collapses the data/feedback/model distinction — the feedback signal emerges from the comparison between current and previous policy outputs, not from an external oracle.
Iterative self-improvement loops (e.g., STaR, ReST^EM) generate reasoning traces, filter correct ones, train on them, and repeat. The "feedback" is the correctness check (rule-based), but the "data" at each iteration is generated by the policy from the previous iteration, and the "algorithm" is standard SFT. The framework can classify this as {on-policy, direct feedback, point-wise} but struggles to capture the iterative, bootstrapping nature of the process — the framework treats each component as static (data comes from somewhere, feedback comes from somewhere, algorithm computes updates) but self-improvement loops are fundamentally dynamic, with the model's own outputs becoming the training data for its next iteration.

The survey does not discuss these boundary cases. Section 4.2 mentions that "using preference data sampled from πθ₀ to train πθt for t > 0 is also off-policy," which acknowledges the temporal aspect but does not address the circularity when πθ is both data generator and optimization target across multiple iterations.

The consequence. Methods that exploit the circular relationship between data generation, feedback provision, and policy optimization — which are among the most promising approaches for scalable alignment because they reduce dependence on external supervision — may be poorly served by a taxonomy that assumes these functions are performed by separate, static components. The framework may classify them as special cases of the standard axes (self-play as {on-policy, LLM-as-a-judge, pair-wise}) but this classification obscures their distinctive mechanism: the feedback signal is emergent from the optimization process itself rather than externally provided.

This limitation matters because self-play and iterative self-improvement are active research frontiers. If the framework cannot cleanly accommodate them, researchers working in these areas may find the taxonomy less useful, and the survey may fail to influence a significant segment of the alignment literature. The taxonomic boundaries that make the framework clean for the standard RLHF/DPO pipeline may make it brittle for more complex feedback loops.

What evidence exists in the paper. The paper cites self-play methods (SPIN is referenced as Wu et al., 2024; self-rewarding language models are cited as Yuan et al., 2024) but classifies them under LLM-as-a-Judge feedback (Section 5.2.3) without discussing the circularity challenge. Constitutional AI is not discussed. Iterative methods like STaR and ReST^EM are mentioned in passing but not analyzed in detail. The framework's clean component separation is demonstrated on the "standard" methods (PPO, DPO, RFT, RAFT) where the separation holds, and the paper does not stress-test the taxonomy on boundary cases.

Mitigation status. Not addressed. The paper acknowledges in Section 1 that its scope is "textual preference alignment" and that it excludes related topics, but the exclusion of constitutional AI and self-play analysis is not explicitly justified. The framework's inability to cleanly accommodate these methods is a limitation of the taxonomy itself, not merely a scope constraint — it suggests that the four-axis decomposition, while powerful for a large class of methods, may not be the right abstraction for all alignment approaches.

6.5 The Survey Provides No Systematic Empirical Evidence That the RL-vs-SFT Divide Is Actually Harmful

The assumption or constraint. The paper's primary motivation and novel contribution is the claim that the traditional RL-vs-SFT categorization of alignment methods is harmful — that it creates an "artificial barrier" between groups of work, obscures shared structure, and impedes cross-pollination of ideas. The authors state this explicitly:

"This split can unconsciously result in a barrier between the two groups of works, which is not conducive to further understanding of researchers for the common core of preference alignment."

This is an empirical claim about the sociology and practice of alignment research: that researchers working under the old taxonomy make suboptimal decisions because the taxonomy misleads them about the relationships between methods. The paper uses this claim to motivate its entire reorganization — if the old taxonomy were adequate, there would be no need for a new one.

The consequence. The paper never validates this motivating claim. It does not demonstrate that researchers in the RL camp have failed to adopt insights from the SFT camp (or vice versa) because of the taxonomy. It does not show that papers using the old taxonomy make systematic errors in experimental design or method selection. It does not provide examples of missed research opportunities that the old taxonomy obscured and the new taxonomy reveals. The claim is purely rhetorical — it appeals to the reader's intuition that a cleaner taxonomy is better — but it provides no evidence that the old taxonomy actually caused harm.

This is significant because the conceptual reorganization the paper proposes is not costless. Adopting a new taxonomy requires the field to relearn how methods relate, retrain intuitions about which comparisons are natural, and restructure how papers are written and reviewed. If the old taxonomy was adequate — if researchers were already aware that PPO and DPO share mathematical structure, and the RL-vs-SFT split was a convenient shorthand rather than a genuine barrier — then the new taxonomy may not be worth the switching cost. The paper never makes the case that the old taxonomy's costs outweigh the new taxonomy's adoption costs, because it never quantifies either.

What evidence exists in the paper. None. The paper does not conduct any analysis of the RL-vs-SFT taxonomy's effects on research practice — no citation analysis showing that RL papers rarely cite SFT papers (or vice versa), no survey of researcher beliefs about method relationships, no historical reconstruction of missed cross-pollination opportunities. The argument is entirely conceptual: "the gradient formulation is unified, therefore the old taxonomy is misleading." But the gap between conceptual unity and practical harm is large, and the paper does not bridge it.

Mitigation status. Not addressed. The paper treats the inadequacy of the old taxonomy as self-evident once the unified gradient formulation is presented, which is a reasonable rhetorical move for a survey introducing a new framework, but it leaves the paper vulnerable to the critique that it is solving a problem that didn't exist. A researcher already comfortable with the RL-vs-SFT split might read this survey and conclude that the new taxonomy is elegant but unnecessary — it reorganizes the literature without revealing genuinely new connections that were invisible under the old taxonomy.

6.6 The Framework's Algorithm Categorization May Not Capture the Most Important Practical Distinctions

The assumption or constraint. The paper categorizes algorithms by the number of samples needed to compute the gradient coefficient δ: point-wise (one sample), pair-wise contrast (two samples), list-wise contrast (N > 2 samples), plus training-free (no gradient). This is a principled, mathematically grounded categorization that directly reflects data requirements and optimization granularity. However, the number of samples per update may not be the most salient distinction for practitioners choosing between algorithms.

Within a single category — say, pair-wise contrast — the surveyed methods differ dramatically in ways that may matter more for practical performance than the fact that they all compare two samples:

Reference model dependence: DPO requires a reference model for its implicit KL penalty; SimPO removes the reference model entirely and adds a length-normalized margin instead. This is a major architectural and computational difference — maintaining a reference model doubles memory requirements and can be a deployment constraint on limited hardware — but the taxonomy places both methods in the "pair-wise contrast" bucket without distinguishing them.
SFT loss integration: ORPO combines SFT and preference alignment into a single stage by adding an odds-ratio term to the standard SFT loss. Standard DPO requires a separate SFT stage before preference optimization. The pipeline difference (one-stage vs. two-stage) has significant implications for training cost, data requirements, and hyperparameter tuning, but the taxonomy treats both as pair-wise contrast.
Online vs. offline deployment: The paper explicitly decouples this from algorithm choice, arguing it is a data collection decision. But in practice, DPO is almost always run offline with pre-collected preference data, while PPO is almost always run online with real-time reward model feedback. The practical defaults matter because they determine infrastructure requirements — setting up online training with real-time feedback is an engineering challenge that offline training avoids, and this practical difference may outweigh any conceptual similarity in δ computation.
Loss function engineering: Within pair-wise contrast, methods differ in their loss functions in ways that affect training dynamics: IPO constrains the score difference upper bound to prevent overfitting; f-DPO generalizes the divergence constraint; EXO reformulates the objective in terms of reverse KL divergence; DPO-positive adds a penalty term to prevent reducing the likelihood of the chosen response. These loss-function-level differences may have larger effects on final model quality than the fact that all methods use two samples per update, but the taxonomy groups them together under a single category.

The consequence. The framework's categorization, while mathematically clean, may obscure the differences that practitioners care about most. A reader looking at the taxonomy learns that DPO, SimPO, ORPO, IPO, and f-DPO are all "pair-wise contrast" methods, which implies they are fundamentally similar. But in practice, choosing between them requires understanding their reference model requirements, training pipeline structure, loss function properties, and hyperparameter sensitivity — distinctions that the taxonomy flattens. The framework's abstraction level may be too high for the decisions it aims to inform.

This limitation is compounded by the fact that the feedback and algorithm categories interact in practice but are treated as independent in the taxonomy. For example, pair-wise contrast methods with LLM-as-a-judge feedback may behave very differently from pair-wise contrast methods with rule-based feedback, because LLM-as-a-judge feedback has its own biases (verbosity preference, position bias, self-preference) that interact with the contrastive loss in complex ways. The framework provides no mechanism for reasoning about these interactions — it treats feedback choice and algorithm choice as separable when they may not be.

What evidence exists in the paper. Table 1 lists loss functions for eight algorithms, showing their surface-level differences, but the surrounding text in Section 6.2 does not systematically compare these methods on dimensions like training stability, computational cost, or sensitivity to data quality. The running examples cover a subset of methods but do not compare them head-to-head. The reader is left to infer practical distinctions from the loss function equations and the brief qualitative descriptions, which is insufficient for decision-making.

Mitigation status. Not addressed. The paper's commitment to a single organizing principle (sample count per update) is intellectually coherent but comes at the cost of flattening other important dimensions. A multi-dimensional categorization that also captures reference model dependence, training pipeline structure, and feedback-algorithm interaction would be more complex but potentially more useful for practitioners. The paper does not discuss this tradeoff or acknowledge that its chosen categorization axis may not be the most informative one for all purposes.

7. Implications and Future Directions

How This Work Changes the Landscape

This survey does not propose a new algorithm or training technique — its contribution is architectural rather than technical. It reorganizes the preference learning literature into a four-axis framework (data, feedback, algorithm, model) unified by a shared gradient coefficient formulation (Equation 1), and in doing so, it aims to shift how the field thinks about alignment research rather than how it performs alignment in practice. The magnitude of this shift is best characterized as a conceptual reframing — not a paradigm shift on the scale of DPO replacing PPO, but a reorientation comparable to the transformer's attention formulation becoming the shared vocabulary for sequence modeling. If adopted, it would transform how alignment papers are introduced, compared, and evaluated.

The reframing has several concrete consequences for how researchers approach the field:

The RL-vs-SFT divide is demoted from a fundamental taxonomy to a surface-level implementation detail. The paper's most aggressive rhetorical move is to label the traditional categorization — reinforcement learning methods versus supervised fine-tuning methods — as an "artificial barrier" that "obscures the common core of preference alignment." By showing that PPO and DPO both reduce to identical gradient templates differing only in how the coefficient δ is computed, the paper argues that the field has been organizing around a distinction that captures neither the mathematical structure of the optimization problem nor the practical design choices that matter most. If the framework gains traction, papers introducing new alignment methods would be expected to specify their choices along the four axes (data source, feedback mechanism, samples-per-update, model architecture) rather than simply declaring allegiance to the "RL" or "SFT" camp. This would make it natural to ask questions that the old taxonomy obscured: "Does online data collection improve pair-wise contrast methods?" "Can list-wise losses be paired with rule-based feedback?" "What happens when we use a process reward model with a DPO-like loss?"

The paper reconciles a persistent contradiction in the literature: why do PPO and DPO sometimes yield similar results and sometimes diverge sharply? Under the unified framework, the answer becomes structural rather than mysterious. When PPO and DPO produce similar results, it is because the feedback signal (whether scalar reward or binary preference) carries equivalent information, and the δ estimates (advantage vs. log-ratio contrast) have comparable quality and variance. When they diverge, it is because one δ estimate is higher-quality or better-calibrated than the other for that particular task, feedback type, or data distribution — not because "RL is fundamentally better than SFT" or vice versa. This reframes the debate from a competition between algorithm families to an engineering problem: given a specific task and available feedback, what is the most reliable way to estimate δ? Prior surveys could not offer this reconciliation because they presented PPO and DPO as competitors rather than as different parameterizations of the same objective.

Research directions that become more attractive under this framework:

Modular innovation. The framework makes it legitimate to improve one axis (say, a better on-policy sampling strategy) and evaluate it independently, composing it with existing feedback and algorithm choices. Under the old taxonomy, an improvement to data collection was typically bundled with the algorithm it was designed for (e.g., MCTS sampling for PPO-based math reasoning). The framework encourages researchers to ask: "Does this data collection strategy also improve DPO? Does it improve list-wise methods?" — questions that were rarely asked because the old taxonomy associated data collection choices with specific algorithm families.
Systematic ablation of the axes. The framework enables studies that hold three axes fixed and vary the fourth, allowing the field to finally determine whether, say, online vs. offline data collection matters more or less than the choice between point-wise and pair-wise algorithms. This is the kind of controlled comparison that the end-to-end pipeline evaluations typical of the literature (PPO vs. DPO on benchmark X) cannot provide, because those comparisons change multiple axes simultaneously.
Feedback quality as a first-class research problem. By separating feedback from algorithm, the framework elevates feedback design and evaluation from an implementation detail to a central axis of the alignment design space. This makes it natural for papers to focus solely on improving feedback mechanisms (better reward models, more reliable LLM-as-a-judge prompts, domain-specific rule-based signals) without needing to also propose a new algorithm — the improved feedback can be evaluated by plugging it into existing algorithms and measuring the gain.

Research directions that become less central under this framework:

Loss-function micro-optimization without diagnostic clarity. The survey covers dozens of DPO variants that differ in their loss functions (IPO, f-DPO, SimPO, ORPO, sDPO, DPO-positive, EXO) but are all pair-wise contrast methods. Under the unified gradient formulation, these variants differ in how they compute δ from the same two-sample input, but the framework provides no a priori reason to believe that one δ computation strategy dominates others across all settings. The proliferation of loss-function variants may reflect overfitting to specific benchmark evaluations rather than genuine progress — and the framework, by revealing that these methods occupy the same point in the design space, should encourage researchers to investigate which differences in δ computation actually matter rather than adding another term to the loss function and reporting a 0.5% improvement on AlpacaEval.
End-to-end pipeline comparisons as the primary evaluation paradigm. If the framework's claim of axis independence holds, then comparing PPO and DPO end-to-end without controlling for data quality, feedback reliability, and hyperparameter tuning is scientifically uninformative — differences in final performance confound too many variables. The framework encourages component-level evaluation: measure feedback quality independently (how well does the reward model predict human preferences?), measure algorithm sample efficiency independently (how quickly does the method converge given fixed feedback?), and then predict end-to-end performance from component-level metrics. This is a more rigorous, engineering-oriented approach to alignment research, but it is also more demanding than the current norm of single-number benchmark comparisons.

Follow-Up Research This Work Enables

Systematic empirical validation of the unified gradient formulation across algorithm categories. The paper's central mathematical claim — that all gradient-based methods reduce to (coefficient × policy gradient) and differ only in δ — is asserted based on algebraic manipulation but has never been empirically tested. A critical follow-up would train PPO, DPO, and GRPO on the same task (e.g., TL;DR summarization) with carefully controlled feedback (using the same reward model to provide scalar rewards for PPO, pairwise preferences for DPO, and list-wise rankings for GRPO) and compare not just final accuracy but optimization trajectories: do the methods produce similar token-level δ estimates when given equivalent information? Do they converge to policies with comparable KL divergence from the reference? Do they exhibit correlated failure modes (e.g., reward hacking at the same training steps)? Positive results would validate the framework's practical utility; negative results — e.g., PPO and DPO produce systematically different δ estimates even from identical preference information — would reveal that implementation details abstracted by Equation 1 dominate in practice, refining our understanding of when the mathematical unification matters.

A large-scale study on the interaction between feedback type and algorithm category. The framework treats feedback and algorithm as independent axes, but there is plausible reason to suspect interaction effects: pair-wise contrast methods may be more robust to noisy feedback (because the contrast provides an implicit baseline), while point-wise methods may be sufficient when feedback is clean and deterministic. A controlled study would fix the data source (e.g., the same set of prompts and reference responses), vary feedback type (rule-based correctness, trained reward model at different quality levels, LLM-as-a-judge with different prompt qualities), and run each feedback type with a point-wise method (RFT/KTO), a pair-wise method (DPO), and a list-wise method (GRPO). The key measurement would be the feedback quality × algorithm interaction: does the performance gap between DPO and RFT shrink as feedback becomes cleaner, as the framework's variance-reduction logic would predict? The study would provide the prescriptive guidance the survey lacks — for a given feedback quality level, which algorithm class is optimal?

Combining process reward models (PRMs) with pair-wise contrast losses for mathematical reasoning. The survey identifies PRMs (Section 5.2.1, feedback axis) and DPO-like losses (Section 6.2, algorithm axis) as independently developed literatures with no existing combination. A concrete experiment: take Math-Shepherd (Wang et al., 2024c) style PRM training (unsupervised step-level supervision via Monte Carlo rollouts), generate on-policy reasoning chains from a 7B base model, and construct preference pairs where the "chosen" response is one that the PRM assigns a high step-level score trajectory and the "rejected" response is one that deviates to low PRM scores despite arriving at the same or comparable final answer. Train with a DPO loss where the implicit reward is the aggregate PRM score along the reasoning path rather than an outcome-level preference label. Compare against (a) standard DPO with outcome-level preference labels, (b) PPO with PRM step-level rewards, and (c) best-of-N selection with PRM scoring. The hypothesis is that PRM + DPO combines the credit assignment granularity of process supervision with the simplicity and stability of contrastive fine-tuning, potentially outperforming both PPO (which is harder to tune) and outcome-level DPO (which cannot distinguish correct reasoning from lucky guesses). This experiment directly fills a gap the taxonomy makes visible.

Online list-wise contrast with rule-based feedback for code generation. The framework reveals {online, rule-based, list-wise} as a coherent combination that the literature has barely explored. Code generation is ideal for this because unit tests provide objective, deterministic feedback at zero annotation cost, and the ranking of multiple candidate solutions by test pass rate carries genuine graded preference information (solutions passing more tests are genuinely better). A concrete experiment: sample N candidate solutions per problem from the current policy, execute them against the test suite, rank by number of passing tests, and train with a list-wise contrast loss (e.g., PRO or LiPO) that weights the gradient by the quality gap between ranks. Compare against (a) offline DPO with binary pass/fail labels (which discards the graded preference signal), (b) PPO with a scalar reward equal to the fraction of tests passed, and (c) rejection sampling fine-tuning that keeps only fully correct solutions. The hypothesis is that online list-wise contrast should outperform point-wise and pair-wise methods because it (a) uses the policy's own distribution (online data avoids distribution shift), (b) exploits the full ranking information (not just binary correct/incorrect), and (c) avoids the instability and critic model overhead of PPO. Success would establish a new strong baseline for code generation alignment; failure would suggest that the benefits of list-wise ranking are empirically small compared to simpler methods, which is also a valuable finding.

A meta-analysis of DPO variant proliferation: which loss function differences actually matter? The survey's Table 1 lists eight pair-wise contrast loss functions, and Section 6.2 describes many more. Under the unified gradient formulation, all these variants compute δ from the same two inputs (chosen and rejected responses) but differ in how they transform the log-ratio, what penalties they add, and whether they include reference models or SFT terms. A systematic study would re-implement these variants in a controlled setting (same data, same base model, same hyperparameter tuning budget) and measure not just final benchmark scores but convergence speed, sensitivity to data quality, output diversity, and KL divergence from the reference. The goal is to identify which loss-function modifications produce robust improvements across tasks and which are essentially cosmetic — a finding that would be enormously valuable for practitioners and would help the field converge on a smaller set of well-understood variants. The survey's framework enables this study by identifying the common structure behind the variants; without the unified gradient formulation, it is natural to treat each variant as a separate algorithm requiring separate analysis, which is precisely why such a systematic comparison has not been done.

Stress-testing the axis independence claim: does changing one axis break the others? The paper's strongest claim is that data, feedback, and algorithm are genuinely independent design axes — any combination is theoretically coherent and potentially useful. This claim should be stress-tested by looking for broken combinations. For example: if we use an LLM-as-a-judge that has known verbosity bias (Zheng et al., 2023) as the feedback source for a list-wise contrast method, does the list-wise ranking amplify the verbosity bias by preferentially ranking verbose responses across all positions, while a pair-wise contrast that only distinguishes "best" from "rest" partially masks it? If so, the combination {LLM-as-a-judge, list-wise} may be strictly dominated by {LLM-as-a-judge, pair-wise} not for fundamental reasons but because of a specific feedback bias × algorithm interaction. Identifying such broken combinations would delineate the boundaries of axis independence and provide the "navigation layer" that the survey currently lacks. The experiment would systematically test the 2×3×4 = 24 combinations of {on-policy, off-policy} × {rule-based, reward model, LLM-judge} × {point-wise, pair-wise, list-wise, training-free} on a single well-characterized task, measuring both performance and pathological behaviors (reward hacking, length exploitation, distribution collapse).

Practical Applications and Downstream Use Cases

Modular alignment pipeline design for organizations with heterogeneous task requirements. The survey's four-axis decomposition is directly useful for engineering teams that need to build alignment pipelines spanning multiple domains (code generation, dialogue, summarization, math reasoning) with different feedback availability and quality requirements. Instead of adopting a single alignment method (e.g., "we use DPO for everything"), the team can use the framework to make per-task component selections: for code generation, pair on-policy sampling with rule-based feedback (unit tests) and a list-wise contrast algorithm that exploits the graded test pass signal; for dialogue, use off-policy human preference data with a pair-wise contrast method (DPO) because list-wise rankings are unavailable; for mathematical reasoning, use MCTS-based on-policy data with a PRM for step-level feedback and a point-wise method (RFT on verified traces). The framework provides the shared vocabulary and conceptual scaffolding to design these per-task pipelines without reinventing the infrastructure for each domain — the same training loop can accommodate different data/feedback/algorithm configurations for different batches, reducing engineering overhead. This modularity is implicit in the literature but has not been explicitly systematized; the survey makes it actionable.

Resource-constrained alignment: selecting the cheapest feedback mechanism that is "good enough." For startups or academic labs with limited annotation budgets and no access to GPT-4 APIs, the framework's separation of feedback from algorithm enables cost-aware decision-making that the literature does not currently support. The key insight: feedback quality can often be traded against algorithm sophistication. If rule-based feedback is available (code correctness, math answer checking), a simple point-wise method like RFT may achieve comparable performance to a more complex method using a trained reward model, because the feedback is deterministic and noise-free — the gradient coefficient δ is perfectly calibrated, so sophisticated baseline subtraction (PPO) or contrastive normalization (DPO) provides minimal benefit. Conversely, if only noisy LLM-as-a-judge feedback is available, the additional variance reduction from pair-wise or list-wise contrasts may be worth the implementation complexity. The survey provides no cost numbers, but its structure makes it straightforward for a practitioner to map their available feedback sources onto the framework and identify the algorithm category that best matches their feedback reliability. This is a form of "alignment engineering" that the field currently lacks — the survey provides the taxonomy for it even without providing the specific recipes.

Curriculum-based alignment: varying data and feedback across training stages using the same taxonomy. The framework's temporal dimension is implicit (the preference learning pipeline in Algorithm 1 iterates) but underexploited. A practical deployment might use different data/feedback/algorithm configurations at different training stages: start with off-policy human preference data and a pair-wise contrast method (DPO) to establish basic alignment cheaply; then switch to on-policy data with LLM-as-a-judge feedback and continue with DPO to reduce distribution shift; finally, use the resulting model as the base for a short PPO phase with a high-quality reward model trained on the policy's own outputs, exploiting the reduced gradient variance of advantage estimation for the final fine-tuning. This curriculum approach is natural under the four-axis framework — each stage is simply a different point in the design space — but has not been systematically explored because the old taxonomy encouraged committing to one "pipeline" (RLHF vs. DPO) for the entire training run. The framework makes it obvious that the axes can be varied dynamically, potentially combining the sample efficiency of offline DPO early in training with the distributional correctness of online PPO late in training.

Evaluating alignment research proposals through a structured design-space lens. For grant reviewers, hiring managers, and research leads deciding which alignment projects to fund or pursue, the framework provides a evaluation rubric that is more systematic than "this paper proposes a new variant of DPO." A research proposal can be assessed by which axis it innovates on and whether the innovation is likely to compose with improvements on other axes. A proposal to improve reward model training (feedback axis) is more valuable if it demonstrates that the improved reward model benefits point-wise, pair-wise, and list-wise algorithms rather than just one, because that validates the axis independence claim and suggests the improvement is fundamental. Conversely, a proposal to add another term to the DPO loss (algorithm axis, within the pair-wise category) should be scrutinized for whether the modification addresses a specific failure mode that the unified gradient formulation can diagnose (e.g., "δ estimates have high variance when the chosen-rejected margin is small, and we fix this by...") or is simply another micro-optimization whose benefit will not generalize. This use of the framework — as a quality filter for research directions — may be its most impactful practical application, because it helps the field allocate scarce research resources toward improvements that compose and away from improvements that are isolated to a single point in an already densely explored part of the design space.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Preference Learning Research Is Fragmented and Disconnected

Why This Problem Matters: Practical and Scientific Stakes

Prior Surveys and Where They Fall Short

How This Paper Positions Itself

The Intellectual Lineage: From RLHF to a Unified Design Space

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Unified Gradient Coefficient Formulation

The Preference Learning Pipeline (Algorithm 1)

Preference Data: On-Policy vs. Off-Policy Collection

On-Policy Data Collection

Off-Policy Data Collection

Feedback: Direct vs. Model-Based Preference Signals

Direct Feedback (No Trained Model Required)

Model-Based Feedback (Requires Training or Prompting a Separate Evaluator)

Training-Free Alignment: Optimizing Without Updating Parameters

Input Optimization (Modify the Prompt)

Output Optimization (Modify the Response Distribution or Post-Process Outputs)

Summary of the Unified Framework's Design Choices

4. Key Insights and Innovations

Innovation 1: The RL-vs-SFT Divide Is an Artifact, Not a Fundamental Distinction

Innovation 2: Decoupling Algorithm, Data, and Feedback as Independent Design Axes

Innovation 3: The Gradient Coefficient as the Universal "Difference-Maker"

Innovation 4: Explicitly Identifying Underexplored Regions of the Design Space as a Research Contribution

Innovation 5: Reframing "Alignment" as a Composition of Four Independent Supply Chains

5. Experimental Analysis

Evaluation Methodology

What the Paper Does Provide (in lieu of experiments)

Main Comparative Claims from the Literature

Claim 1: DPO can achieve comparable or superior performance to PPO while being computationally simpler

Claim 2: Online methods generally outperform offline methods due to reduced distribution shift

Claim 3: SimPO (removing the reference model) outperforms DPO on certain benchmarks

Claim 4: KTO (point-wise method requiring only binary preference labels) matches DPO performance with simpler data requirements

What the Paper's Framework Claims vs. What Is Empirically Validated

Coverage Analysis: What Methods Are Included and What Is Missing

The Framework's Predictive Power: What Experiments Would Test It

Critical Assessment

Does the framework actually unify, or does it merely relabel?

The survey does not empirically validate its own motivation.

The "point-wise / pair-wise / list-wise" categorization is clean but may not capture the most important algorithmic differences.

Missing: a systematic analysis of failure modes.

The paper's strength is also its weakness: it is a map, not a guide.

6. Limitations and Trade-offs

6.1 The Framework Is a Conceptual Reorganization, Not an Empirically Validated Theory

6.2 The Framework Does Not Guide Decision-Making — It Maps the Design Space but Provides No Navigation

6.3 The Framework Does Not Account for Computational Cost, Training Stability, or Implementation Complexity

6.4 The Framework May Not Accommodate Methods That Blur Component Boundaries

6.5 The Survey Provides No Systematic Empirical Evidence That the RL-vs-SFT Divide Is Actually Harmful

6.6 The Framework's Algorithm Categorization May Not Capture the Most Important Practical Distinctions

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function

Language Models Learn to Mislead Humans via RLHF