Direct Language Model Alignment from Online AI Feedback

ArXiv: 2402.04792

🎯 Pitch

DPO and similar direct alignment methods suffer from a critical flaw: they train on stale, off-policy data generated by a different model. This paper shows that simply switching to online, on-policy feedback from an LLM annotator—having it judge the current model's own outputs during training—boosts average human win rate to ~66% over offline baselines. Remarkably, this online feedback is also steerable via simple prompts, enabling control over attributes like response length without extra training.

---

1. Executive Summary

This paper proposes Online AI Feedback (OAIF), a method that makes direct alignment from preferences (DAP) methods online and on-policy by using an LLM annotator to provide preference feedback on responses sampled from the current policy during training (e.g., prompting PaLM 2-L to choose between two generations from the evolving PaLM 2-XS policy), rather than relying on a pre-collected, static preference dataset from a different model. Across TL;DR, Anthropic Helpfulness, and Harmlessness benchmarks with PaLM 2 models, OAIF enables online DAP methods—DPO, IPO, and SLiC—to achieve an average human win rate of ~66% over their offline counterparts, with online DPO preferred 58% of the time over RLHF, RLAIF, and the SFT baseline in 4-way comparisons. The paper further demonstrates that the LLM annotator's feedback is prompt-controllable—reducing average response length from ~120 to ~40 tokens by instructing the annotator to prefer shorter responses—while establishing that OAIF's effectiveness depends on annotator quality, as performance degrades when using smaller annotating LLMs.

2. Context and Motivation

The Core Problem: Offline and Off-Policy Data in DAP Methods

The fundamental issue this paper addresses is deceptively simple: the preference data used by direct alignment from preferences (DAP) methods like DPO is collected from the wrong model, at the wrong time, and never updated. This creates two intertwined problems that undermine the effectiveness of DAP methods, even though DAP methods are theoretically simpler and more stable than traditional RLHF.

To understand why this matters, we need to trace the data pipeline. DAP methods like DPO, IPO, and SLiC all operate on a fixed dataset $\mathbb{D}$ containing triples $({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-)$—a prompt, a preferred response, and a dispreferred response. This dataset is typically collected once, before training begins, by sampling responses from some existing LLM $\rho$ and having humans or AI label which response they prefer. The paper identifies two distinct problems with this setup:

Problem 1: Offline feedback (Section 2, "Offline feedback"). The preference data is pre-collected and frozen. The policy being aligned, $\pi_{\bm{\theta}}$, can never get feedback on its own generations during training. This is the offline nature of DAP methods—the feedback loop is broken. As the authors note, collecting online preferences from humans is "not feasible, as there is no human-in-the-loop." This means the alignment signal is always stale, never adapting to how the model's outputs change over time.

Problem 2: Off-policy learning (Section 2, "Off-policy learning"). Even more critically, the responses in the dataset were generated by a different model $\rho$ than the one being aligned. At training step $t$, the policy $\pi_{{\bm{\theta}}^t}$ is updating, but the training data still comes from the static distribution induced by $\rho$. This is the off-policy problem: $\pi_{{\bm{\theta}}^t} \neq \rho$. As Figure 2 illustrates, there are actually two distribution shifts at play:

• An initial distribution shift: $\rho \neq \pi_{{\bm{\theta}}^0}$. The model that generated the dataset ($\rho$) is fundamentally different from even the initial SFT baseline ($\pi_{{\bm{\theta}}^0}$). DPO attempts to mitigate this by SFT-finetuning $\pi_{\bm{\theta}}$ on $\mathbb{D}$ so that $\pi_{{\bm{\theta}}^0} \approx \rho$, but this is only an approximation.
• A gradual distribution shift during training: $\pi_{{\bm{\theta}}^0} \neq \pi_{{\bm{\theta}}^t}$. Even if the initial policy matches the data-generating policy, the policy being aligned keeps evolving during DAP training. The data remains static, but the model doesn't—so the training becomes increasingly off-policy as training progresses.

The paper provides an empirical verification of this distribution shift in Appendix B (Figure 8), using GPT-2 Large as the policy $\pi_{\bm{\theta}}$ and PaLM 2-S as $\rho$ (producing off-policy responses). The log-probabilities assigned by GPT-2 Large to on-policy responses (${\bm{y}}^+$, ${\bm{y}}^-$) versus off-policy responses ($\bar{{\bm{y}}}$) show a clear gap, confirming that models assign significantly lower likelihood to generations from a different model family—exactly the scenario when training on a pre-collected dataset.

Why This Problem Matters

The offline and off-policy nature of DAP methods has concrete, measurable consequences that the paper demonstrates empirically:

Overfitting to the static dataset. In Figure 3, the authors track the win rate of offline DPO against the SFT baseline on TL;DR. The red curve shows a sharp drop around training step 3,500—offline DPO rapidly overfits the fixed, off-policy preference data. This is a classic failure mode: the model learns to game the patterns in the static dataset rather than actually improving its generation quality.

Suboptimal performance compared to what could be achieved. The human evaluation in Table 2 quantifies the cost: on TL;DR, offline DPO achieves a win rate of only 7.69% against online DPO, with a quality score of 3.46 vs. 3.95 (on a 1–5 scale). Similar gaps appear across Helpfulness and Harmlessness tasks. The model isn't failing to learn—it's learning from the wrong signal.

A structural disadvantage relative to RLHF. The RL step in RLHF is inherently online and on-policy: at each step, responses are sampled from $\pi_{{\bm{\theta}}^t}$, scored by a reward model $r(\cdot; {\bm{\phi}})$, and used to update the policy immediately (Equation 4 in Appendix A). This continuous feedback loop prevents the overfitting and distribution shift that plague offline DAP. As the authors note in Section 2, "Thanks to the online nature of RL, RL methods are also on-policy, as the responses used to update $\pi_{{\bm{\theta}}^t}$ are all sampled from it." This is precisely the advantage that DAP methods lack out of the box.

If DAP methods are simpler, more stable, and more efficient than RLHF (no separate RM, no policy gradients, no value function), but they suffer from a structural data problem that prevents them from reaching their potential, then solving this data problem would combine the best of both worlds. This is the paper's central motivation: bridge the gap between DAP and RLHF by making DAP online.

Prior Approaches to Making DAP Online—and Where They Fall Short

The paper identifies that prior work has recognized this problem and attempted solutions, but each has a fundamental limitation:

RM-based online feedback (Iterative DPO, RSO, West-of-N). The most intuitive solution is to train a reward model (RM) on the offline preference dataset $\mathbb{D} \sim \rho$, then use that RM to provide online feedback—scoring or ranking responses sampled from $\pi_{{\bm{\theta}}^t}$ during DAP training. Methods like Iterative DPO (Xu et al., 2023), RSO (Liu et al., 2023), and West-of-N (Pace et al., 2024) take this approach, essentially letting the RM pseudo-label on-policy generations.

The paper identifies a crucial flaw in this strategy (Section 2, "RM-based online feedback for DAP methods"): the distribution shift doesn't disappear—it just moves to the RM. The RM is trained on responses from $\rho$, but at inference time (during DAP training), it must score responses from $\pi_{{\bm{\theta}}^t}$, where $\pi_{{\bm{\theta}}} \neq \rho$. This is an out-of-distribution RM inference problem, as detailed in Appendix A.3. The RM's preference predictions may not generalize to the policy's evolving output distribution, creating a hidden distribution shift that undermines the supposed online nature of the method.

The paper provides experimental evidence for this claim (Section 4.4): "We also trained an online DPO with the same RM used for RLAIF. It outperforms RLAIF, but significantly underperforms online DPO with OAIF, with a win rate of <30% judged by Gemini Pro." Using an RM for online feedback is better than purely offline DPO, but it's substantially worse than using an LLM annotator directly—because the LLM annotator doesn't suffer from the RM's distribution-shift problem (the LLM wasn't trained on a specific response distribution; it's a general-purpose language model that can evaluate any text).

Retraining the RM synchronously. In theory, the RM could be periodically retrained on responses from $\pi_{{\bm{\theta}}^t}$ (Ziegler et al., 2019). The authors acknowledge this is "feasible theoretically" but note it "would greatly complicate the training pipeline and increase training cost." The complexity and expense of this approach make it impractical as a general solution.

Self-rewarding (Yuan et al., 2024). The concurrent work by Yuan et al. proposes having the policy being aligned ($\pi_{{\bm{\theta}}^t}$) itself serve as the annotator—the "self-rewarding" approach. This is genuinely online and avoids external models entirely. However, as the authors point out in Section 5 ("Self-annotating models"), this imposes a constraint: "the model architecture and size have to be the same" for both the generation and annotation tasks. If you have access to a stronger LLM for annotation (e.g., a larger model), you can't leverage it under this paradigm. Moreover, the generative and discriminative capabilities of the same model may not be equally strong—a model that generates reasonable responses might not be a reliable judge of preferences.

How OAIF Positions Itself Relative to These Approaches

OAIF's key insight is to use an LLM as an online annotator—not a trained RM, and not (necessarily) the policy being aligned itself. This is summarized in Table 1, which compares the characteristics of different DAP approaches:

OAIF occupies a unique position in this taxonomy: online, on-policy, and RM-free. It achieves this by following the RLAIF paradigm (Bai et al., 2022b; Lee et al., 2023)—using an LLM to provide preference judgments—but applying it during DAP training on the policy's own generations, rather than using it to label an offline dataset.

The crucial advantage over RM-based approaches is that the LLM annotator avoids the RM's distribution shift problem entirely. An LLM providing pairwise preference judgments (e.g., "which summary is better?") is performing a natural language understanding task that generalizes across different response distributions. It wasn't trained to score responses from a specific model $\rho$; it was pretrained on vast corpora and can assess text quality broadly. This means the feedback remains reliable even as $\pi_{{\bm{\theta}}^t}$ evolves.

The advantage over self-rewarding is flexibility: the annotating LLM can be arbitrarily larger, stronger, or differently-tuned than the policy being aligned. As Section 4.5 demonstrates, using a PaLM 2-L annotator produces much stronger results than using a PaLM 2-XS annotator, and the paper explicitly argues that "the choice of LLM annotator should not necessarily be limited to the model being aligned, especially when an LLM annotator of larger size or higher quality is available" (Section 5).

A further advantage OAIF inherits from its prompting-based nature is text-controllability. Because the feedback signal comes from prompting an LLM (not from a fixed, trained RM), the desired alignment behavior can be adjusted simply by changing the prompt instructions to the annotator. Need shorter responses? Tell the annotator to prefer shorter responses. Need more empathetic responses? Adjust the prompt accordingly. This flexibility is impossible with a pre-trained RM and would require expensive retraining in RLHF/RLAIF. This controllability is demonstrated experimentally in Section 4.6 using response length as a testbed.

The Gap This Paper Fills

In summary, the paper identifies a clear gap: DAP methods are simpler and more efficient than RLHF, but their reliance on offline, off-policy preference data creates a fundamental performance ceiling. Prior attempts to solve this (RM-based pseudo-labeling, self-rewarding, synchronous RM retraining) either shift the distribution-shift problem rather than solving it, or impose undesirable constraints on model choice and training complexity. OAIF fills this gap by introducing a method that is simultaneously online, on-policy, RM-free, and flexible in annotator choice—combining the structural advantages of DAP with the online nature of RLHF.

3. Technical Approach

3.1 Reader Orientation

This paper develops a training procedure that modifies how preference-based alignment losses (like DPO) consume data: instead of feeding a pre-collected, static dataset of preferred/dispreferred response pairs into the loss function over and over, the system generates fresh responses from the current model at each training step and obtains real-time preference judgments on those responses by querying a separate, frozen LLM annotator. The problem OAIF solves is the combination of off-policy and offline data in direct alignment methods—by making both the response generation and the preference labeling happen at training time, from the current policy, the system eliminates the distribution shift between the data-generating policy and the policy being trained, while avoiding the complexity of training and maintaining a separate reward model. The "shape" of the solution is a loop: sample two responses from the current model → ask an LLM which is better → compute a DAP loss on the LLM's preference → update the model → repeat.

3.2 Big-Picture Architecture (Diagram in Words)

OAIF is built from four interacting components that operate in a tight loop at each training step:

1. The policy being trained ((\pi_{{\bm{\theta}}^t})): a PaLM 2-XS model initialized from an SFT checkpoint, denoted (\pi_{{\bm{\theta}}^0}). This is the model that generates responses and gets updated.

2. The prompt dataset ((\mathbb{D}_{\mathcal{X}})): a set of prompts ({{\bm{x}}i}{i=1}^N) extracted from the original preference dataset (\mathbb{D}) by discarding the responses and keeping only the input prompts. No preference pairs are pre-stored.

3. The LLM annotator: a separate, frozen PaLM 2-L model that receives a prompt containing two candidate responses (generated by (\pi_{{\bm{\theta}}^t})) and outputs a preference—which one is better. This annotator is queried via a pairwise prompting scheme.

4. The DAP loss function (\ell({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-, {\bm{\theta}})): one of DPO, IPO, or SLiC loss (Equations 1–3 in the paper), which takes the prompt, the LLM-annotated preferred/dispreferred pair, and the policy parameters, and returns a scalar loss whose gradient updates the policy.

Information flow at each training step (t): A prompt ({\bm{x}}) is sampled from (\mathbb{D}{\mathcal{X}}) → two responses (({\bm{y}}^1, {\bm{y}}^2)) are sampled from (\pi{{\bm{\theta}}^t}(\cdot | {\bm{x}})) using temperature 0.9 → the LLM annotator receives ({\bm{x}}), ({\bm{y}}^1), and ({\bm{y}}^2) through a structured prompt and outputs a preference for one over the other, producing the pair (({\bm{y}}^+, {\bm{y}}^-)) → the loss (\ell({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-, {\bm{\theta}}^t)) is computed → the gradient (\nabla_{\bm{\theta}}\ell) (with a stop_gradient on the sampling and annotation steps) updates ({\bm{\theta}}^t \to {\bm{\theta}}^{t+1}). This loop is formalized in Algorithm 1.

3.3 Roadmap for the Deep Dive

• First, the formal definition of the online/on-policy vs. offline/off-policy distinction (Section 2, Appendices A.1–A.2) must be crystallized, because OAIF's entire contribution rests on transforming a problem from one category to the other. I will make precise what it means for a DAP method to be offline and off-policy, establishing the notation that OAIF breaks.

• Second, the core OAIF algorithm (Algorithm 1) and its gradient computation strategy, including the crucial stop_gradient design choice that makes the method computationally tractable.

• Third, the LLM annotation mechanism: the pairwise prompting scheme, how position bias is addressed, how the preference score is extracted from the annotator's output, and the specific prompts used across different tasks.

• Fourth, the loss functions (DPO, IPO, SLiC) and how they integrate with OAIF—what changes mathematically when the preference data is on-policy versus off-policy, and why the same loss functions work either way.

• Fifth, the hyperparameter configuration and training details that make the system work, including model sizes, temperatures, optimizer settings, and the specific choices that differentiate online from offline DAP experiments.

• Sixth, the text-controllability mechanism (Section 4.6): how the LLM annotator's preference function can be altered by changing the prompt instructions, without any retraining, and why this is a practical advantage over RM-based approaches.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that replacing the pre-collected, static preference dataset in DAP methods with online, on-policy preference judgments from an LLM annotator eliminates the distribution shift that causes offline DAP methods to overfit and underperform, while retaining the simplicity of DAP (no separate reward model, no policy gradients, no value function). The method makes no changes to the DAP loss functions themselves—it changes where the data fed into those losses comes from.

---

The Offline/Off-Policy Problem, Formalized

Before explaining how OAIF works, we need to understand precisely what it replaces. This section formalizes the paper's definitions from Section 2 and Appendix A, which are essential prerequisites.

What "offline" means in DAP. A DAP training procedure is offline if the preference data (({\bm{y}}^+, {\bm{y}}^-)) used to compute the loss at training step (t) was generated before training began by a potentially different process, and is kept fixed throughout training. Formally, for each training example (({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-)) in the batch (\mathbb{B}):

$({\bm{y}}^+, {\bm{y}}^-) \sim \rho(\cdot | {\bm{x}})$

where (\rho) is the policy that generated the dataset (e.g., GPT-2 Large, or an earlier version of the model being aligned), and the preference judgment was made once and frozen. The model being trained, (\pi_{{\bm{\theta}}^t}), never receives feedback on its own generations.

What "off-policy" means in DAP. The training is off-policy if the responses used to compute the loss were not generated by the current policy (\pi_{{\bm{\theta}}^t}). Even if the preference data were collected from (\pi_{{\bm{\theta}}^0}) (the initial SFT checkpoint), the policy evolves during training so that (\pi_{{\bm{\theta}}^t} \neq \pi_{{\bm{\theta}}^0}) for (t > 0). Formally, the condition for on-policy learning is:

$({\bm{y}}^+, {\bm{y}}^-) \sim \pi_{{\bm{\theta}}^t}(\cdot | {\bm{x}})$

Both responses must be sampled from the current policy. If either response comes from a different distribution, the training is off-policy.

Why this matters: the two distribution shifts. Figure 2 in the paper illustrates that in standard offline DAP, there are actually two independent shifts:

1. Initial shift: (\rho \neq \pi_{{\bm{\theta}}^0}). The data-generating model is different from even the starting checkpoint. DPO mitigates this partially by SFT-finetuning (\pi_{\bm{\theta}}) on (\mathbb{D}) so that (\pi_{{\bm{\theta}}^0} \approx \rho), but this is imperfect.

2. Gradual shift: (\pi_{{\bm{\theta}}^0} \neq \pi_{{\bm{\theta}}^t}). As training proceeds, the model changes, so even if the initial distribution matched, the training data becomes increasingly stale.

The consequence, as shown in Figure 3 (the red curve for offline DPO), is that the model initially improves but then overfits to the static dataset — the win rate against the SFT baseline drops sharply around step 3,500 because the model learns to exploit patterns in the fixed preference data rather than genuinely improving response quality.

How RLHF avoids this. In RLHF (Equation 4 in Appendix A.2), the objective is:

$\max_{\bm{\theta}} \mathbb{E}_{{\bm{x}} \sim p_{\mathcal{X}}, {\bm{y}} \sim \pi_{\bm{\theta}}({\bm{y}}|{\bm{x}})} \left[ r({\bm{x}}, {\bm{y}}; {\bm{\phi}}) - \beta \log\left( \frac{\pi_{\bm{\theta}}({\bm{y}}|{\bm{x}})}{\pi_{{\bm{\theta}}^0}({\bm{y}}|{\bm{x}})} \right) \right]$

The response ({\bm{y}}) is sampled from (\pi_{\bm{\theta}}) (the current policy) at each step, and the reward model (r(\cdot;{\bm{\phi}})) provides immediate feedback. This is online (feedback is obtained at training time) and on-policy (responses come from the current policy). The limitation is the complexity: policy gradient methods are needed to estimate the gradient through the expectation, and a value function is typically used to reduce variance, requiring an additional model in memory.

OAIF's goal, restated formally. OAIF aims to transform DAP from offline/off-policy to online/on-policy by replacing the static preference dataset with an LLM annotator that provides preference judgments on the fly, without requiring a trained RM, policy gradients, or a value function. The training data at step (t) becomes:

$({\bm{y}}^1, {\bm{y}}^2) \sim \pi_{{\bm{\theta}}^t}(\cdot | {\bm{x}}), \quad ({\bm{y}}^+, {\bm{y}}^-) = \text{LLM-annotator}({\bm{x}}, {\bm{y}}^1, {\bm{y}}^2)$

This is online because the preference judgment is obtained at training time, and on-policy because the responses are from the current policy.

---

The OAIF Algorithm (Algorithm 1)

Algorithm 1 in the paper specifies the OAIF training loop. I will walk through it line by line, explaining what each component does, what inputs it requires, and what choices were made.

Inputs to the algorithm:

• (T): the number of training steps. This is the total number of gradient updates the policy will undergo.

• (\mathbb{D}_{\mathcal{X}} = {{\bm{x}}i}{i=1}^N): a dataset of prompts only, extracted from the original preference dataset by discarding the response pairs. This is not a preference dataset — it contains only the input text (e.g., the TL;DR posts to summarize, the Anthropic dialogue contexts). Critically, this means OAIF does not require any pre-existing preference labels to operate; it generates its own labels on the fly.

• (\pi_{{\bm{\theta}}^0}): the SFT baseline model. This is a PaLM 2-XS model that has been supervised-fine-tuned on the target task (e.g., on the chosen responses from the TL;DR dataset). It serves as both the starting point for alignment and as the reference policy in the DAP loss functions (the (\pi_{{\bm{\theta}}^0}) terms in Equations 1–3).

• An LLM annotator: a separate, frozen language model (PaLM 2-L in the main experiments) that will provide preference judgments. This model is not trained during OAIF — it is queried for inference only.

• A DAP loss function (\ell({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-, {\bm{\theta}})): one of DPO, IPO, or SLiC loss, parameterized by a scalar (\beta).

The training loop (for simplicity, described with batch size 1):

For each step (t = 0, 1, \ldots, T):

1. Sample a prompt: ({\bm{x}} \sim \mathbb{D}_{\mathcal{X}}). A single input prompt is drawn from the prompt dataset.

2. Sample two responses: ({\bm{y}}^1, {\bm{y}}^2 \sim \pi_{{\bm{\theta}}^t}(\cdot | {\bm{x}})). The current policy generates two independent responses to the prompt. The paper sets the sampling temperature to 0.9 during training to ensure diversity between the two sampled responses — if the temperature were too low, both responses might be nearly identical, making the preference judgment uninformative or degenerate.

3. Get preference from LLM annotator: The annotating LLM receives a structured prompt containing ({\bm{x}}), ({\bm{y}}^1), and ({\bm{y}}^2), and outputs a preference indicating which response is better. The output is processed to produce the pair (({\bm{y}}^+, {\bm{y}}^-)), where ({\bm{y}}^+) is the preferred response and ({\bm{y}}^-) is the less preferred one. The exact mechanism for extracting this preference is described in the next sub-section ("The LLM Annotation Mechanism").

4. Update the policy: The parameters ({\bm{\theta}}^t) are updated to ({\bm{\theta}}^{t+1}) using the gradient (\nabla_{\bm{\theta}} \ell({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-, {\bm{\theta}}^t)). This is a standard gradient step using the chosen DAP loss function.

Output: The aligned policy (\pi_{{\bm{\theta}}^T}).

How the gradient is computed — the stop_gradient decision. A critical technical detail, discussed in the paper's Section 3 ("Gradient computation"), is that the gradient computation treats the sampling and annotation steps as constant with respect to ({\bm{\theta}}). Specifically, the gradient used is:

$\nabla_{\bm{\theta}} \ell({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-, {\bm{\theta}})$

where the dependence of ({\bm{y}}^1, {\bm{y}}^2) on ({\bm{\theta}}) (through the sampling step) and the dependence of (({\bm{y}}^+, {\bm{y}}^-)) on ({\bm{\theta}}) (through the LLM annotator receiving samples as input) are not differentiated through. This is equivalent to applying a stop_gradient (or detach) operation on the sampled responses before feeding them into the loss.

Why this design choice? There are three reasons, one explicit and two implicit in the paper:

1. Computational tractability (explicit): As the paper notes, "involving (\bm{\theta}) in both the response sampling and in the DAP loss function" would mean the gradient must flow through the discrete sampling operation (which requires REINFORCE or similar policy gradient estimators, defeating the purpose of DAP's simplicity) and through the LLM annotator (which is a frozen large model — backpropagating through it would be prohibitively expensive). By contrast, in offline DAP, (\bm{\theta}) is involved only in the loss, and in RLHF, (\bm{\theta}) is involved only in the sampling (the feedback comes from a frozen RM). OAIF would require (\bm{\theta}) in both the sampling and the loss and through the annotator, creating a three-way gradient path that is computationally infeasible.

2. Stability: Differentiating through the LLM annotator's preference output with respect to the responses (and hence with respect to (\bm{\theta})) would create a training signal that depends on the annotator's internal reasoning, which may be noisy, miscalibrated, or optimized in perverse ways. The stop_gradient breaks this path, meaning the only signal driving policy improvement is whether the annotator preferred one response over the other — a simple binary signal that is harder to hack than a differentiable reward.

3. Methodological purity: OAIF uses the LLM annotator as a replacement for a human labeler, not as a differentiable reward model. Humans provide discrete preferences; OAIF treats the LLM annotator the same way. The gradient does not flow through the "labeler" — it flows only through the policy's own probabilities via the DAP loss. This maintains the conceptual simplicity of DAP methods (no reward model gradients, no value function) while adding the benefits of online, on-policy data.

Consequence of this choice: the DAP loss gradients in OAIF are identical in form to the gradients in offline DAP. The only difference is the data: offline DAP computes (\nabla_{\bm{\theta}} \ell({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-, {\bm{\theta}})) using pre-collected (({\bm{y}}^+, {\bm{y}}^-) \sim \rho), while OAIF computes the same gradient form but using fresh (({\bm{y}}^+, {\bm{y}}^-)) from (\pi_{{\bm{\theta}}^t}) annotated by the LLM. This means OAIF is a data-side modification, not a loss-side modification — any DAP loss function works with OAIF without modification.

---

The LLM Annotation Mechanism

The LLM annotator is the component that transforms OAIF from a conceptual idea into a working system. This sub-section explains exactly how an LLM is prompted to provide preference judgments, how the preference is extracted, and how biases are mitigated.

Pairwise prompting scheme. The paper adopts the approach from Lee et al. (2023): the annotating LLM is shown a prompt, two responses, and asked to choose which one is better. The prompt is a structured text template that includes:

• Instructions describing the evaluation criteria
• The input context (the prompt ({\bm{x}}))
• The two responses, clearly labeled (e.g., "Response 1" and "Response 2")
• A final question asking the model to indicate its preference

The specific prompts used differ by task. For TL;DR summarization (Table 5 in Appendix E), the annotator receives definitions of four evaluation axes (coherence, accuracy, coverage, overall quality), the original text, two summaries, and is asked to output "1 or 2 to indicate which summary best adheres to coherence, accuracy, coverage, and overall quality." For Helpfulness (Table 6), the criteria include helpfulness, honesty, and conciseness. For Harmlessness (Table 7), the criteria emphasize avoiding offensive, discriminatory, dangerous, or illegal content, while also being helpful.

Extracting the preference score from the annotator's output. Rather than parsing a free-form text response (which could be unreliable), the paper uses a probabilistic extraction method. The LLM annotator is prompted to output either "1" or "2" (indicating preference for Response 1 or Response 2). The preference score is computed from the log-probabilities of generating the tokens "1" vs. "2":

$P(\text{prefer response 1}) = \frac{\exp(\text{logit}_1)}{\exp(\text{logit}_1) + \exp(\text{logit}_2)}$

where (\text{logit}_1) and (\text{logit}_2) are the unnormalized log-probabilities of the tokens "1" and "2" respectively, according to the annotator LLM. This is the standard softmax over two tokens. The preference probability for Response 2 is simply (1 - P(\text{prefer response 1})).

What this computes: The annotator's belief, as expressed through its token-level log-probabilities, about which response better satisfies the criteria specified in the prompt. The output is a scalar between 0 and 1 representing the probability that response 1 is preferred.

Why this probabilistic extraction? Using log-probabilities over a constrained token set ("1" and "2") is more reliable than parsing free-text output because it avoids parsing errors, provides a natural measure of certainty (a score near 0.5 indicates the annotator is uncertain), and is more reproducible. It also aligns with the soft-label spirit of the annotation — the annotator might have nuanced preferences that a hard binary choice loses.

Handling position bias. Lee et al. (2023) observed that LLM annotators exhibit position bias: the order in which responses are presented (Response 1 vs. Response 2) can affect the preference judgment. To mitigate this, OAIF computes the preference score twice — once with (({\bm{y}}^1, {\bm{y}}^2)) in positions (1, 2) and once with the order reversed (({\bm{y}}^2, {\bm{y}}^1)) in positions (1, 2) — and averages the two probabilities:

$P_{\text{avg}}(\text{prefer }{\bm{y}}^1) = 0.5 \cdot P_{\text{order1}}(\text{prefer }{\bm{y}}^1) + 0.5 \cdot P_{\text{order2}}(\text{prefer }{\bm{y}}^1)$

where (P_{\text{order1}}) is the probability from the original ordering and (P_{\text{order2}}) is from the reversed ordering. If (P_{\text{avg}}(\text{prefer }{\bm{y}}^1) > 0.5), then ({\bm{y}}^+ = {\bm{y}}^1) and ({\bm{y}}^- = {\bm{y}}^2); otherwise, ({\bm{y}}^+ = {\bm{y}}^2) and ({\bm{y}}^- = {\bm{y}}^1).

Why this averaging? Position bias is a known failure mode in LLM evaluation — models sometimes default to preferring whichever response is listed first regardless of quality. Averaging over both orderings cancels this bias in expectation, making the preference judgment depend only on the content of the responses. The cost is 2× the annotation compute per training step (two forward passes through the annotator instead of one), which is manageable because the annotator is fixed (no gradient computation) and the cost is in inference FLOPs only.

The annotating LLM is frozen. In all OAIF experiments in Section 4, the annotating LLM is a separate, pre-trained model that is never updated during OAIF training. It is queried purely for inference. The paper uses PaLM 2-L as the annotator in the main experiments, with ablations using PaLM 2-XS and PaLM 2-S in Section 4.5. This frozen nature is crucial: if the annotator were updated, the preference signal would shift during training (making the learning target non-stationary), and the computational cost would increase dramatically.

How the annotator prompt differs from the evaluation prompt. A subtle but important detail: the paper uses PaLM 2-L for online feedback during training, but uses Gemini Pro for automatic evaluation at test time (Section 4.1). This is an intentional design choice to reduce the risk of overfitting to the annotator: if the model is trained on preferences from PaLM 2-L and then evaluated by PaLM 2-L, there's a risk of reward hacking — the policy might learn to exploit quirks of the annotator rather than genuinely improving. Using a different, stronger model (Gemini Pro) for evaluation provides an independent assessment. Appendix C validates that Gemini Pro's judgments align with human preferences at a similar level to PaLM 2-L (alignment accuracy of 70.21% vs. 70.72%), supporting this cross-model evaluation strategy.

The annotator prompt is not the same as the training objective prompt. The annotator is prompted with instructions about what makes a good response (e.g., "coherent, accurate, good coverage" for summarization), but the DAP loss does not "know" these criteria — it only knows which response the annotator preferred. The annotator's criteria are embedded in the data labeling process, not in the optimization objective. This separation is why the preference function can be changed by modifying the annotator prompt (Section 4.6, text-controllability) without changing the DAP loss at all — the loss just sees a different set of (({\bm{y}}^+, {\bm{y}}^-)) pairs.

---

Loss Functions: DPO, IPO, and SLiC in OAIF

OAIF does not modify the DAP loss functions. It only changes the data source. However, understanding how these losses operate within the OAIF loop — and why the same mathematical forms work for both offline and online settings — is essential.

The DPO loss (Equation 1 in the paper):

$-\log \sigma\left( \beta \log \frac{\pi_{\bm{\theta}}({\bm{y}}^+|{\bm{x}})\pi_{{\bm{\theta}}^0}({\bm{y}}^-|{\bm{x}})}{\pi_{{\bm{\theta}}^0}({\bm{y}}^+|{\bm{x}})\pi_{\bm{\theta}}({\bm{y}}^-|{\bm{x}})} \right)$

where (\pi_{\bm{\theta}}) is the current policy (the model being trained), (\pi_{{\bm{\theta}}^0}) is the reference policy (the SFT checkpoint, frozen), (\beta) is a temperature-like hyperparameter controlling how strongly the preference signal drives the policy away from the reference, and (\sigma) is the logistic (sigmoid) function.

What the term inside the sigmoid computes: The log-ratio (\log \frac{\pi_{\bm{\theta}}({\bm{y}}^+|{\bm{x}})}{\pi_{{\bm{\theta}}^0}({\bm{y}}^+|{\bm{x}})}) is the log-probability improvement of the preferred response under the current policy relative to the reference policy. Similarly, (\log \frac{\pi_{\bm{\theta}}({\bm{y}}^-|{\bm{x}})}{\pi_{{\bm{\theta}}^0}({\bm{y}}^-|{\bm{x}})}) is the same for the dispreferred response. The difference of these two terms measures how much more the current policy favors the preferred response over the dispreferred response, compared to the reference policy. Multiplying by (\beta) scales this difference. Passing the result through (-\log \sigma(\cdot)) converts it into a binary-cross-entropy-like loss: the loss is low when the log-ratio difference is large and positive (meaning the current policy strongly prefers ({\bm{y}}^+) over ({\bm{y}}^-) relative to the reference), and high when it is negative (the current policy prefers the dispreferred response).

What changes in OAIF: In offline DPO, (({\bm{y}}^+, {\bm{y}}^-)) are sampled from (\rho) and are fixed throughout training. The loss sees the same preference pairs every epoch. In OAIF, (({\bm{y}}^+, {\bm{y}}^-)) are sampled from (\pi_{{\bm{\theta}}^t}) and annotated by the LLM at each step (t). The loss sees fresh, on-policy preference pairs at every step. Mathematically, the loss function is identical — the only difference is the distribution that (({\bm{y}}^+, {\bm{y}}^-)) is drawn from. This is why the paper can claim that OAIF works "with any differentiable DAP loss function" — it is purely a data pipeline change.

Why this matters: The DPO loss is derived under the assumption that the preference data comes from an underlying Bradley-Terry model of human preferences. If the preference pairs are off-policy (from (\rho)), the model is being optimized under a distribution that does not match its own output distribution, which can lead to overfitting (as seen in Figure 3). By making the preference pairs on-policy, the optimization target better reflects the true distribution of responses the policy will produce, reducing the mismatch between training and evaluation.

The IPO loss (Equation 2):

$\left( \log\left( \frac{\pi_{\bm{\theta}}({\bm{y}}^+|{\bm{x}})\pi_{{\bm{\theta}}^0}({\bm{y}}^-|{\bm{x}})}{\pi_{\bm{\theta}}({\bm{y}}^-|{\bm{x}})\pi_{{\bm{\theta}}^0}({\bm{y}}^+|{\bm{x}})} \right) - \frac{1}{2\beta} \right)^2$

What this computes: The squared difference between the log-ratio (same as in DPO) and a target value (1/(2\beta)). Unlike DPO, which uses a sigmoid cross-entropy formulation, IPO uses a squared-error formulation: the loss pushes the log-ratio to be exactly (1/(2\beta)) rather than to be arbitrarily large. This means IPO has a built-in regularization against extreme probability ratios — even on a perfectly preferred response, the loss does not encourage (\pi_{\bm{\theta}}({\bm{y}}^+ | {\bm{x}})) to become arbitrarily large relative to (\pi_{{\bm{\theta}}^0}({\bm{y}}^+ | {\bm{x}})).

Why this form: The squared loss avoids the saturation problem of the sigmoid in DPO. In DPO, when the log-ratio is already very large, the sigmoid is saturated (close to 1) and the gradient is near zero, meaning the model stops learning from that example even if further improvement is possible. IPO's quadratic formulation continues to provide a gradient as long as the log-ratio deviates from the target, potentially enabling more sustained learning. The target (1/(2\beta)) sets a fixed "aspiration level" for how much the policy should deviate from the reference.

The SLiC loss (Equation 3):

$\max\left( 0, 1 - \beta \log\left( \frac{\pi_{\bm{\theta}}({\bm{y}}^+|{\bm{x}})\pi_{{\bm{\theta}}^0}({\bm{y}}^-|{\bm{x}})}{\pi_{\bm{\theta}}({\bm{y}}^-|{\bm{x}})\pi_{{\bm{\theta}}^0}({\bm{y}}^+|{\bm{x}})} \right) \right)$

What this computes: A hinge loss: if (\beta \log(\text{ratio}) \geq 1), the loss is zero — the model has achieved a sufficient margin between the preferred and dispreferred response. If the margin is less than 1, the loss is positive and linear in the shortfall. This means SLiC only penalizes the model when the preference margin is insufficient; once the margin is "good enough," the loss stops providing a gradient.

Why this form: The hinge loss provides a different inductive bias than both DPO and IPO. DPO encourages ever-larger margins (though with diminishing gradients due to saturation). IPO encourages a specific target margin. SLiC encourages a minimum margin and then stops. This "satisficing" behavior may be more robust to noise: if the annotator is uncertain or makes mistakes, SLiC will not over-optimize to achieve an extremely large margin, which could help prevent the overfitting observed in offline DPO. However, the paper does not specifically ablate this property — SLiC is included primarily to demonstrate the generality of OAIF across loss functions.

Hyperparameter settings. The paper uses the following (\beta) values based on preliminary experiments (Section 4.1):

• DPO: (\beta = 0.1)
• IPO: (\beta = 1.0)
• SLiC: (\beta = 0.002)

These values are not explicitly justified in terms of their effect, but the large differences (0.1 vs. 1.0 vs. 0.002) reflect the different mathematical roles (\beta) plays in each loss: in DPO, (\beta) scales the log-ratio inside a sigmoid (small (\beta) means smaller gradients); in IPO, (\beta) sets the target margin (larger (\beta) means smaller target); in SLiC, (\beta) scales the log-ratio inside a hinge (smaller (\beta) means a larger required ratio to reach zero loss).

---

Training Configuration and Hyperparameters

This sub-section details the training setup used for all OAIF experiments in Section 4, as specified in Section 4.1 and the appendices.

Model architecture and initialization. All policy models ((\pi_{{\bm{\theta}}^0}) and the policy being aligned) are PaLM 2-XS (Extra Small). The SFT baseline (\pi_{{\bm{\theta}}^0}) is obtained by supervised finetuning PaLM 2-XS on the target task's training data. For the annotating LLM, the paper uses PaLM 2-L (Large) in the main experiments, with ablations using PaLM 2-XS and PaLM 2-S in Section 4.5. For evaluation, Gemini Pro is used as the automatic judge to avoid overfitting to the annotator's preferences.

Optimizer and learning rate schedule. The paper uses Adafactor (Shazeer and Stern, 2018) as the optimizer, a memory-efficient variant of Adam designed for large models. The specific configuration:

• Learning rate: (5 \times 10^{-7})
• Batch size: 128
• Warm-up period: 150 steps

The choice of Adafactor over standard AdamW is practical: Adafactor uses less memory by factorizing the second-moment accumulator, which is important when training large language models where optimizer states can dominate memory usage. The learning rate of (5 \times 10^{-7}) is notably small, reflecting the caution needed when fine-tuning LLMs — larger learning rates can cause catastrophic forgetting of pre-training knowledge.

Response sampling configuration. During training, responses are sampled from (\pi_{{\bm{\theta}}^t}) with a temperature of 0.9. This temperature is deliberately high to ensure diversity between the two sampled responses (({\bm{y}}^1, {\bm{y}}^2)). If the temperature were too low, both responses would be near-deterministic and nearly identical, making it impossible for the annotator to provide a meaningful preference — the loss gradient would be near zero because the log-ratio of the two responses would be approximately identical. The temperature of 0.9 provides enough stochasticity for the annotator to express a genuine preference while still keeping responses reasonable.

The number of training steps is not specified as a fixed value but is determined by monitoring the development set win rate against the SFT baseline, judged by Gemini Pro. The paper selects the "best performing online and offline DPO models according to both manual inspection and their development set win rate" (Section 4.2). In Figure 3, training appears to run for about 6,000–8,000 steps on TL;DR, with performance continuing to improve for online DPO throughout this range.

Comparison setup to ensure fairness. The paper adopts several measures to ensure fair comparison between online and offline methods:

• Both online and offline DAP methods use the same base model ((\pi_{{\bm{\theta}}^0})) and the same optimizer configuration.
• Online DAP uses fresh preference pairs generated at each step; offline DAP uses the pre-collected preference dataset (\mathbb{D}).
• RLAIF uses PaLM 2-L as the AI feedback model for RM training, making the AI feedback source identical between OAIF and RLAIF (both use PaLM 2-L for labeling).
• RLHF uses a reward model trained on the pre-collected human preference dataset.
• For evaluation, all models are judged by the same evaluator (Gemini Pro for automatic evaluation, human raters for human evaluation) using the same prompts.

The data budget. A key practical consideration: OAIF generates preference pairs on the fly, which means it can, in principle, see an unlimited number of unique preference annotations. Offline DAP is limited to the pre-collected dataset size. The paper does not explicitly compare OAIF at different data budgets (e.g., comparing online DPO with 100K annotations vs. offline DPO with the same 100K annotations), but the implicit advantage is that OAIF never runs out of novel training data — at each step, it can sample new responses and get new annotations, preventing the overfitting that arises from repeatedly training on the same static dataset.

---

Text-Controllability of the Annotator

Section 4.6 demonstrates a unique capability of OAIF that is not possible with RM-based methods: the LLM annotator's preference criteria can be changed simply by modifying the prompt text, without any retraining of the annotator or the loss function.

The mechanism. The annotator is prompted with natural language instructions describing what makes a response preferred. By modifying these instructions, the preference function changes — and since the DAP loss simply learns from the annotator's choices, the policy adapts to the new criteria automatically. The paper demonstrates this using response length as a controllable attribute on the Helpfulness task.

Three annotator configurations. The paper trains three versions of online DPO, differing only in the prompt given to the LLM annotator:

1. "Helpful only" (Table 6): The standard helpfulness prompt, which instructs the annotator to prefer responses that are "thoughtful, honest, and reasonable." No mention of length.

2. "Helpful and short" (Table 8, top): The standard prompt augmented with the instruction: "When the quality of two responses is similar, the shorter one should always be preferred."

3. "Helpful and very short" (Table 8, bottom): A more aggressive prompt that emphasizes conciseness throughout, instructing the annotator to prefer responses that help "in the shortest way" and to prefer "helpful and concise" responses.

What the training procedure sees differently. From the perspective of the OAIF training loop (Algorithm 1), nothing changes except the prompt sent to the annotator. The DPO loss is identical; the optimizer hyperparameters are identical; the policy initialization is identical; the training prompts (\mathbb{D}_{\mathcal{X}}) are identical. The only thing that differs is which response the annotator selects as ({\bm{y}}^+) — because the annotator's criteria have changed. This means the (({\bm{y}}^+, {\bm{y}}^-)) pairs fed into the DPO loss are systematically different: the annotator now prefers shorter responses when quality is comparable, and this preference propagates into the policy through standard DPO training.

Results. Figure 6(a) shows that the average response length drops from approximately 120 tokens ("helpful only") to approximately 90 tokens ("helpful and short") to approximately 40 tokens ("helpful and very short"). This is a direct behavioral change driven entirely by prompt modification. Figure 6(b) shows that all three models still improve over the SFT baseline in helpfulness (win rates greater than 50%), though the "very short" model is less helpful than the "helpful only" model, as judged by Gemini Pro. Human evaluation confirms a quality score drop from 4.08 to 3.26 (on a 1–5 scale), but all OAIF variants still outperform the SFT baseline (3.19).

Why this is a practical advantage. In RLHF or RLAIF, achieving the same effect would require re-annotating the preference dataset with the new length criteria (expensive), re-training the reward model (computationally costly), and then re-running RL (time-consuming). With OAIF, the change requires only editing a text prompt — no data re-annotation, no model retraining, no additional compute. This makes OAIF inherently more flexible for adapting to changing alignment desiderata, which the paper argues is important because "human expectations vary greatly across regions and cultures, and may evolve over time" (Section 4.6).

The generalization of this capability. While the paper demonstrates controllability only for response length (a simple, easily measurable attribute), they argue that the same principle extends to "more qualitative desiderata" such as "helpfulness and impartiality" (Section 5, "Qualitative preference annotation from LLMs"). The key requirement is that the desired behavior can be described in natural language in a way that the annotator LLM can reliably apply when comparing responses. This connects to the broader capability of LLMs as zero-shot preference judges — if an LLM can understand and apply a preference criterion when prompted, OAIF can optimize a policy toward that criterion without any task-specific reward modeling.

---

What OAIF Does NOT Change: The DAP Loss Landscape

A crucial point for understanding OAIF's relationship to existing DAP methods: OAIF does not alter the optimization landscape of DPO, IPO, or SLiC. The loss functions, their gradients with respect to (\bm{\theta}), and the role of the reference policy (\pi_{{\bm{\theta}}^0}) are all unchanged. OAIF operates entirely on the data generation side — it changes which (({\bm{y}}^+, {\bm{y}}^-)) pairs the loss sees, but the loss itself remains mathematically identical to its offline counterpart.

This means that theoretical properties of DAP methods (e.g., DPO's equivalence to RLHF under the Bradley-Terry model, IPO's bounded gradient properties) transfer directly to their OAIF-online versions. The only thing that changes is the distribution of ({({\bm{x}}, {\bm{y}}^+, {\bm{y}}^-)}) triples: offline DAP sees a fixed set of triples drawn from (\rho); OAIF sees an evolving stream of triples drawn from (\pi_{{\bm{\theta}}^t}) with labels from the annotator LLM. This distributed shift — from static off-policy to dynamic on-policy — is the entire contribution of the method.

Why this matters for interpretation: Any performance difference between online and offline DAP, holding the loss function fixed, can be attributed to the data distribution change rather than to a change in the optimization algorithm. This clean experimental separation is what allows the paper to attribute the observed gains (Figure 3, Tables 2–3) specifically to the online, on-policy nature of OAIF rather than to any confounding algorithmic factor.

4. Key Insights and Innovations

Innovation 1: Reframing DAP's Limitations as a Data Distribution Problem, Not an Algorithm Problem

The paper's most fundamental conceptual move is its diagnostic framing of why direct alignment methods underperform their potential. Prior work implicitly treated DPO and related methods as algorithmically complete solutions—the loss function was derived correctly, the optimization was stable, and if performance lagged behind RLHF, the assumption was that the algorithmic simplification (removing the reward model, removing policy gradients) carried an inherent cost. OAIF challenges this assumption by arguing that the problem is not in the DAP loss functions themselves but in the data distribution they consume.

Before OAIF, the field's understanding of DAP's limitations was fragmented. Some papers observed that DPO overfits (Gao et al., 2023), others noted that offline preference data creates distribution shift (Xu et al., 2023; Liu et al., 2023), and still others explored iterative or online variants using reward models. But the conceptual distinction between offline feedback (the timing of annotation) and off-policy sampling (which model generated the responses) was not crisply separated. OAIF disentangles these into two orthogonal axes (Section 2, Appendix A) and argues that standard DAP methods are both offline and off-policy, while RLHF is online and on-policy—and that addressing both simultaneously is what matters.

This reframing is significant because it shifts the research question from "how can we make DAP better?" to "what would DAP look like if it had the data properties of RLHF?" The answer, OAIF shows, is that DAP methods already work well when given the right data. The DPO loss doesn't need to be changed; it needs to be fed on-policy, online preference pairs. This is a diagnostic insight, not just a new method: it explains why offline DPO overfits (Figure 3, the sharp drop at step 3,500) and why making it online eliminates the overfitting (the blue curve keeps improving). The evidence in Table 2—online DPO achieving 63.74% win rate and 3.95 quality vs. offline DPO's 7.69% and 3.46—is not just a performance gain; it's a validation of the diagnostic hypothesis that the data distribution, not the algorithm, was the binding constraint.

This framing also provides a unified explanation for conflicting results in the literature. The paper's observation that "the distribution shift problem still exists when training the RM" (Section 2, "RM-based online feedback") explains why methods like Iterative DPO and RSO, which use an RM for pseudo-labeling, partially help but fall short: they address the on-policy aspect (the RM scores fresh samples) but the RM itself is trained on off-policy data from ρ, so the shift is merely displaced, not eliminated. The experimental finding that OAIF with an LLM annotator outperforms the same DPO loss with an RM annotator (<30% win rate, Section 4.4) directly supports this diagnostic claim.

Innovation 2: Demonstrating That an LLM Can Serve as an Online Preference Annotator for Policy Training—Without Any Training Itself

Using LLMs to evaluate or label text is not new. RLAIF (Lee et al., 2023) had already shown that LLM annotators can provide preference labels that correlate with human judgments, and Constitutional AI (Bai et al., 2022b) used LLM feedback as a training signal. What is innovative in OAIF is the demonstration that an LLM annotator can provide online feedback during the training loop of a policy being aligned, at scale, without being trained or fine-tuned for the task, and without suffering from the distribution-shift problem that plagues trained reward models.

This is a fundamentally different way of thinking about the annotator's role. In RLAIF, the LLM labels a fixed dataset ahead of time—the annotation is offline, even though the annotator is an LLM. In RLHF, the reward model is trained on human labels and then used online—it's a learned function that provides online feedback. OAIF does something neither does: the LLM annotator provides online feedback without being trained to do so. Its preference function emerges entirely from the prompt instructions and its pre-trained knowledge. This means the annotator avoids the train-test distribution shift that afflicts reward models: an RM trained on responses from ρ will struggle to evaluate responses from (\pi_{{\bm{\theta}}^t}), but a pre-trained LLM annotator has no such training distribution—it evaluates text based on its general language understanding, which transfers across the policy's evolving output distribution.

The significance of this beyond OAIF's immediate results is that it establishes LLM annotators as a different category of feedback source than trained reward models—one that is simultaneously online-capable, distribution-shift-robust, and zero-shot flexible. This has implications for alignment research beyond DAP methods: any alignment algorithm that needs preference feedback could potentially replace a trained RM with a prompted LLM annotator, gaining online capability and distributional robustness at the cost of annotation quality (which depends on the annotator's size and capability, as Section 4.5 shows).

The paper provides systematic evidence for this claim. The ablation in Section 4.4 comparing "online DPO with OAIF" vs. "online DPO with the same RM used for RLAIF" isolates the annotator type as the only variable—both use the DPO loss, both are online and on-policy, but one uses an LLM annotator and the other uses a trained RM. The LLM annotator version substantially outperforms the RM version, confirming that the LLM's distributional robustness is a real effect, not an artifact of the online/on-policy setup. Furthermore, the annotator size ablation in Section 4.5 (Figure 5) shows that even a same-size annotator (PaLM 2-XS labeling PaLM 2-XS responses) provides useful feedback, though larger annotators produce stronger results—this maps out the annotator quality frontier and suggests that OAIF's effectiveness scales with annotator capability, an insight with practical deployment implications.

Innovation 3: Prompt-Controllable Alignment Objectives via the Annotator—Changing What the Model Optimizes for Without Retraining Any Component

Perhaps the most practically distinctive insight in this paper is that the LLM annotator's preference function—and therefore the alignment objective of the entire training process—can be changed at will by editing a text prompt, with no retraining of the annotator, no re-annotation of data, and no modification to the DAP loss function or optimizer. The paper demonstrates this using response length as a controlled testbed (Section 4.6), but the conceptual contribution is broader: it establishes that prompt-based AI feedback creates alignment systems where the objective function is soft, editable, and composable.

Prior alignment methods bake the objective into fixed artifacts. In RLHF, the reward model encodes a specific preference function learned from a specific dataset; changing the objective requires re-annotating data and retraining the RM. In offline DAP, the preference dataset itself encodes the objective; changing it similarly requires re-collection. Even in RLAIF, the LLM annotator's preferences are captured in a static labeled dataset before training. In all these cases, the alignment objective is "compiled" into a fixed artifact before policy optimization begins.

OAIF breaks this pattern because the annotator is queried at training time with a prompt that defines the evaluation criteria. The annotator's output is a function of both the responses and the prompt, and the prompt can be modified between training runs (or even during training). The objective is not compiled—it is interpreted at query time. This means the alignment target becomes a soft, natural-language-specified constraint that can be refined, combined, or replaced by editing text, not by re-running expensive data pipelines.

The experimental demonstration using length control (Figures 6a, 6b) is elegant because it isolates this capability. Three versions of online DPO are trained identically except for the annotator prompt: "helpful," "helpful and short," and "helpful and very short." The resulting policies produce responses of ~120, ~90, and ~40 tokens respectively, while all maintaining improvement over the SFT baseline. The user gets a dial for response length that requires no additional training—just a prompt change. This is significant not because length control is inherently important, but because it demonstrates a general principle: any preference that can be described in natural language and reliably applied by the annotator LLM can become an alignment objective without engineering effort beyond writing the prompt.

The authors correctly identify in Section 5 that this extends naturally to qualitative desiderata like impartiality, empathy, or cultural sensitivity—objectives that are notoriously hard to encode in reward functions or preference datasets because they are nuanced, context-dependent, and culturally variable. The prompt-controllability of OAIF offers a path toward alignment objectives that are as flexible as natural language itself, which is a qualitatively different capability than anything RM-based methods can offer at comparable cost.

Innovation 4: Establishing That Online, On-Policy Data Is Sufficient to Close the Gap Between DAP and RLHF—Without Requiring RLHF's Complexity

The paper's experimental program, culminating in the 4-way human comparison in Figure 4a, makes a strong empirical claim: when DAP methods are given online, on-policy data (via OAIF), they can match or exceed the performance of RLHF and RLAIF, while retaining DAP's simplicity (no separate reward model, no policy gradients, no value function). Online DPO is preferred over RLHF, RLAIF, and the SFT baseline in 58% of the time in the 4-way comparison on TL;DR. Combined with the consistent ~66% average win rate of online DAP methods over their offline counterparts across DPO, IPO, and SLiC (Tables 2–3), the evidence suggests that the online/on-policy property—not the choice of optimization algorithm—is the primary determinant of alignment effectiveness.

This is a significant result because it addresses a standing question in the alignment literature: is RLHF's advantage over DAP methods due to its algorithmic properties (policy gradients enabling better exploration, the value function reducing variance, the KL penalty being applied differently) or due to its data properties (online, on-policy)? The paper's experiments cannot fully disentangle these, but the fact that OAIF—which changes nothing about the DAP loss functions except the data they consume—achieves parity with RLHF strongly suggests that data properties dominate. If algorithmic differences were the primary driver, we would expect RLHF to maintain an advantage even when DAP methods are given the same data properties. The 58% preference for online DPO over RLHF in Figure 4a suggests the opposite: when given the data advantages of RLHF, DAP methods may even be preferable, perhaps because they avoid the optimization instabilities and reward hacking that can arise from RL training.

This finding has practical implications for practitioners deciding between alignment approaches. RLHF pipelines are complex: they require training a reward model, implementing policy gradient optimization (often with PPO), managing a value function, and tuning numerous RL-specific hyperparameters. If the same or better results can be achieved by simply running DPO with an LLM annotator providing online feedback, the engineering and computational savings are substantial. The paper's finding that OAIF outperforms RLAIF by a significant margin (online DPO preferred 58% of the time vs. RLAIF in the 4-way comparison) further narrows the practical recommendation: if you have access to a capable LLM annotator, online DPO with OAIF dominates both offline DPO and RLAIF, and is competitive with or better than RLHF, while being simpler than both RLAIF and RLHF.

A crucial caveat, which the paper acknowledges implicitly through its experimental design, is that this finding is demonstrated with PaLM 2 models at XS/L scale and may not transfer to all model scales or families. However, the conceptual contribution stands: the paper provides strong evidence that the data regime is the primary determinant of alignment method effectiveness, with algorithm choice playing a secondary role, at least within the family of preference-based alignment methods. This refocuses the research agenda from developing new loss functions to developing better feedback mechanisms—exactly the direction OAIF pursues.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses three tasks: TL;DR (Stiennon et al., 2020) for summarization quality, Anthropic Helpfulness (Bai et al., 2022a) for producing useful and honest responses, and Anthropic Harmlessness (Bai et al., 2022a) for avoiding harmful content. For each task, the prompt dataset $\mathbb{D}_{\mathcal{X}}$ is constructed by extracting only the input prompts from the original preference dataset $\mathbb{D}$, discarding the pre-existing response pairs — so OAIF receives no pre-collected preference data. The TL;DR task is the primary testbed; Helpfulness and Harmlessness serve as generalization benchmarks.

• Base model(s). All aligned policy models are initialized from PaLM 2-XS (Extra Small) (Anil et al., 2023), first supervised-fine-tuned on the target task to produce the SFT baseline $\pi_{{\bm{\theta}}^0}$. The annotating LLM is PaLM 2-L (Large) in the main experiments, with PaLM 2-XS and PaLM 2-S used in annotator-size ablations (Section 4.5). Automatic evaluation uses Gemini Pro (Gemini Team et al., 2023) to reduce overfitting risk — the evaluator differs from the training annotator. The paper argues PaLM 2 models are "representative of the capabilities of many contemporary LLMs" (Section 1), though this claim is limited to a single model family.

• Metrics. The primary metric is win rate: the fraction of evaluations where one model's response is preferred over another, reported as a percentage. For automatic evaluation, win rates are computed using Gemini Pro with the same pairwise prompting scheme as annotation, extracting preference scores from log-probabilities of "1" vs. "2" tokens and averaging over both response orderings to cancel position bias. For human evaluation, three raters independently score response quality on a 1–5 scale (5 = highest) and pick the best response; the paper reports win/tie/loss rates between models, average quality scores, and in Figure 4b, quality scores conditioned on response length (bucketed into six length groups). Appendix C validates Gemini Pro as an automatic evaluator by measuring alignment accuracy against human judgments: Gemini Pro achieves 70.21% average alignment accuracy across TL;DR, Helpfulness, and Harmlessness, comparable to PaLM 2-L's 70.72% (Table 4).

• Baselines. The paper compares against six distinct baselines:

  • SFT baseline ($\pi_{{\bm{\theta}}^0}$): the PaLM 2-XS model after supervised fine-tuning but before any preference-based alignment.
  • Offline DPO (Rafailov et al., 2023): standard DPO trained on the pre-collected, static preference dataset $\mathbb{D}$ from $\rho$ (the original data-generating policy, distinct from PaLM 2-XS).
  • Offline IPO (Azar et al., 2023): IPO trained on the same static $\mathbb{D}$.
  • Offline SLiC (Zhao et al., 2023): SLiC trained on the same static $\mathbb{D}$.
  • RLAIF (Lee et al., 2023): RL from AI feedback, where PaLM 2-L provides preference labels to train a reward model, which is then used in the standard RLHF pipeline (policy gradients with a value function). The AI feedback model (PaLM 2-L) is identical to OAIF's annotator, making this a controlled comparison.
  • RLHF (Stiennon et al., 2020): standard RL from human feedback, using a reward model trained on the pre-collected human preference dataset.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time; it measures computation indirectly through number of training steps (e.g., tracking win rate from 0 to ~8,000 steps in Figure 3) and through annotator model size (different PaLM 2 scales in Section 4.5). All online and offline methods are compared at the same number of training steps. OAIF's annotation cost — two forward passes per training step through the annotator LLM (for the two response orderings) — is not explicitly accounted for in any comparison, nor is it compared to the cost of loading pre-collected preference data. OAIF generates preference pairs on-the-fly, meaning it can see unlimited unique annotations; offline methods are bounded by dataset size. This asymmetry is acknowledged implicitly but not experimentally controlled (e.g., no comparison at equal annotation budget).

• Cross-validation / statistical protocol. No formal cross-validation or statistical testing (confidence intervals, significance tests) is reported. Model selection for human evaluation is based on "both manual inspection and their development set win rate against the SFT baseline by Gemini Pro" (Section 4.2). Human evaluation uses only three raters; inter-rater agreement is not reported. The small rater pool and absence of statistical rigor mean that the reported win/loss/tie rates and quality scores should be interpreted as point estimates without quantified uncertainty.

---

Main Quantitative Results

Online vs. Offline DPO: Training Dynamics and Overfitting

The paper's central diagnostic plot is Figure 3, which tracks the win rate of online DPO and offline DPO against the SFT baseline on TL;DR over training, judged by Gemini Pro. The headline result: offline DPO overfits rapidly and catastrophically, while online DPO continues to improve.

At the start of training, both methods improve over the SFT baseline. However, around training step 3,500, offline DPO's win rate drops sharply — the red curve in Figure 3 shows a clear peak followed by a substantial decline. This is the paper's key empirical evidence for the offline data problem: the policy overfits to the static, off-policy preference pairs in $\mathbb{D}$, learning to exploit dataset-specific patterns rather than genuinely improving generation quality. By contrast, online DPO's win rate (blue curve) continues to increase throughout training, surpassing offline DPO after approximately 4,000 steps. The continuous improvement without overfitting is attributed to the fresh, on-policy preference pairs generated at each step — the model never sees the same data twice in exactly the same form, preventing memorization of fixed patterns.

This result is corroborated with a different evaluator (PaLM 2-L instead of Gemini Pro) in Appendix D (Figure 9), where the same pattern holds: offline DPO peaks and declines; online DPO keeps improving. The cross-evaluator consistency strengthens the claim that the overfitting is a real phenomenon, not an artifact of evaluator bias.

Human Evaluation: Online DPO vs. Offline DPO Across Tasks

Table 2 presents the paper's strongest evidence: side-by-side human evaluation comparing online DPO and offline DPO on TL;DR, Helpfulness, and Harmlessness. The results are dramatic and consistent:

• TL;DR: Online DPO achieves a 63.74% win rate over offline DPO (with 28.57% ties, only 7.69% losses), and a quality score of 3.95 vs. 3.46 (on a 1–5 scale).
• Helpfulness: Online DPO wins 58.60% (21.20% ties, 20.20% losses), quality score 4.08 vs. 3.44.
• Harmlessness: Online DPO wins 60.26% (35.90% ties, 3.84% losses), quality score 4.41 vs. 3.57.

In all three tasks, online DPO's win rate exceeds 58% and its loss rate is at or below 20.20%. The quality score advantage of online over offline DPO ranges from 0.49 to 0.84 points. The Harmlessness task shows the most extreme asymmetry (60.26% win, 3.84% loss), suggesting offline DPO is particularly vulnerable to overfitting patterns that compromise harmlessness. The paper presents these as validation that "using the offline feedback and off-policy generations in a pre-collected preference dataset $\mathbb{D}$ can be detrimental for LLM alignment" (Section 4.2).

Generality Across DAP Loss Functions: Online vs. Offline IPO and SLiC

Table 3 extends the online-vs-offline comparison to IPO and SLiC on TL;DR, again with human evaluation:

• IPO: Online IPO achieves a 64.39% win rate over offline IPO (29.17% ties, only 6.44% losses), quality score 3.95 vs. 3.49.
• SLiC: Online SLiC achieves a 71.05% win rate over offline SLiC (22.69% ties, 6.26% losses), quality score 3.88 vs. 3.41.

The win rates across all three DAP methods (DPO, IPO, SLiC) average approximately 66%, with SLiC showing the largest gap (71.05%) and DPO the smallest (63.74%). The paper emphasizes that "the consistent ineffectiveness of offline DAP methods confirms that the existence of the offline and off-policy issue in DAP methods... greatly hinders the performance" and that "the consistent superiority of online DAP methods via OAIF against their offline counterparts demonstrates that OAIF is a general framework effectively addressing these challenges" (Section 4.3).

A noteworthy pattern: the tie rates are substantial across all comparisons (22.69% to 35.90%), indicating that in many cases, online and offline methods produce responses of comparable quality. The asymmetry comes from the win portion being 8× to 10× larger than the loss portion — when there is a discernible quality difference, it almost always favors the online method.

Comparison Against RLHF and RLAIF: The 4-Way Comparison

Figure 4a presents a 4-way human comparison on TL;DR, where raters choose among responses from online DPO, offline DPO, RLAIF, and RLHF. Online DPO is preferred 58.00% of the time — meaning in a majority of cases, raters selected the online DPO response as the best among the four. The paper emphasizes that the RM used in RLAIF and RLHF is not updated during policy training, so its assessment ability may not generalize as $\pi_{{\bm{\theta}}^t}$ evolves (Section 4.4).

An additional experiment mentioned in Section 4.4 (not in a figure): the authors trained an online DPO variant using the same RM from RLAIF (instead of the LLM annotator) to provide preference feedback. This RM-based online DPO outperforms RLAIF but "significantly underperforms online DPO with OAIF, with a win rate of <30% judged by Gemini Pro." This is a critical ablation: it isolates the annotator type (LLM vs. trained RM) while holding the training procedure (online, on-policy DPO) constant, demonstrating that the LLM annotator specifically provides better feedback than the RM — consistent with the hypothesis that the RM suffers from its own distribution-shift problem.

Length-Controlled Quality Analysis

Figure 4b addresses a potential confound: length bias. Both LLM and human evaluators often prefer longer responses (Singhal et al., 2023), and OAIF tends to produce longer responses. If OAIF's apparent advantage were solely due to length bias, the quality advantage should disappear when controlling for length.

Figure 4b plots average human quality scores for responses bucketed into six length groups. At every length bucket, online DPO achieves higher quality scores than offline DPO, RLAIF, and RLHF. The error bars (standard errors) show generally non-overlapping intervals, particularly at shorter lengths. This demonstrates that OAIF's quality advantage is not an artifact of length bias — online DPO produces genuinely better responses at fixed length. The paper uses this to validate that "online DPO with OAIF provides responses of higher quality than the other methods at fixed length" (Section 4.4).

Annotator Size Ablation

Section 4.5 (Figure 5) examines how the size of the annotating LLM affects online DPO's performance on TL;DR, comparing PaLM 2-XS, PaLM 2-S, and PaLM 2-L annotators. The win rates against four baselines (SFT, offline DPO, RLAIF, RLHF) are shown, all judged by Gemini Pro.

The clear pattern: annotator size matters significantly. PaLM 2-L annotator produces the highest win rates across all baseline comparisons, followed by PaLM 2-S, then PaLM 2-XS. However, even the smallest annotator (PaLM 2-XS, same size as the policy being trained) produces online DPO that outperforms the SFT baseline and is competitive with offline DPO and RLHF. Specifically, OAIF with PaLM 2-XS "achieves comparable performance to RLHF, although the latter learns from human feedback" (Section 4.5). Human evaluation confirms this: OAIF with PaLM 2-XS achieves a quality score of 3.41, slightly better than RLHF's 3.38 and comparable to offline DPO's 3.46.

The finding that a same-size annotator provides useful feedback is practically important — it means OAIF does not strictly require a stronger model for annotation; the policy can, in principle, benefit from its own feedback (though larger annotators provide stronger signals).

Prompt-Controllability: Response Length as a Testbed

Section 4.6 (Figure 6) demonstrates OAIF's text-controllability. Three online DPO models are trained on Helpfulness with identical setups except for the annotator prompt:

• "Helpful only" (standard prompt)
• "Helpful and short" (annotator told to prefer shorter responses when quality is similar)
• "Helpful and very short" (annotator told to prefer concise responses throughout)

Figure 6a tracks average response length over training: the "helpful" model produces ~120 tokens, "helpful and short" drops to ~90 tokens, and "helpful and very short" drops to ~40 tokens. The length reduction is substantial and monotonic with prompt intensity, demonstrating that the policy's behavior responds directly to the annotator's prompt-defined preferences.

Figure 6b tracks win rate against the SFT baseline: all three models maintain win rates well above 50%, though the "very short" model's win rate (~57%) is lower than the "helpful only" model's (~68%). This creates a controllability-quality tradeoff: shorter responses come at a cost to helpfulness, but even the most aggressively shortened responses still improve over the untuned SFT baseline. Human evaluation corroborates: quality scores decline from 4.08 ("helpful") to 3.72 ("helpful and short") to 3.26 ("helpful and very short"), but all remain above the SFT baseline's 3.19.

The reference lines for SFT, RLHF, and RLAIF in Figure 6b (all around 100–130 tokens, with RLAIF/RLHF at ~50–55% win rate) contextualize the result: even the "helpful and very short" model outperforms RLAIF and RLHF in win rate while producing dramatically shorter responses (40 vs. ~120 tokens).

Weak-to-Strong: Can a Weaker Annotator Improve a Stronger Policy?

Section 4.7 (Figure 7) tests whether a weaker AI annotator (PaLM 2-XS) can improve a stronger policy (PaLM 2-S). The setup mirrors the main experiments but with the policy scaled up one tier. Two online DPO variants are compared: one with PaLM 2-XS annotator (weak teacher), one with PaLM 2-L annotator (strong teacher). Win rates are judged by Gemini Pro against the SFT baseline and offline DPO.

Results: both variants improve over the SFT baseline, with the strong teacher (PaLM 2-L) outperforming the weak teacher (PaLM 2-XS). The weak teacher still provides meaningful improvement, demonstrating that OAIF can function in a weak-to-strong generalization setting (Burns et al., 2023). The paper emphasizes a key difference from Burns et al.'s setup: "in our work, the role of teacher is a simpler discriminative task (labelling preference), whereas the student model being aligned is given a more difficult one (generating proper responses)" — essentially, annotation is an easier task than generation, so a weaker model can still provide useful discriminative feedback for a stronger generative model.

---

Ablation Studies and Robustness Checks

• DAP loss function generality: OAIF is demonstrated with three different loss functions — DPO, IPO, and SLiC. All three show consistent online-over-offline advantages in human evaluation (Tables 2–3), with win rates of 63.74%, 64.39%, and 71.05% respectively. This is the paper's primary generality claim and is robustly supported across all three losses.

• Task generality: OAIF is demonstrated across three distinct tasks — TL;DR (summarization), Helpfulness (dialogue quality), and Harmlessness (safety). Table 2 shows consistent online-over-offline advantages on all three, with win rates of 63.74%, 58.60%, and 60.26% respectively. The Harmlessness task shows the most extreme asymmetry (only 3.84% losses to offline DPO), suggesting OAIF is particularly beneficial for safety alignment, possibly because off-policy data is especially unreliable for judging harmful content.

• Evaluator robustness: Figure 3 uses Gemini Pro as evaluator; Appendix D (Figure 9) replicates the online-vs-offline training dynamics using PaLM 2-L. The same overfitting pattern and cross-over point are observed, confirming that the result is not an artifact of evaluator choice. Appendix C validates Gemini Pro's alignment with human judgments (70.21% average alignment accuracy across all three tasks, comparable to PaLM 2-L's 70.72%, per Table 4).

• RM-vs-LLM annotator: The online DPO variant using the RLAIF-trained RM as annotator (Section 4.4, mentioned in text) achieves <30% win rate against OAIF with the LLM annotator, judged by Gemini Pro. This is a critical ablation isolating annotator type as the variable, supporting the claim that the LLM annotator specifically avoids the distribution-shift problem that afflicts RMs trained on off-policy data.

• Length-controlled quality: Figure 4b buckets responses by length and shows online DPO maintaining quality advantages at every length bucket. This rules out the confound that OAIF's advantages are solely due to generating longer responses.

• Annotator prompt variation for length control: Figure 6 shows that three different annotator prompts produce three different policies with systematically varying response lengths (~120, ~90, ~40 tokens), demonstrating that the annotator prompt meaningfully controls the policy's behavior.

• Weak annotator for strong policy: Figure 7 shows that PaLM 2-XS annotator provides useful feedback for PaLM 2-S policy, with the aligned policy outperforming both SFT and offline DPO baselines, though underperforming the PaLM 2-L annotator variant. This establishes a lower bound on useful annotator capability.

• Negative results and missing ablations: The paper does not report several ablations that would strengthen the central claims:

  • No data-budget comparison: OAIF generates fresh preference pairs at every step; offline methods reuse a fixed dataset. No experiment matches the total annotation budget between online and offline methods (e.g., comparing online DPO with 100K unique annotations to offline DPO trained on 100K pre-collected pairs for the same number of epochs). Without this, it is unclear whether OAIF's advantage comes from the online/on-policy property or simply from seeing more diverse preference data.
  • No annotator temperature or decoding ablation: The policy samples responses at temperature 0.9, but the annotator's decoding strategy (greedy vs. sampled, temperature) is not specified or ablated. The preference extraction uses log-probabilities, which are sensitive to the annotator's calibration.
  • No prompt robustness check: The annotator prompts (Tables 5–8) are elaborate, multi-paragraph instructions. It is unknown whether OAIF's effectiveness depends on prompt quality — would simpler prompts work? Are certain prompt phrasings critical?
  • No SFT data ablation: The SFT baseline $\pi_{{\bm{\theta}}^0}$ is trained on the chosen responses from the preference dataset $\mathbb{D}$, which gives it an initial distribution close to $\rho$. How OAIF performs starting from a more different initial policy is not explored.

---

Critical Assessment

Central Claim 1: OAIF turns offline DAP methods into online methods, substantially improving performance (~66% average win rate over offline counterparts)

Supported, with evidence from three loss functions and three tasks. Tables 2–3 provide consistent human evaluation results showing large and asymmetric win rates (58–71% across all comparisons). The training dynamics in Figure 3 provide a clear mechanistic explanation: offline DAP overfits while online DAP keeps improving. The fact that this pattern holds across DPO, IPO, and SLiC (Table 3) strongly supports the generality claim — the improvement is not tied to a specific loss function's properties.

However, the experiments demonstrate only that OAIF-online methods outperform one specific offline configuration: DAP trained on the original preference dataset $\mathbb{D}$ from policy $\rho$. There is no comparison to:

• Offline DAP trained on $\mathbb{D}$ with data augmentation or regularization
• Offline DAP with early stopping before overfitting (Figure 3 suggests offline DPO peaks around step 3,500; Table 2 compares the best offline model, but we don't know if the offline best is at the peak or after decline)
• Offline DAP with iterative data collection (periodically sampling new responses from the current policy and obtaining fresh labels)

The offline baseline is the simplest possible offline configuration. A stronger offline baseline (e.g., early-stopped offline DPO, or offline DPO with a larger and more diverse dataset) might narrow or eliminate the gap. The paper does not characterize how much of OAIF's advantage comes from the online aspect vs. the on-policy aspect vs. simply seeing more unique preference annotations — these are confounded in the experimental design.

Central Claim 2: Online DPO with OAIF is preferred over RLHF and RLAIF (58% in 4-way comparison, Figure 4a)

Supported on TL;DR with PaLM 2 models. Figure 4a provides a clean head-to-head human evaluation showing online DPO as the most preferred method. The length-controlled analysis in Figure 4b strengthens this by showing the advantage persists at every response length.

However, this result is demonstrated on a single task (TL;DR) with a single model family (PaLM 2). The RLHF and RLAIF baselines, while matching the AI feedback source (PaLM 2-L for RLAIF) and human feedback source (for RLHF), may use suboptimal RL configurations — the paper does not extensively tune the RL pipeline. The known sensitivity of RLHF to hyperparameters (KL penalty coefficient, PPO clipping, value function architecture) means that a more carefully tuned RLHF baseline could perform differently. The claim that online DPO is "preferred over RLHF" should be qualified as "preferred over the specific RLHF configuration tested on TL;DR with PaLM 2-XS."

Central Claim 3: The LLM annotator can be controlled via prompt instructions (demonstrated via length control, Figure 6)

Well-supported for the specific attribute tested (length). The monotonic relationship between prompt intensity ("helpful" → "helpful and short" → "helpful and very short") and response length (~120 → ~90 → ~40 tokens) is convincing. The fact that all variants maintain improvement over the SFT baseline shows the controllability doesn't come at the cost of any quality improvement.

However, the paper's broader claim that this demonstrates controllability for "qualitative desiderata" (Section 5) is not experimentally supported. Length is a particularly simple, measurable, and objective attribute. The paper does not demonstrate controllability for subjective or nuanced attributes (politeness, formality, empathy, cultural sensitivity). It is plausible that LLM annotators are much better at assessing length than at assessing nuanced qualitative attributes, and that more subjective criteria might introduce noise that degrades alignment quality. The length experiment is a proof-of-concept for controllability, not a demonstration that arbitrary attributes can be reliably controlled.

Central Claim 4: OAIF avoids the distribution-shift problem that plagues RM-based online approaches

Partially supported by the RM ablation (Section 4.4). The finding that online DPO with an RM annotator achieves <30% win rate against online DPO with an LLM annotator is the paper's most direct evidence for this claim, since both are otherwise identical (online, on-policy DPO). This is a strong within-experiment comparison.

However, the RM being used (the one from RLAIF) is trained on the original preference dataset $\mathbb{D} \sim \rho$ and never updated. This is the simplest possible RM configuration and is known to suffer from distribution shift. The paper does not compare against:

• An RM that is periodically retrained on fresh responses from $\pi_{{\bm{\theta}}^t}$ (as recommended by Ziegler et al., 2019)
• An RM trained with data augmentation or uncertainty quantification to be robust to distribution shift
• An ensemble of RMs that might be more robust than a single RM

The paper's claim that "synchronously retraining the RM... would greatly complicate the training pipeline and increase training cost" (Section 4.4) is an argument about practicality, not capability. A retrained RM might match or exceed the LLM annotator's performance at higher computational cost. The paper demonstrates that the LLM annotator is more practical, not necessarily that it is more capable in absolute terms.

Missing Experiments That Would Have Strengthened the Paper

• Annotator prompt sensitivity analysis: How robust is OAIF to the specific phrasing of the annotator prompt? The prompts in Tables 5–8 are elaborate; would shorter prompts work? Would prompts from different annotator models produce consistent preferences?

• Ablation on number of annotation orderings: The paper averages over two orderings to cancel position bias. What happens with a single ordering? How large is the position bias in practice?

• Data budget matching: Online DPO sees fresh preference pairs at every step; offline DPO reuses a fixed set. A comparison where both see the same number of unique annotations (e.g., by subsampling the online annotations) would isolate the online/on-policy effect from the data-quantity effect.

• Policy scale generalization: All experiments use PaLM 2-XS policies. Whether OAIF's advantages persist when aligning larger policies (e.g., PaLM 2-S or PaLM 2-L) is untested. The paper's only scale experiment (Section 4.7) varies the annotator size, not the policy size.

• Annotation quality evaluation: The paper relies on Gemini Pro for automatic evaluation and three human raters for validation. A larger-scale human evaluation with proper inter-rater agreement metrics (e.g., Fleiss' kappa) would strengthen the reliability of the human preference results. The three-rater setup with unreported agreement is a weakness, particularly given the subjective nature of "helpfulness" and "harmlessness."

• Comparison to best-of-N or majority voting baselines: The paper compares against aligned policies (SFT, RLHF, RLAIF, offline DAP) but not against inference-time strategies. Would a simple best-of-N or majority voting over the SFT baseline's outputs match or exceed the aligned policies? Without this baseline, the absolute improvement from alignment is unclear.

Subtle Limitations in the Experimental Design

• The prompt dataset $\mathbb{D}_{\mathcal{X}}$ is in-distribution: The paper extracts prompts from the same preference dataset used to train baselines, so all evaluation is on in-distribution prompts. The performance of OAIF-aligned models on out-of-distribution prompts (different tasks, different styles) is entirely unknown. The paper acknowledges this limitation in Section 5 but does not address it experimentally.

• Model selection uses Gemini Pro: The "best performing" online and offline models are selected based on Gemini Pro's evaluation. If Gemini Pro has systematic biases (e.g., favoring longer responses, which the paper shows is a known bias), the selected models may be optimal according to Gemini Pro but not according to humans. The human evaluation partially addresses this by confirming the selected online model outperforms the selected offline model, but the absolute performance of both could be suboptimal relative to models selected by a different criterion.

• Annotation cost asymmetry is unaccounted for: OAIF requires two forward passes through a large annotator LLM per training step. Offline DAP requires loading pre-collected data, which has negligible computational cost. The wall-clock time and FLOPs comparison is not reported, making it difficult to assess whether OAIF's performance advantage is worth the additional annotation compute. This is particularly relevant for the annotator-size ablation (Section 4.5): PaLM 2-L annotations cost more than PaLM 2-XS annotations, so the performance improvement from larger annotators must be weighed against their inference cost.

• The 4-way comparison (Figure 4a) has a potential confound: online DPO tends to produce longer responses, and all three human raters might share a length bias. While Figure 4b attempts to control for this by plotting quality vs. length, the length-bucketed analysis reduces statistical power (each bucket has fewer responses) and the error bars suggest overlapping confidence intervals at some length ranges. The claim that online DPO is genuinely better at fixed length is plausible but not statistically conclusive given the small rater pool and unreported variance.

6. Limitations and Trade-offs

The Annotator Inference Cost Is Unaccounted For in Any Comparison

The assumption or constraint. OAIF requires two forward passes through the annotator LLM (to average over both response orderings and cancel position bias) at every single training step. This means the total compute cost of alignment includes both the cost of training the policy and the cost of querying a large, frozen LLM annotator thousands of times. Despite the central role of this annotation cost, the paper never accounts for it in any experimental comparison. Section 4 describes batch size 128, temperatures, learning rates, and optimizer choice for the policy, but includes no discussion of annotator FLOPs, wall-clock time, or total compute budget. The offline baselines (offline DPO, IPO, SLiC) have negligible annotation cost since the preference dataset $\mathbb{D}$ is pre-collected. The RLHF and RLAIF baselines have their own annotation costs (RM inference at each RL step), but these are also not factored into any head-to-head compute-equivalent comparison.

The consequence. A practitioner evaluating whether to adopt OAIF cannot determine whether its performance advantage over offline DAP is worth the additional computational cost. Consider the annotator-size ablation in Section 4.5 (Figure 5): PaLM 2-L produces the best results, but querying PaLM 2-L twice per training step (for the two orderings) for thousands of steps is dramatically more expensive than training offline DPO, which simply loads pre-collected data from disk. The paper's headline finding—online DPO with PaLM 2-L annotator achieves 63.74% win rate and 3.95 quality vs. offline DPO's 7.69% and 3.46 (Table 2)—might look very different if compared at equal total FLOPs. If offline DPO can be trained for many more epochs at the same cost as OAIF's single pass, the performance gap could narrow or reverse. Similarly, the comparison against RLHF and RLAIF (Figure 4a) does not control for total compute: online DPO with a PaLM 2-L annotator might use more or less total compute than the full RLHF pipeline (RM training + PPO), but we cannot tell from the reported data.

What evidence exists in the paper. None. The paper never reports annotator FLOPs per step, total training wall-clock time, or any compute-equivalent comparison. The only cost-related discussion is an acknowledgment in Section 5 that self-annotating models avoid the need for a separate annotator, but this is framed as a flexibility advantage, not a cost concern. The paper's conclusion that OAIF is "simple and effective" (Section 6) does not account for the inference cost of the annotator.

Mitigation status. Not addressed. The paper does not suggest any cost-mitigation strategies (e.g., using the annotator once per step instead of twice by accepting position bias, using a smaller annotator, caching annotations, or periodically rather than continuously querying the annotator). The annotation cost is simply absent from the analysis. Future work would need to establish cost-performance tradeoff curves to determine whether OAIF is practically deployable or whether its advantages require an annotator budget that makes it economically inferior to alternatives. This is the paper's most significant practical limitation.

---

All Results Are on a Single Model Family (PaLM 2) at a Single Policy Scale (XS)

The assumption or constraint. Every experiment in Section 4 uses PaLM 2-XS as the policy being aligned, with PaLM 2 variants (XS, S, L) as annotators. The only scale variation is in Section 4.7, where the policy is upgraded to PaLM 2-S, but this single-data-point shift (XS → S) is a minor scale change within the same model family. The paper contains no experiments with policies at the scale of PaLM 2-L or larger, no experiments with a different model family entirely (e.g., Gemma, LLaMA, GPT variants), and no experiments where the policy and annotator come from different model families (e.g., PaLM 2 policy annotated by Gemini Pro). The authors state in Section 1 that they "believe this model is representative of the capabilities of many contemporary LLMs" and acknowledge in Section 5 that "whether our conclusion holds after scaling up is not investigated."

The consequence. Two distinct generalizability failures are possible. First, policy scale: larger models may behave differently under OAIF. As policies become more capable, their initial pass@1 on tasks improves, making it harder for alignment to produce measurable gains. The distribution shift between $\pi_{{\bm{\theta}}^t}$ and $\pi_{{\bm{\theta}}^0}$ may be larger or smaller at different scales. The overfitting dynamics observed for offline DPO in Figure 3 might change — larger models with more capacity might overfit faster to the static dataset, making OAIF's advantage even larger, or they might be more robust to distribution shift due to better generalization, narrowing the gap. Second, annotator-policy interaction: the paper's central claim that LLM annotators avoid the distribution-shift problem (Section 2, Table 1) is tested only when both annotator and policy are PaLM 2 models. If the annotator and policy come from different model families with different pretraining distributions, the annotator's preference judgments might not be equally reliable. The paper's RM-vs-LLM annotator ablation (Section 4.4) demonstrates that a PaLM 2-L annotator outperforms a trained RM for a PaLM 2-XS policy, but we don't know if this holds when, say, a PaLM 2 annotator evaluates responses from a LLaMA-based policy.

Additionally, as the authors note in Section 5, "it is harder to distinguish responses of higher quality." This is a specific concern for OAIF: as policies improve, the gap between preferred and dispreferred responses shrinks, making the annotator's task harder. At some capability threshold, the annotation signal-to-noise ratio may degrade to the point where OAIF provides no benefit over offline alternatives, or even hurts. The paper provides no evidence about where this threshold lies or whether it was approached in the PaLM 2-XS experiments.

What evidence exists in the paper. The sole scale-variation experiment is Section 4.7 (Figure 7), where PaLM 2-S policy benefits from both PaLM 2-XS and PaLM 2-L annotators. This is a single additional scale point within the same model family, providing minimal evidence for scale generalization. The paper directly acknowledges that scaling up is an open question (Section 5).

Mitigation status. Not addressed experimentally. Acknowledged explicitly in Section 5 with a call for further study. A responsible practitioner should treat the OAIF findings as demonstrated for a narrow model-family-and-scale regime and should not assume that the same advantages hold when aligning much larger or architecturally different models.

---

The Annotator's Prompts Are Highly Engineered and Prompt Robustness Is Untested

The assumption or constraint. OAIF's entire feedback mechanism depends on the annotator LLM receiving a well-designed prompt that reliably elicits preference judgments aligned with the desired behavior. The prompts used in the paper (Tables 5–8 in Appendix E) are elaborate, multi-paragraph instructions that define detailed evaluation criteria. For TL;DR summarization (Table 5), the prompt defines four evaluation axes (coherence, accuracy, coverage, overall quality) with paragraph-length descriptions of each. For Harmlessness (Table 7), the prompt distinguishes between rating prompts and AI feedback prompts, with different criteria for each. Crucially, the paper presents these prompts as fixed artifacts without any sensitivity analysis: there is no experiment showing that OAIF works with simpler prompts, with prompts written by different people, or with prompts that define the preference criteria differently. The text-controllability experiment (Section 4.6, Figure 6) demonstrates that changing the prompt changes behavior, but it does not demonstrate that the specific prompt design used in the main experiments is necessary, replaceable, or optimal.

The consequence. The practical fragility of OAIF is unknown. If annotator prompt design is a critical hyperparameter — meaning that a poorly written prompt produces low-quality feedback that degrades alignment — then OAIF's effectiveness is contingent on prompt engineering skill, not on a fundamental algorithmic property. This would make OAIF less robust than RM-based methods, where the preference function is learned from data and (in principle) can be validated and calibrated, or offline DAP methods, where the preference data is fixed and its quality can be audited before training. A practitioner adopting OAIF would need to invest substantial effort in prompt design and validation, with no guidance from the paper on what makes a good annotator prompt or how to detect when the prompt is inadequate.

Furthermore, the annotator prompts encode specific values and criteria. The TL;DR prompt's four axes (coherence, accuracy, coverage, overall quality) represent a particular conception of summary quality that may not generalize to other summarization tasks or user populations. The Helpfulness prompt's emphasis on "thoughtful, honest, and reasonable" responses encodes a specific set of conversational norms. The paper does not discuss whose values these prompts encode, whether they are culturally specific, or whether different prompt designs would lead to meaningfully different aligned behaviors beyond the surface-level length control demonstrated in Section 4.6.

A related concern is that the annotator prompt might interact with the evaluator prompt. If both use similarly structured prompts (e.g., both define coherence/accuracy/coverage axes), the aligned policy might learn to produce responses that score well on those axes according to the annotator's interpretation, but the alignment might not generalize to a different evaluator using different criteria. The paper partially mitigates this by using Gemini Pro (different model, different prompt) for evaluation, but does not ablate evaluator prompt design.

What evidence exists in the paper. None. The prompts are listed in Appendix E but never ablated. The paper does not investigate whether OAIF's performance degrades with simpler prompts, whether different prompt phrasings produce different aligned behaviors (beyond length control), or whether prompt quality is a significant factor in OAIF's effectiveness.

Mitigation status. Not addressed. The paper treats the annotator prompts as a fixed implementation detail rather than as a variable that might affect the method's reliability. A complete evaluation of OAIF would require demonstrating that the method works across a range of prompt designs, or at minimum, characterizing how prompt quality affects alignment outcomes. Future work on prompt design for AI feedback annotation is clearly needed but not discussed.

---

Offline Baselines Are Not Optimized; the Online Advantage May Be Partially a Data-Quantity Effect

The assumption or constraint. The paper's central empirical claim is that OAIF (online, on-policy DAP) substantially outperforms offline DAP. To support this, the paper compares online DPO/IPO/SLiC against the simplest possible offline configuration: training on the pre-collected preference dataset $\mathbb{D} \sim \rho$ with no modifications. However, this offline baseline suffers from several correctable weaknesses that the paper does not attempt to address:

• No early stopping. Figure 3 shows that offline DPO overfits dramatically around step 3,500, with performance degrading sharply afterward. Table 2 reports the "best performing" offline model, but the selection criterion is based on Gemini Pro's development-set evaluation (Section 4.2), which itself may be noisy. A properly early-stopped offline DPO model might substantially outperform the offline model actually used in human evaluations.

• No data augmentation or regularization. Standard techniques for mitigating overfitting to small or off-distribution datasets—data augmentation, dropout tuning, weight decay optimization, or KL regularization strength tuning—are not applied to the offline baselines. The offline DPO model uses the same $\beta = 0.1$ as online DPO, but $\beta$ controls the strength of the KL penalty relative to the preference signal: a larger $\beta$ might prevent the overfitting observed in Figure 3 by keeping the policy closer to $\pi_{{\bm{\theta}}^0}$. No $\beta$ sweep is reported for offline DPO.

• No dataset expansion or iterative collection. Offline DPO is trained on exactly one dataset of size $N$. The paper does not compare against a version where the offline dataset is augmented (e.g., by sampling additional responses from $\rho$ and obtaining more labels) or where the dataset is periodically refreshed with new samples from the current policy. The comparison is between "static dataset of fixed size from a different model" and "infinite stream of fresh on-policy annotations"—a comparison that confounds multiple variables (off-policy vs. on-policy, finite vs. infinite data, stale vs. fresh labels).

The consequence. The ~66% average win rate of online DAP methods over offline counterparts (Tables 2–3) may overstate the advantage attributable to the online/on-policy property specifically. Some fraction of the improvement likely comes from OAIF's access to more total preference annotations (since it generates fresh pairs at every step) and from the regularizing effect of constantly changing training data (which prevents overfitting to a fixed set). If offline DPO were trained on a larger dataset, with early stopping, or with tuned regularization, the performance gap might narrow considerably. The paper does not disentangle how much of OAIF's advantage comes from the online property (feedback obtained at training time), the on-policy property (responses from the current policy), and the data-quantity property (unlimited fresh annotations), because these are all changed simultaneously when switching from offline to OAIF.

What evidence exists in the paper. The overfitting evidence in Figure 3 is the closest the paper comes to showing that the online property specifically matters, since it demonstrates that offline DPO degrades during training rather than starting from a lower performance level. However, without early stopping or regularization baselines, this shows only that naive offline DPO overfits, not that offline DPO is fundamentally limited when properly regularized. The RM ablation in Section 4.4 (online DPO with RM annotator vs. online DPO with LLM annotator) isolates the annotator type, but no experiment isolates the online/on-policy property from the data-quantity property.

Mitigation status. Not addressed. The paper does not acknowledge the confounding between online, on-policy, and data-quantity effects. A proper ablation would compare online DPO to offline DPO with matched total annotation budget (e.g., both seeing exactly $T \times B$ unique preference pairs, where $T$ is training steps and $B$ is batch size) to determine whether the online/on-policy property provides benefits beyond simply having more data.

---

Evaluation Is on In-Distribution Prompts Only; Generalization to Novel Prompts Is Unknown

The assumption or constraint. The prompt dataset $\mathbb{D}_{\mathcal{X}}$ used for OAIF training is constructed by extracting prompts from the same preference dataset $\mathbb{D}$ used to train the offline baselines and the RMs for RLHF/RLAIF. This means all evaluation—both automatic (Gemini Pro) and human—is conducted on prompts drawn from the same distribution as the training prompts. There is no experiment measuring the performance of OAIF-aligned policies on out-of-distribution prompts: prompts from different tasks, different domains, different styles, or different difficulty levels than those in $\mathbb{D}_{\mathcal{X}}$. The paper acknowledges this explicitly in Section 5: "Since we extract prompts from the given preference dataset, our study assumes an in-distribution of prompts used for evaluation, thus lacks of evaluating the performance of aligned LLMs on out-of-distribution prompts."

The consequence. The paper provides no evidence about whether OAIF's alignment benefits generalize beyond the specific prompt distribution it was trained on. This is particularly concerning because OAIF's online, on-policy nature means the policy continuously adapts to the annotator's preferences on the training prompt distribution. If the training prompts systematically differ from deployment prompts (e.g., different topic distributions, different user demographics, different task formulations), the alignment might not transfer. Worse, the policy might overfit to the annotator's preferences in the context of the training prompt distribution in ways that produce undesirable behavior on novel prompts.

This limitation is especially relevant for safety alignment (Harmlessness), where OOD generalization is critical: a policy that is harmless on the training prompt distribution might produce harmful responses on novel prompts that trigger different failure modes. The paper's finding that Harmlessness shows the largest online-over-offline advantage (60.26% win, 3.84% loss in Table 2) is encouraging but does not address whether this advantage persists under distribution shift. The prompt-controllability experiment (Section 4.6) further heightens this concern: if the annotator's preferences can be steered by prompt instructions, and the policy adapts to these preferences on a specific prompt distribution, the resulting behavior may be brittle—working well on prompts similar to training but failing unpredictably on dissimilar prompts.

What evidence exists in the paper. None. All evaluation is on in-distribution prompts from the same datasets used for training ($\mathbb{D}_{\mathcal{X}}$ extracted from $\mathbb{D}$). The three tasks (TL;DR, Helpfulness, Harmlessness) represent different domains but within each domain, the train and test prompts come from the same distribution. There is no cross-task evaluation (e.g., training on Helpfulness and evaluating on Harmlessness) and no systematic perturbation of prompts to test robustness.

Mitigation status. Acknowledged explicitly in Section 5 but not addressed experimentally. This is a significant gap for a method that aims to be a practical alignment solution. A complete evaluation would require measuring performance on out-of-distribution prompts—ideally from different datasets, different domains, or adversarially constructed prompts designed to probe the policy's robustness. Without such evaluation, practitioners cannot assess whether OAIF produces alignment that is deep enough to generalize or is surface-level adaptation to the training prompt distribution.

---

Human Evaluation Uses Only Three Raters with Unreported Inter-Rater Agreement

The assumption or constraint. The paper's strongest evidence comes from human evaluation (Tables 2–3, Figure 4). However, all human evaluations rely on only three raters, and no inter-rater agreement metrics (e.g., Fleiss' kappa, Krippendorff's alpha, or even raw agreement percentage) are reported. Section 4.1 describes the protocol: "three raters are presented with responses generated from a set of policy models. Each rater is then asked to independently score the responses' quality (from 1 to 5 where 5 denotes the highest) and to pick the best one, and the average score is then used to compare the models." There is no discussion of rater training, rater qualification, rater agreement, or how disagreements were resolved.

The consequence. The reported win rates and quality scores carry unknown statistical uncertainty. With only three raters, a single rater with atypical preferences or inconsistent scoring can substantially shift the averages. Consider the TL;DR results in Table 2: online DPO achieves 63.74% win rate and 3.95 quality vs. offline DPO's 7.69% and 3.46. If two raters strongly preferred online DPO and the third was indifferent or preferred offline DPO, the aggregate statistics would look very different than if all three agreed. Without inter-rater agreement metrics, we cannot distinguish between genuine consensus (all raters agree online responses are better) and idiosyncratic agreement (raters disagree but the averaging procedure happens to favor online DPO).

The 4-way comparison in Figure 4a is particularly sensitive to this limitation. With three raters evaluating four models, each rater's individual ranking carries substantial weight in the aggregate preference statistic (58.00% for online DPO). If rater preferences are correlated (e.g., all three share a length bias or a stylistic preference), the 58% figure might reflect rater-specific biases rather than genuine quality differences. Figure 4b attempts to address length bias by bucketing responses by length, but with three raters evaluating responses distributed across six length buckets, the number of evaluations per bucket per rater becomes very small, making the bucketed quality scores noisy.

The quality scores (1–5 scale) add further granularity but with only three raters, the standard errors on these averages are large. A difference of 0.49 points (3.95 vs. 3.46 for online vs. offline DPO on TL;DR, Table 2) might be statistically significant or might be within the range of rater disagreement—we cannot tell. The paper reports no confidence intervals, no significance tests, and no power analysis.

What evidence exists in the paper. None for inter-rater reliability. The paper reports only aggregate win/tie/loss rates and average quality scores, with no indication of rater-level variance, agreement, or disagreement patterns. The only nod to evaluation reliability is Appendix C, which validates Gemini Pro against human judgments—but this validation itself relies on the same human raters whose agreement is unreported, creating a circular dependency.

Mitigation status. Not addressed. This is a methodological weakness that affects the interpretability of all human evaluation results. A responsible evaluation would report inter-rater agreement metrics, use more raters (or formal power analysis to justify the sample size), and provide rater-level breakdowns to allow readers to assess result reliability. The paper's conclusions about human preference for OAIF-aligned models—while plausible given the consistency across tasks and methods—would be considerably strengthened by rigorous evaluation methodology.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a reframing of DAP methods' failure modes as a data distribution problem rather than an algorithmic deficiency. Before OAIF, the conversation around DPO and related methods implicitly treated their performance ceiling relative to RLHF as an inherent cost of simplification—trading away the reward model and policy gradients naturally meant trading away some performance. The paper challenges this assumption with clean empirical evidence: when DAP methods are given online, on-policy data (via OAIF), they not only close the gap with RLHF but can exceed it (58% preference in 4-way human evaluation, Figure 4a), while still avoiding RLHF's complexity. This reframing shifts the research focus from developing better loss functions toward developing better feedback mechanisms.

This is better characterized as a diagnostic breakthrough than as a paradigm shift. The individual components of OAIF—DAP losses, LLM-as-evaluator prompting, online training loops—all existed prior. What is new is the experimental demonstration that combining them changes the nature of the alignment problem: the DPO loss, which appeared to overfit when trained on static datasets (Figure 3, sharp drop at step 3,500), does not overfit when fed on-policy preference pairs (the blue curve keeps climbing). The implication is that the offline, off-policy data regime, not the algorithmic formulation of DAP, was the binding constraint on DAP performance. This is a diagnostic finding with broad implications: it suggests that much of the RLHF-vs-DAP debate has been asking the wrong question ("which algorithm is better?") when the right question is "which algorithm gets better data?"

The paper also reconciles a tension in the literature around the role of reward models. Prior work had shown that RM-based online approaches (Iterative DPO, RSO) partially improve over offline DAP, but a clear explanation for why they fall short of RLHF was absent. OAIF provides that explanation: the RM itself is trained on off-policy data ($\mathbb{D} \sim \rho$), so the distribution-shift problem is displaced to the RM rather than eliminated (Appendix A.3, Section 2). The LLM annotator avoids this because it has no training distribution—its preference function emerges from pre-training and prompting, not from fitting a specific dataset. The experimental proof is in the RM-vs-LLM annotator ablation (Section 4.4): online DPO with the RLAIF-trained RM achieves <30% win rate against online DPO with the LLM annotator, despite using identical DPO loss and identical online, on-policy sampling. This single result reframes the role of trained reward models in alignment: they are not inherently online-capable in the way an LLM annotator is, because their training creates a distributional commitment that degrades as the policy evolves.

Several research directions become more attractive as a result of this work:

• LLM annotators as a first-class alignment primitive. The paper establishes that frozen LLMs can serve as online preference oracles without training, meaning the alignment field now has a feedback source that is simultaneously online, distribution-shift-robust, and prompt-programmable. This opens a design space distinct from both trained reward models (which are online but distribution-shift-vulnerable) and human feedback (which is distribution-shift-robust but not scalable online). Future alignment methods can assume the existence of such annotators and design algorithms around their specific properties (prompt-controllability, scale-dependent quality, position bias).

• Controllable alignment objectives via natural language. The prompt-controllability experiment (Section 4.6, Figure 6) demonstrates that the alignment objective can be changed by editing text, not by re-training components. This makes alignment objectives soft, editable, and composable in a way that fixed reward models or static preference datasets cannot match. The implication is that future alignment systems might treat the objective function as a prompt rather than as a compiled artifact—a substantial shift in how we think about specifying desired model behavior.

• Weak-to-strong generalization via discriminative feedback. Section 4.7 demonstrates that a weaker annotator (PaLM 2-XS) can improve a stronger policy (PaLM 2-S). The paper argues this works because "the role of teacher is a simpler discriminative task (labelling preference), whereas the student model being aligned is given a more difficult one (generating proper responses)" (Section 4.7). This suggests that the weak-to-strong generalization problem (Burns et al., 2023) may be more tractable in alignment settings than in supervised learning settings, because the teacher's task (judging) is inherently easier than the student's task (generating). This reframes weak-to-strong alignment as a feasible near-term goal rather than a distant research challenge.

Conversely, some directions become less attractive:

• Developing increasingly complex DAP loss functions. The paper shows that three different loss functions (DPO, IPO, SLiC) all benefit similarly from OAIF (~66% average win rate over offline counterparts, Table 3). If the primary determinant of alignment quality is the data regime (online/on-policy vs. offline/off-policy) rather than the specific mathematical form of the loss, then marginal improvements to DAP losses are likely to yield diminishing returns compared to improving the feedback mechanism.

• Iterative DAP methods that use a frozen, off-policy RM for pseudo-labeling. The paper's RM ablation (Section 4.4) provides direct evidence that an RM trained on off-policy data $\mathbb{D} \sim \rho$ produces substantially worse online feedback than an LLM annotator, even when the RM and annotator are architecturally comparable (both PaLM 2-L). Methods that retrain the RM synchronously (Ziegler et al., 2019) remain theoretically viable but practically expensive—OAIF provides a simpler path to online feedback that avoids the retraining problem entirely.

---

Follow-Up Research This Work Enables

Characterizing the online/on-policy vs. data-quantity confound. The paper demonstrates that OAIF (online, on-policy, unlimited data) substantially outperforms offline DAP (offline, off-policy, fixed data). But these are three variables changed simultaneously. A controlled experiment would compare online DPO to offline DPO where both see exactly the same number of unique preference annotations (e.g., by subsampling the online annotations to match the offline dataset size, or by expanding the offline dataset to match the online annotation budget). The specific question: does OAIF's advantage come from the online, on-policy nature of the feedback, or from simply seeing more diverse preference data? If the latter, then offline DAP with a sufficiently large and diverse offline dataset might match OAIF's performance, which would change the practical recommendation from "run OAIF" to "collect more preference data." If the former, then the online, on-policy property has irreducible value that cannot be replicated by expanding static datasets.

OAIF with a retrained RM baseline (not a frozen one). The paper's RM ablation uses the RLAIF-trained RM without any retraining during policy optimization, and the <30% win rate is attributed to distribution shift. A stronger baseline would retrain the RM periodically on fresh responses from $\pi_{{\bm{\theta}}^t}$ (as suggested by Ziegler et al., 2019) and compare this retrained-RM-based online DPO against OAIF with an LLM annotator, at equal total compute budget. The specific question: can a periodically-retrained RM eventually match the LLM annotator's feedback quality, or does the LLM annotator's zero-shot generalization provide a persistent advantage that retraining cannot overcome? This experiment would establish whether LLM annotators are practically more convenient (true by construction) or fundamentally more capable (unknown) as online preference oracles.

Prompt sensitivity analysis for the annotator. The annotator prompts in Appendix E are elaborate, multi-paragraph instructions that define detailed evaluation criteria. A systematic ablation would vary prompt complexity (from simple "choose the better response" to the full multi-axis prompts) and measure OAIF's downstream alignment quality. The specific question: is prompt engineering a critical hyperparameter for OAIF, or does the method work robustly across a wide range of prompt designs? If prompt quality is critical, OAIF's practical adoption requires prompt-engineering expertise that the paper does not currently provide guidance for. A negative result (OAIF works well even with simple prompts) would strengthen the method's claim to simplicity.

Cross-model-family generalization of OAIF. All experiments use PaLM 2 for both policy and annotator. A cross-family experiment would train a policy from a different model family (e.g., Gemma, LLaMA) using a PaLM 2 annotator, and vice versa. The specific question: does the LLM annotator's preference quality degrade when evaluating responses from a model with a different pretraining distribution, architectural inductive biases, or tokenization scheme? The paper's claim that LLM annotators "avoid the distribution-shift problem" (Section 2) implicitly assumes that pre-trained LLMs generalize across model families in their evaluation capability—an assumption that is untested. A degradation would reveal that annotator-policy compatibility is a hidden constraint on OAIF's applicability.

Annotator calibration and over-optimization in OAIF. The paper observes that offline DPO overfits (Figure 3) but does not investigate whether online DPO can also overfit—specifically, to the annotator's specific preferences rather than to genuine response quality. The known phenomenon of reward over-optimization in RLHF (Gao et al., 2023) should apply to OAIF as well: as the policy learns to exploit the annotator's preference patterns, the annotator's feedback becomes less informative. A long-training experiment (extending Figure 3 to many more steps) would test whether online DPO's win rate eventually plateaus or declines due to annotator over-optimization. The specific observation: does online DPO's performance eventually saturate, and if so, does the saturation point depend on annotator size (PaLM 2-XS vs. S vs. L) in a predictable way? This would establish whether annotator quality sets a ceiling on OAIF's achievable alignment, analogous to how RM quality sets a ceiling on RLHF.

Text-controllability for genuinely qualitative attributes. The length-control experiment (Section 4.6) demonstrates prompt-controllability for a simple, measurable attribute. A stronger test would attempt to control qualitative attributes that are harder for LLMs to judge reliably—for example, politeness, empathy, formality, or cultural sensitivity. The experimental design: train multiple online DPO variants with annotator prompts that specify different qualitative criteria, then evaluate (via human raters) whether the resulting policies actually differ along the intended qualitative dimensions. The specific question: does the prompt-controllability mechanism break down when the annotator LLM struggles to reliably distinguish the qualitative attribute, or does it still provide a useful signal that steers the policy in the intended direction? A negative result (policies do not differ on qualitative dimensions despite different annotator prompts) would reveal that OAIF's controllability is limited to attributes the annotator can reliably assess—which may be a narrow set.

---

Practical Applications and Downstream Use Cases

Rapid prototyping of alignment objectives. The prompt-controllability of OAIF enables a workflow where alignment objectives are iterated by editing text rather than by re-collecting preference data and retraining reward models. A product team wanting to adjust their LLM's behavior (e.g., making responses more concise, more formal, or more empathetic) could modify the annotator prompt and retrain with OAIF—a matter of hours rather than the days or weeks required for human data collection and RM retraining. The length-control experiment (Figure 6) demonstrates a concrete example: reducing average response length from ~120 to ~40 tokens by changing annotator instructions, while still improving over the SFT baseline. This capability is directly applicable to production LLM deployments where alignment desiderata evolve based on user feedback, regulatory requirements, or product strategy shifts.

Cost-efficient alignment for resource-constrained deployments. The paper shows that OAIF works with annotators as small as the policy being trained (PaLM 2-XS annotating PaLM 2-XS, Section 4.5), and even with a weaker annotator improving a stronger policy (PaLM 2-XS annotating PaLM 2-S, Section 4.7). This means organizations without access to large annotator models can still benefit from online alignment—a same-size model can provide its own training signal. The practical workflow: deploy a PaLM 2-XS-scale model, run OAIF with itself as annotator, and achieve alignment quality comparable to RLHF trained on human feedback (quality score 3.41 for OAIF with XS annotator vs. 3.38 for RLHF, human evaluation in Section 4.5). For applications where human annotation is expensive or slow (specialized domains, low-resource languages, rapidly changing requirements), self-annotated OAIF provides a scalable alignment path.

Safety alignment with reduced human exposure to harmful content. The Harmlessness results (Table 2) show online DPO achieving a 60.26% win rate with only 3.84% losses against offline DPO—the most asymmetric advantage across all tasks. This suggests OAIF is particularly effective for safety alignment, where off-policy preference data may be especially unreliable (harmful responses from $\rho$ may not represent harmful responses from $\pi_{{\bm{\theta}}^t}$). A practical safety pipeline: continuously run OAIF with a harmlessness-focused annotator prompt during model training, using the annotator to judge the safety of the current policy's own outputs rather than relying on a static dataset of harmful responses. This reduces the need for human annotators to review genuinely harmful content (since the annotator is an LLM) while ensuring the safety signal adapts as the policy evolves and potentially discovers new failure modes.

Personalized alignment from user feedback. Section 5 of the paper discusses the possibility of replacing the LLM annotator with real online users: "it is technically plausible to replace them with real online users." While the paper notes that sample efficiency (~256,000 samples to visibly change behavior at batch size 128) is a bottleneck for single-user personalization, the OAIF framework provides a natural architecture for group-level personalization. A deployment serving a specific user community (e.g., medical professionals, legal practitioners, students in a particular educational system) could collect online preferences from that community's members, using OAIF to align the policy toward community-specific values without requiring a pre-collected preference dataset. The key practical advantage over RLHF is that OAIF requires only pairwise preferences ("which response is better?") rather than absolute scores, which is a simpler and more reliable annotation task for non-expert users. The prompt-controllability of the annotator also means community-specific alignment criteria can be specified in natural language, making the alignment objective transparent and auditable.