Sample-Efficient Alignment for LLMs

ArXiv: 2411.01493

🎯 Pitch

SEA uses Thompson sampling to make LLM alignment dramatically more query-efficient, achieving a 205% improvement in win rate over reference responses on TL;DR summarization with the same annotation budget. It treats alignment as a contextual dueling bandit problem, actively selecting which response pairs to label rather than passively sampling them.

---

1. Executive Summary

This paper introduces SEA (Sample-Efficient Alignment), a Thompson sampling-based algorithm for aligning LLMs with human preferences using budgeted online feedback, framed as a contextual dueling bandit problem. The method incorporates three named mechanisms—an epistemic reward model (a deep ensemble of MLP heads on a frozen transformer that approximates posterior sampling over reward functions), policy-guided search (generating candidate responses from the latest online policy and selecting the dueling pair by maximizing sampled reward or preference variance), and mixed preference learning (training the policy on a blend of oracle-labeled data and synthetic labels from the epistemic reward model)—and is empirically validated across three model scales (1B, 2.8B, 6.9B parameters from the Pythia family) and three preference learning algorithms (DPO, IPO, SLiC) on the TL;DR summarization task. SEA achieves 2–5× better sample efficiency compared to passively online methods, delivering relative improvements of +84% to +205% in win rate against reference responses at convergence. The paper establishes that active exploration with online policy updates substantially outperforms both offline and passively online alignment, but only when the exploration strategy matches the deployment objective—explore-and-exploit for online user-facing systems, or best-arm identification for crowdsourcing scenarios.

2. Context and Motivation

The Core Problem: Human Feedback Is the Bottleneck in LLM Alignment

Aligning large language models with human preferences—making them helpful, harmless, and honest—requires massive amounts of human annotations. The standard RLHF pipeline (Christiano et al., 2017; Ouyang et al., 2022) involves training a reward model on tens of thousands of human-labeled preference comparisons, then using reinforcement learning to align the LLM's behavior to that reward signal. More recent direct alignment methods like DPO (Rafailov et al., 2023) eliminate the explicit RL step but still depend on the same large offline preference datasets. In either case, the volume of human feedback is the primary economic and logistical bottleneck—human annotation is expensive, slow, and difficult to scale.

This bottleneck is particularly acute because of a striking asymmetry that the paper highlights (Section 1):

"verifying is believed to be easier than synthesizing novel behaviors"

In principle, an LLM could learn superhuman capabilities by generating many candidate outputs and receiving human feedback on which ones are better, even if humans cannot produce those outputs themselves. But this vision—iteratively generating massive new candidates and asking for human feedback—is only realizable if the feedback loop is sample-efficient: the model must extract maximal learning from every preference query. The central question this paper tackles is therefore practical and pressing: how can we align LLMs using as few human preference labels as possible?

The Missing Piece: Active Exploration in Online Alignment

The paper identifies that most prior alignment methods fail to satisfy two properties that are jointly necessary for sample efficiency (Section 3):

Property 1 (Online interaction). The agent should interact and learn online—acting with its latest learned policy, collecting feedback on those actions, and immediately using that experience to improve. This prevents the distributional mismatch that plagues offline methods, where the policy is trained on data collected by a different (usually worse) behavior policy, leading to the well-known problem of distribution shift: the learned reward model is inaccurate for outputs the LLM wants to generate but that were never shown to human annotators.

Property 2 (Active exploration). The agent should strategically select which responses to show humans for feedback, rather than passively sampling pairs from its current policy. In bandit terms, the agent faces an exploration-exploitation tradeoff: it can show responses it already thinks are good (exploitation, high immediate quality but limited learning), or responses it is uncertain about (exploration, potentially lower immediate quality but greater information gain). A sample-efficient agent actively chooses the dueling pairs that maximize learning per query.

The paper sits at the intersection of these two requirements. As Table 2 in the paper shows, most prior work satisfies at most one of them:

• Offline methods (DPO, IPO, SLiC applied to static datasets; Zhao et al., 2023; Rafailov et al., 2023; Azar et al., 2024) satisfy neither: they learn from a fixed dataset collected once with a reference policy, with no online interaction and no active selection of what to label.
• Iteratively online methods (Xu et al., 2023; Dong et al., 2024) partially satisfy Property 1 by doing a few rounds of re-sampling from the latest policy, but their batches are large and infrequent, reducing the benefit of online learning. They also lack active exploration.
• Passively online methods (Guo et al., 2024, in their OAIF framework) satisfy Property 1 fully—training continuously on on-policy data—but still use passive exploration: they simply sample both responses for the duel from the current policy $\pi_\theta$ and assume the resulting preference data will be informative. As shown in Figure 5, this yields improvements over offline training, but leaves substantial sample efficiency on the table.
• Active exploration methods for reward models (Mehta et al., 2023; Das et al., 2024; Dwaracherla et al., 2024) satisfy Property 2 by using uncertainty-aware reward models to select which responses to compare, but they keep the proposal policy fixed. The LLM that generates the candidate responses is never updated (Figure 3c), so these methods can only learn a better reward model and use it for inference-time selection (e.g., Best-of-N reranking). This severely limits their potential: if the proposal policy cannot generate good responses, no amount of active reward learning will find them. As the paper notes,

"all these works primarily focus on learning uncertainty-aware RMs online without updating LLM policies. Therefore, all responses are sampled from a fixed proposal policy $\pi_\beta$ (or even a fixed dataset), making the data coverage a critical concern."

• Active exploration with policy updates, but limited implementation (Zhang et al., 2024a; Xie et al., 2024; Muldrew et al., 2024) attempt to satisfy both properties, but face two limitations. First, their methods are tightly coupled to the DPO loss function—they add optimistic bias terms to DPO or use DPO's implicit reward margin for selection—making them incompatible with other direct optimizers like IPO or SLiC. Second, their experiments are limited to "a few online iterations" rather than fully online training (discussed in Section 3), which the paper attributes to implementation difficulty. This batch-iterative setup creates a mismatch with the theoretical guarantees that require many online interaction rounds.

Only three methods—Zhang et al. (2024a), Xie et al. (2024), and Muldrew et al. (2024)—appear at the bottom of Table 2 as satisfying both Properties 1 and 2, and the paper reproduces them under fully online conditions for fair comparison (Section 6.1).

A Unifying Framework: Contextual Dueling Bandits

A significant portion of the paper's intellectual contribution is framing LLM alignment as a Contextual Dueling Bandit (CDB) problem (Section 2), which provides a precise mathematical vocabulary for discussing sample efficiency. The CDB framework describes an agent that, at each round $t$:

1. Receives a context $c_t$ (in alignment, a prompt $x_t$ sampled from a prompt distribution $p_\mathcal{X}$).
2. Selects two actions $(a_t, a'_t)$ (two text responses $y_t, y'_t$) for comparison.
3. Receives stochastic feedback $z_t \sim \text{Ber}\left(P(a_t \succ a'_t | c_t)\right)$ (a binary human preference label following the Bradley-Terry model, where $P(y_t \succ y'_t | x_t) = \sigma(r^\star(x_t, y_t) - r^\star(x_t, y'_t))$, with $r^\star$ being an implicit human reward function).
4. Suffers immediate regret $R_t = P(a^\star_t \succ a_t | c_t) + P(a^\star_t \succ a'_t | c_t) - 1$, where $a^\star_t$ is the optimal action given full knowledge of $P$.

The agent's goal is to learn a von Neumann winner policy $\pi^\star$ that beats or ties every other policy on average (Equation 2). This framing is valuable not only for conceptual clarity but because it reveals that there are actually two distinct learning objectives hidden within the alignment problem, which call for different exploration strategies:

• Explore-and-Exploit (E&E): This applies to scenarios where the LLM is serving real users online, and the quality of every response matters. The objective is to minimize cumulative regret $\sum_{t=1}^T R_t$—each suboptimal response shown to a user incurs a cost. For example, when ChatGPT asks users to choose between two responses (see Figure 10 in Appendix E for a screenshot), it must keep both responses reasonably good while still learning. This requires balancing exploration with immediate quality.

• Best Arm Identification (BAI): This applies to crowdsourcing scenarios where annotators are paid to provide feedback. The objective is to find the optimal policy with minimum labeling cost, while the quality of responses shown during data collection is irrelevant as long as the collected preferences are information-rich. This allows for more aggressive exploration—the agent can intentionally show mediocre or uncertain response pairs because the cost is purely financial, not measured in user experience.

The paper emphasizes that these two settings are not just academic distinctions; they map directly onto real deployment modes, and the exploration strategy should be matched to the objective. In the E&E setting (Section 4.1, Algorithm 1 Line 5), both responses in a duel should maximize the sampled reward function—this guarantees reasonable quality per response but can lead to poor asymptotic performance for BAI because confidently suboptimal but reward-maximizing responses may be chosen repeatedly, preventing exploration of potentially better regions. In the BAI setting (Line 6), the second response should maximize uncertainty about the preference relative to the first response—pointing the agent toward maximally informative comparisons, even at the cost of showing lower-quality responses.

Where Existing Exploration Strategies Go Wrong

The paper identifies several specific failings in prior approaches to active exploration for LLMs (Section 3, final paragraphs):

Objective mismatch in Dwaracherla et al. (2024). This prior work applies Double Thompson Sampling (DTS), which is designed for the E&E setting (minimizing cumulative regret), but evaluates the method on anytime average performance as in the BAI setting. The paper argues this is a mismatch: an algorithm optimized for E&E will not be optimal for BAI, because

"sub-optimal responses with confidently high rewards might be tried for a long time at the expense of not exploring other potentially better choices" (Section 4.1).

In the E&E setting, exploiting a known-good response makes sense because each round incurs real cost. In the BAI setting, the algorithm should be willing to sacrifice per-round quality to more rapidly identify the best overall policy.

Pure exploration is not optimal for BAI. Das et al. (2024) selects dueling pairs that maximize epistemic uncertainty about the preference outcome, a pure exploration strategy. The paper later shows (Section 6.3, Figure 7) that this "Uncertainty" strategy indeed achieves the worst online (E&E) performance of three compared strategies. But even for BAI, it is outperformed by the paper's BAI-TS strategy, which balances both reward maximization and information gain. The paper's conclusion is nuanced:

"exploration with both reward and information maximization is better than exploration with only information maximization" (Section 6.3).

Tight coupling to DPO. Methods like XPO (Xie et al., 2024) and APL (Muldrew et al., 2024) modify the DPO loss function or use DPO's implicit reward signal for exploration. This coupling means the exploration strategy cannot be used with other direct optimizers (IPO, SLiC) that have different loss formulations or reward parameterizations. In contrast, SEA uses an explicit epistemic reward model separate from the policy, making it optimizer-agnostic.

The Implementation Gap: Why Fully Online Active Alignment Hasn't Been Properly Tested

An important contextual motivation that the paper surfaces is a systems engineering gap. The paper observes that many prior works claiming to do online active alignment are in practice limited to "a few iterations of batch learning" because a performant, open-source system for continuous online LLM alignment simply did not exist (Section 5.1). This creates a disconnect between theory and practice:

"The absence of a performant open-source online alignment system has restricted many existing works to only a few iterations of batch learning... which creates a mismatch with their theories that typically require a large number of online interaction rounds."

The computational bottleneck is two-fold: (1) autoregressive response generation is slow (the actor workload), and (2) preference labeling by large reward models (used as simulated oracles in experiments) is also slow. Standard implementations like HuggingFace's TRL online DPO trainer run all three stages (generation, labeling, learning) sequentially on the same hardware, leading to severe underutilization.

The paper positions its open-source system oat (online alignment tool) as a necessary enabler for the research itself, drawing on distributed deep RL architectures (Espeholt et al., 2018) to decouple actors (vLLM for fast generation), oracles (Mosec for parallelized reward model serving), and learners (DeepSpeed ZeRO for memory-efficient training) into separate, scalable workloads. This system-engineering contribution makes it possible to run fully online training over 50,000 interaction rounds across multiple model scales and seeds—which the authors note is, to their knowledge, the first such experimental validation of active exploration for online LLM alignment.

Summary: The Paper's Positioning

In one sentence: SEA is a unified Thompson sampling algorithm for LLM alignment that jointly learns both an epistemic reward model (for active exploration) and a generative policy (for improved response proposals) in a fully online loop, with the exploration strategy adaptively matched to whether the feedback source is online users (E&E) or paid annotators (BAI).

The paper positions itself as filling the gap marked by the bottom row of Table 2: methods that satisfy all three desiderata—online interaction (Property 1), active exploration (Property 2), and a continuously updated proposal policy $\pi_\theta$—but without the tight coupling to a specific loss function that limits prior work. The key enabling insights from bandit theory are that Thompson sampling provides a natural mechanism for balancing exploration and exploitation through posterior sampling, and that the second duel response should be selected differently depending on whether the objective is regret minimization (E&E) or policy identification (BAI).

3. Technical Approach

3.1 Reader Orientation

This paper develops a practical online learning agent that selects which pairs of text responses to show a human for comparison, uses the resulting preference labels to jointly train both a reward model and the LLM itself, and adapts its selection strategy depending on whether the goal is serving real users (where every response must be good) or collecting labeled data from paid annotators (where only the final policy quality matters). The "shape" of the solution is a bandit-inspired active exploration loop: the agent maintains an ensemble of reward models to estimate uncertainty about human preferences, uses that uncertainty to pick informative duel pairs, trains the LLM on a mix of real human labels and synthetic labels from its own reward ensemble, and continuously updates all components as new feedback arrives — all running in a distributed system that decouples generation, labeling, and learning into parallel workloads.

3.2 Big-Picture Architecture (Diagram in Words)

The SEA system has four major interacting components arranged in a continuous loop (Figure 3d, Algorithm 2):

1. The LLM Policy ($\pi_\theta$): A generative language model (Pythia 1B/2.8B/6.9B with supervised fine-tuning on TL;DR as initialization) that serves two roles. It is the proposal distribution — for each incoming prompt, it generates $M = 20$ candidate responses at temperature $\eta = 0.7$ that form the pool of possible dueling responses. It is also the final product — after alignment, this model should produce responses that humans prefer. The policy is continuously updated via a direct preference optimization loss (DPO, IPO, or SLiC) using a mixture of real human-labeled data and synthetic data from the epistemic reward model.

2. The Epistemic Reward Model (ERM, $R_\Phi$): An ensemble of $K = 20$ MLP heads sitting on top of a frozen 0.4B transformer (Jiang et al., 2023). Each MLP has 2 hidden layers of 128 nodes. The ERM estimates the implicit human reward function $r^\star$ by maintaining a distribution over plausible reward functions. Crucially, because it is an ensemble of independently-trained predictors, the variance across ensemble members captures epistemic uncertainty — the learner's uncertainty about what the true reward function is, which shrinks as more data is collected. The ERM is continuously trained on the growing buffer of human preference labels using a regularized negative log-likelihood loss (Equation 8).

3. The Duel Selector (Active Exploration Mechanism): For each prompt $x_t$, the agent samples $M$ candidate responses from the current policy $\pi_{\theta_{t-1}}$, forming a candidate set $\mathcal{S}_t$. It then selects two responses $(y_t, y'_t)$ for the duel according to one of two strategies:

   • E&E strategy: Sample one reward function $r_\phi \sim \text{Uniform}(\Phi_{t-1})$ from the ERM ensemble. Both $y_t$ and $y'_t$ are chosen to maximize this single sampled reward, i.e., $y_t = \arg\max_{y \in \mathcal{S}_t} r_\phi(x_t, y)$ and similarly for $y'_t$ (with the constraint $y'_t \neq y_t$). This is standard Thompson sampling — both responses exploit a plausible reward function, but because the sampled function is drawn from the posterior, the selection naturally explores uncertain regions.
   • BAI strategy: $y_t$ is selected via Thompson sampling as above. But $y'_t$ is instead chosen to maximize the variance of the preference probability over the first response: $y'_t = \arg\max_{y \in \mathcal{S}_t} \mathbb{V}_\phi\left[\sigma(r_\phi(x_t, y_t) - r_\phi(x_t, y))\right]$, where the variance is computed across the $K$ ensemble members. This selects the second response that the agent is maximally unsure about relative to the first — it is the comparison that will be most informative for reducing uncertainty.

4. The Preference Oracle ($P$): In the experiments, this is a strong frozen scalar reward model (Skywork-Reward-Llama-3.1-8B, top-ranked on RewardBench) that simulates human feedback. For each duel, it returns a binary preference label $z_t$ indicating which response is better (or equivalently, $(y_t^+, y_t^-)$ — the winning and losing responses). In the LLM-as-a-judge experiments (Section 6.4), the oracle is replaced by GPT-4o-mini queried through the OpenAI API.

Information flow per round (following Algorithm 2):

1. Prompt arrives: $x_t \sim p_\mathcal{X}$ is sampled from the TL;DR prompt distribution.
2. Candidate generation: The current policy $\pi_{\theta_{t-1}}$ generates $M = 20$ responses $\{y_t^i\}_{i=1}^M$ at temperature 0.7, forming the candidate set $\mathcal{S}_t$.
3. Duel selection: The active exploration mechanism (E&E or BAI variant) selects two responses $(y_t, y'_t)$ from $\mathcal{S}_t$ using the ERM $R_{\Phi_{t-1}}$.
4. Oracle labeling (with probability $\gamma$): With probability $\gamma = 0.7$ (after a burn-in of 1,000 samples where $\gamma = 1$), the selected dueling pair is sent to the preference oracle $P$, which returns the winning response $y_t^+$ and losing response $y_t^-$. This labeled triplet $(x_t, y_t^+, y_t^-)$ is added to the real experience buffer $\mathcal{D}_t$.
5. ERM pseudo-labeling (with probability $1 - \gamma$): With probability 0.3, the oracle is not queried. Instead, a randomly sampled ensemble member $r_{\phi_k} \in \Phi_{t-1}$ provides a synthetic preference label $(\tilde{y}_t^+, \tilde{y}_t^-)$ for the same dueling pair. This synthetic labeled data $\mathcal{B}^{\text{ERM}}_t$ is mixed into the policy training batch but is not added to $\mathcal{D}_t$ — the ERM is only trained on real oracle labels.
6. ERM update: The ERM ensemble $\Phi_{t-1}$ is updated to $\Phi_t$ by taking $m = 5$ gradient steps on randomly sampled batches from $\mathcal{D}_t$ using the regularized NLL loss (Equation 8). Each ensemble member is trained independently with its own optimizer state.
7. Policy update: The policy $\pi_{\theta_{t-1}}$ is updated to $\pi_{\theta_t}$ by one gradient step on a mixed batch $\mathcal{B}^{\text{mix}}_t$ that combines $\gamma$-weighted real oracle data and $(1-\gamma)$-weighted ERM pseudo-labeled data, using the chosen DAP loss (DPO/IPO/SLiC). The reference policy $\pi_{\text{ref}}$ is fixed to $\pi_{\text{sft}}$ (the initial supervised fine-tuned model).

What makes this different from prior architectures:

• Unlike RLHF (Figure 3a), there is no separate "reward model training then RL" phase — the ERM and policy are updated concurrently and continuously.
• Unlike passive online DAP (Figure 3b), the dueling responses are not simply two samples from $\pi_\theta$ but are actively selected using the ERM's uncertainty.
• Unlike fixed-proposal active exploration (Figure 3c), the proposal policy $\pi_\theta$ improves over time, expanding the set of candidates the ERM can explore.

3.3 Roadmap for the Deep Dive

I will explain the technical approach in this order:

• First, the CDB formalization and regret definition (Section 2 material not covered in the prior sections) — because the two learning objectives (E&E vs BAI) and their different exploration strategies only make sense once we understand what is being optimized and how optimality is defined.
• Second, the Thompson sampling algorithm at the conceptual level (Algorithm 1) — this establishes the principles that the practical implementation approximates, and explains why the duel selection differs between E&E and BAI.
• Third, the three practical components one by one — the epistemic reward model (how uncertainty is modeled and updated), policy-guided search (how arg max over the discrete text space is approximated), and mixed preference learning (how the policy is trained and why pseudo-labels help).
• Fourth, the full practical algorithm (Algorithm 2) — synthesizing the three components into the end-to-end SEA agent, with concrete hyperparameters and design choices.
• Finally, the distributed learning system — because the engineering architecture (Actor-Learner-Oracle) is essential for making fully online active alignment computationally feasible at scale, and the paper treats it as a significant contribution.

This order builds from abstract principles to concrete implementation, so the reader understands why each component exists before seeing how it is built.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper with a strong systems engineering contribution. The core algorithmic idea is that Thompson sampling — a well-established bandit algorithm — can be practically instantiated for LLM alignment by approximating three intractable operations: (1) maintaining and sampling from a reward posterior (solved via deep ensembles), (2) computing $\arg\max_{y \in \mathcal{Y}}$ over the discrete text space (solved via policy-guided candidate generation plus greedy search within the candidate set), and (3) converting a reward-centric agent into a generative policy (solved by training the policy with direct preference optimization on data from the exploration process). The secondary contribution is the distributed system that makes fully online training computationally tractable, which the paper argues is a prerequisite for proper empirical evaluation of online active alignment methods.

---

Contextual Dueling Bandits: Formalizing the Alignment Problem

Before explaining the algorithm, we need the full mathematical scaffolding from Section 2.1. A Contextual Dueling Bandit (CDB) problem is defined by a tuple $(\mathcal{C}, \mathcal{A}, P)$, where $\mathcal{C}$ is the context space (in alignment, the set of all possible text prompts), $\mathcal{A}$ is the action space (the set of all possible text responses, which is discrete and astronomically large because it includes all token sequences of any length), and $P : \mathcal{A} \times \mathcal{A} \times \mathcal{C} \mapsto [0, 1]$ is the unknown preference oracle. The oracle $P(a \succ a' | c)$ gives the probability that action $a$ is preferred over $a'$ in context $c$.

At each round $t$:

1. A context $c_t \sim p_\mathcal{C}$ is presented (in LLM terms: a prompt $x_t$ is sampled from the prompt distribution $p_\mathcal{X}$).
2. The agent selects two actions $a_t, a'_t \in \mathcal{A}$ (two text responses $y_t, y'_t$).
3. The environment returns stochastic feedback $z_t \sim \text{Ber}(P(a_t \succ a'_t | c_t))$ (a binary preference label: $z_t = 1$ means $a_t$ won, $z_t = 0$ means $a'_t$ won).
4. The agent suffers immediate regret:

$R_t = P(a^\star_t \succ a_t | c_t) + P(a^\star_t \succ a'_t | c_t) - 1$

where $a^\star_t$ is the best action the agent would take at round $t$ if it had complete knowledge of $P$ — formally, an action satisfying $P(a^\star_t \succ a | c_t) \geq \frac{1}{2}$ for all $a \in \mathcal{A}$.

What it computes: For each of the two actions the agent selected, compute the probability that the optimal action would beat it. Sum these two probabilities. If both selected actions are exactly as good as the optimal action, then $P(a^\star_t \succ a_t | c_t) = \frac{1}{2}$ and similarly for $a'_t$, so regret is $\frac{1}{2} + \frac{1}{2} - 1 = 0$. If both selected actions are maximally bad (the oracle always prefers $a^\star_t$ over them, so $P(a^\star_t \succ a_t | c_t) = 1$), regret is $1 + 1 - 1 = 1$. The regret measures how much worse the agent's selected pair is compared to the best possible pair.

Why this form: The $-1$ term normalizes so that zero-regret means the agent is performing identically to the optimal policy. Without it, even optimal actions would register $\frac{1}{2} + \frac{1}{2} = 1$ total. The form captures the intuition that a dueling bandit agent must select two actions, and both matter — showing a great response alongside a terrible one still incurs regret because users see both.

Bradley-Terry assumption. The paper assumes that the preference oracle follows the Bradley-Terry (BT) model, which is standard in the LLM alignment literature:

$P(y_t \succ y'_t | x_t) = \frac{\exp(r^\star(x_t, y_t))}{\exp(r^\star(x_t, y_t)) + \exp(r^\star(x_t, y'_t))} = \sigma(r^\star(x_t, y_t) - r^\star(x_t, y'_t))$

where $r^\star : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ is the human's implicit reward function (unknown to the agent) and $\sigma(\cdot)$ is the sigmoid function $\sigma(z) = 1 / (1 + e^{-z})$.

What it computes: The probability that response $y_t$ beats response $y'_t$ depends only on the difference in their reward scores under the implicit human reward function. If $r^\star(x_t, y_t) \gg r^\star(x_t, y'_t)$, the probability approaches 1. If they are equal, the probability is exactly $\frac{1}{2}$. The sigmoid squashes the difference to a valid probability.

Why this form: The BT model reduces the comparison to a scalar reward, which makes it possible to learn a reward function from pairwise comparisons (you only need to predict which of two responses has higher reward, not the absolute reward value). It also makes the regret definition cleaner: under the BT assumption, the immediate regret can be rewritten as $R_t = r^\star(x_t, y^\star_t) - (r^\star(x_t, y_t) + r^\star(x_t, y'_t)) / 2$, where $y^\star_t$ is the response with highest $r^\star$ for prompt $x_t$. This directly measures the reward gap between optimal and selected responses.

The von Neumann winner policy. The solution concept is the von Neumann winner (Dudík et al., 2015). A policy $\pi \in \Delta_\mathcal{X}^\mathcal{Y}$ (a mapping from prompts to distributions over responses) is optimal if it beats or ties every other policy on average:

$\forall \pi' \in \Delta_\mathcal{X}^\mathcal{Y}, \quad \mathbb{E}_{x \sim p_\mathcal{X}}\left[\mathbb{E}_{y \sim \pi(\cdot|x)} \mathbb{E}_{y' \sim \pi'(\cdot|x)} \left[P(y \succ y' | x)\right]\right] \geq \frac{1}{2}$

What it states: The optimal policy $\pi^\star$ is such that when you sample a prompt $x$, then sample one response from $\pi^\star$ and one from any competitor policy $\pi'$, the probability that $\pi^\star$'s response wins is at least 1/2. This is a game-theoretic equilibrium notion: no other policy can consistently beat it.

Why this form: In dueling bandits, there is no absolute reward scale — only relative comparisons. So we cannot define optimality as "maximizing expected reward" in the usual bandit sense. The von Neumann winner provides a well-defined notion of optimality based purely on pairwise win probabilities. Under the BT model, the von Neumann winner is simply $\pi^\star = \arg\max_\pi \mathbb{E}_{x \sim p_\mathcal{X}} \mathbb{E}_{y \sim \pi(\cdot|x)} [r^\star(x, y)]$ — the policy that maximizes expected reward under the implicit human reward function. This connects the dueling bandit formulation back to standard reward maximization, which is what RLHF and DAP methods optimize.

Two learning objectives. The paper distinguishes two settings based on how regret is evaluated:

• Explore-and-Exploit (E&E): The objective is to minimize cumulative regret $\sum_{t=1}^T R_t$. This is the standard online learning objective: every round matters, because real users see every response. The agent must trade off exploring uncertain responses (which might be worse) against exploiting known good responses. The paper associates this with "aligning from online users' feedback" — e.g., ChatGPT asking users to compare two responses while both must be reasonable.

• Best Arm Identification (BAI): The objective is anytime regret — the agent's average performance evaluated at any round $t$, without caring about the quality of intermediate selections. This allows pure exploration: the agent can intentionally show suboptimal response pairs if they yield informative feedback. The paper associates this with "aligning from crowdsourcing" — hiring annotators and wanting to minimize total labeling cost to reach a good policy.

The distinction is not about different algorithms at the final policy level, but about what the agent optimizes during data collection. In both settings, the agent learns a policy $\pi_\theta$ and an ERM $R_\Phi$. The difference is only in how the second dueling response $y'_t$ is selected (Algorithm 1, Lines 5 vs 6), because the selection criterion encodes what the agent cares about during training.

---

The Conceptual Thompson Sampling Algorithm (Algorithm 1)

Thompson sampling (TS) is a Bayesian bandit algorithm with a deceptively simple recipe: at each round, sample a reward function from your posterior distribution over reward functions, then act greedily with respect to that sampled function. The intuition: when your posterior is uncertain (early in learning), the sampled reward function will differ substantially from the true reward, causing you to explore actions that might be good. As you collect more data, the posterior concentrates around the true reward, so the sampled function approximates the truth and you increasingly exploit.

Adapting TS to dueling bandits with contexts. Algorithm 1 adapts standard TS to the dueling bandit setting with two duel responses. The key design question is: how should the second response be chosen, given that the first was selected by standard TS?

The answer depends on the objective:

Line 4 (common to both settings): Select $y_t$ via standard Thompson sampling.

Sample $r \sim p(r | \mathcal{D}_{t-1})$ and set $y_t \leftarrow \arg\max_{b \in \mathcal{Y}} r(x_t, b)$.

What it does: Draw one reward function from the current posterior distribution over reward functions (the agent's belief state about $r^\star$ given all data seen so far). Then find the response $y_t$ that maximizes this sampled reward function for the current prompt $x_t$. Because the sampled $r$ is a random draw from the posterior, $y_t$ is a plausible "best response" under the agent's current uncertainty.

Why this form: This is textbook Thompson sampling. The randomness comes from sampling the reward function, not from randomizing the action selection (the action selection is deterministic greedy with respect to the sampled $r$). This concentrates exploration naturally: when the posterior is diffuse (high uncertainty), different samples produce very different argmax responses, so the agent explores broadly. When the posterior is peaked (low uncertainty), most samples agree on the argmax, so the agent exploits.

Line 5 (E&E setting): Select $y'_t$ by repeated Thompson sampling until a different response appears.

Repeat: Sample $r \sim p(r | \mathcal{D}_{t-1})$ and set $y'_t \leftarrow \arg\max_{b \in \mathcal{Y}} r(x_t, b)$. Until $y'_t \neq y_t$.

What it does: Keep drawing reward functions from the posterior and computing their argmax until you find one whose "best response" differs from $y_t$. Both $y_t$ and $y'_t$ are then plausible best responses under different draws from the posterior. The dueling comparison asks: which of these two plausible best responses does the human actually prefer?

Why this form: In the E&E setting, every response shown to a user should be "exploitative" in the sense that it maximizes some plausible reward function. This ensures both $y_t$ and $y'_t$ are reasonable candidates — neither is intentionally bad. However, this algorithm can have "poor asymptotic performance for BAI problems" (Section 4.1) because it tends to repeatedly compare responses that are already known to be good, rather than exploring to discover whether truly better responses exist. The paper cites Russo (2016) for this limitation: "sub-optimal responses with confidently high rewards might be tried for a long time at the expense of not exploring other potentially better choices."

Line 6 (BAI setting): Select $y'_t$ to maximize preference uncertainty given $y_t$.

Set $y'_t \leftarrow \arg\max_{b \in \mathcal{Y}} \mathbb{V}\left[\sigma(r(x_t, y_t) - r(x_t, b))\right]$, where $\mathbb{V}[\cdot]$ computes variance over the posterior $p(r | \mathcal{D}_{t-1})$.

What it does: For each candidate response $b$, compute the variance (across posterior samples of $r$) of the preference probability $\sigma(r(x_t, y_t) - r(x_t, b))$ — the probability that the first response $y_t$ beats $b$. Select the $b$ that maximizes this variance. This identifies the comparison that the agent is maximally unsure about: the posterior distribution over who would win this duel is the widest, meaning the comparison outcome is the most informative for reducing uncertainty about $r^\star$.

Why this form: In the BAI setting, response quality during data collection is irrelevant — the goal is to identify the best policy with minimal labeling. So the agent should choose dueling pairs that yield maximal information gain about the reward function. The variance of the preference probability directly measures epistemic uncertainty: if all posterior samples of $r$ agree that $y_t$ beats $b$, the variance is near zero (the agent is certain about this comparison). If the samples disagree sharply (some think $y_t$ wins, others think $b$ wins), the variance is high (the agent is uncertain, so labeling this comparison will be informative). This is a form of uncertainty sampling tailored to dueling feedback.

The paper notes that this is related to but different from pure exploration (Das et al., 2024), which would select both $y_t$ and $y'_t$ purely to maximize uncertainty without anchoring one of them via TS. The BAI-TS approach balances reward-seeking (via the TS first response) with information-seeking (via the variance-maximizing second response), which the paper later shows empirically outperforms pure uncertainty maximization (Section 6.3).

Why Algorithm 1 is intractable for LLMs (three reasons from Section 4.1). The paper explicitly acknowledges that Algorithm 1 cannot be implemented directly:

1. Posterior sampling is intractable. Maintaining and sampling from a full posterior distribution over reward functions is computationally infeasible for large transformer-based reward models. Exact Bayesian inference is impossible, and even approximate methods like MCMC or variational inference would be prohibitively expensive at LLM scale.
2. The $\arg\max$ over $\mathcal{Y}$ is intractable. The response space $\mathcal{Y}$ is the set of all token sequences, which is discrete, combinatorially vast, and lacks structure that would enable efficient global optimization. Computing $\arg\max_{b \in \mathcal{Y}} r(x_t, b)$ requires searching over all possible strings, which is impossible.
3. The algorithm is centered around a reward posterior, not a generative model. The output of Algorithm 1 is a reward distribution $p(r|\mathcal{D})$. But what we actually need is a generative language model $\pi_\theta$ that can produce good responses given a prompt. The reward posterior alone does not directly give us a sampling policy — we need to convert reward knowledge into a model that generates text.

The practical SEA algorithm (Algorithm 2) addresses each of these three intractabilities with a specific approximation technique: deep ensembles for (1), policy-guided search for (2), and mixed preference learning for (3).

---

Epistemic Reward Model for Posterior Sampling (Addresses Intractability 1)

To approximate posterior sampling over reward functions, the paper uses deep ensembles (Lakshminarayanan et al., 2017). The core idea is simple: instead of maintaining a single best reward model, train multiple reward models independently, and treat the distribution of their predictions as a proxy for the posterior. Sampling from the "posterior" then simply means randomly picking one ensemble member and using its predictions.

Architecture. The ERM $R_\Phi$ consists of $K = 20$ independent MLP reward heads $\{\phi_k\}_{k=1}^K$, all sharing the same frozen transformer backbone from a pretrained 0.4B parameter model (Jiang et al., 2023). The base transformer produces a contextualized representation of the prompt-response pair, and each MLP head maps that representation to a scalar reward estimate. Each MLP has 2 hidden layers of 128 nodes. The transformer is not fine-tuned during ERM training — only the MLP heads are updated. This is a deliberate design choice to keep training efficient, since the 0.4B backbone is already a reasonable representation for the task, and training only the heads means the ERM can be updated quickly (5 gradient steps per round) without massive computational cost.

Why an ensemble models epistemic uncertainty. The key property exploited by deep ensembles is that independently initialized and independently trained models tend to agree on in-distribution inputs but disagree on out-of-distribution inputs. At the beginning of training, all ensemble members see different random subsets of the data (due to random batch sampling) and have different random initializations, so their predictions vary widely — reflecting high epistemic uncertainty. As training proceeds and all members see more data covering the relevant regions of response space, their predictions converge — reflecting reduced uncertainty. The variance across ensemble members at any point estimates how much the model's prediction would change if it had been trained on a slightly different dataset, which is exactly what epistemic uncertainty captures.

Training loss. Each ensemble member $\phi_k$ is trained to minimize the standard reward modeling negative log-likelihood (Equation 5), plus a regularization term:

$\mathcal{L}_R(\Phi_t | \mathcal{D}_t) = \sum_{k=1}^K \left(\mathcal{L}_r(\phi^t_k | \mathcal{D}_t) - \lambda ||\phi^t_k - \phi^0_k||\right)$

where $\mathcal{L}_r(\phi | \mathcal{D}) = -\mathbb{E}_{(x, y^+, y^-) \sim p_\mathcal{D}}\left[\log \sigma\left(r_\phi(x, y^+) - r_\phi(x, y^-)\right)\right]$ is the standard Bradley-Terry NLL, $\phi^0_k$ is the initial random weight vector for head $k$, $\phi^t_k$ is its weight vector after $t$ rounds of training, and $\lambda = 0.5$ controls the regularization strength.

What it computes: For each ensemble member $k$, compute two terms and sum them. The first term is the standard preference learning loss: for every preference triplet in the batch $\mathcal{D}_t$, predict the log-probability that the winning response $y^+$ has higher reward than the losing response $y^-$, and minimize the negative of that log-probability. This encourages the reward model to correctly rank response pairs. The second term is an L2 penalty on how far the current weights have drifted from their initial random values. The total loss is the sum of these two terms across all $K$ ensemble members.

Why this form — the regularization term specifically: The $-\lambda ||\phi^t_k - \phi^0_k||$ term is crucial for maintaining diversity across ensemble members. Without it, all $K$ heads might converge to similar solutions (since they all see the same data distribution $\mathcal{D}_t$), collapsing the ensemble and destroying the uncertainty signal. The regularization toward each head's distinct random initialization $\phi^0_k$ acts as a "centrifugal force" — each head is pulled toward its own unique starting point, preventing them from all converging to the same weights. The paper cites Dwaracherla et al. (2024) for this technique. The coefficient $\lambda = 0.5$ was chosen after "a coarse hyperparameter search" (Appendix D).

Why ensemble size $K = 20$: The paper does not provide an ablation over $K$, but 20 is a common choice in the deep ensemble literature that balances computational cost (training 20 heads vs. 1) against uncertainty estimation quality. Each head is small (2-layer MLP with 128 hidden units), so the computational overhead of the ensemble is modest relative to the shared transformer backbone.

Incremental online update. At each round $t$, the ERM is updated by taking $m = 5$ gradient steps on randomly sampled batches from the growing experience buffer $\mathcal{D}_t$. The paper states that $m = 5$ "suffices to achieve reasonable accuracy" (Algorithm 2, Line 9). This incremental update scheme is essential for the online nature of the algorithm — the ERM's uncertainty estimates must track the growing dataset, shrinking as more comparisons are collected. A static ERM trained once on a fixed dataset would not provide the dynamic uncertainty signal needed for active exploration to naturally decay from exploration to exploitation.

Key design choice: the ERM is a separate, smaller model, not a head on the policy model. This is a deliberate architectural separation. The ERM uses a 0.4B frozen transformer — much smaller than the policy models (1B, 2.8B, 6.9B) and different from the oracle RM (8B). This reflects the realistic assumption that "human preferences can be more complex than what the agent can model" (Appendix D). In a real deployment, the agent's internal model of human preferences is necessarily approximate; the oracle (actual human or strong RM) represents the true but inaccessible preference function. The ERM's epistemic uncertainty thus captures not just parameter uncertainty given limited data, but also model mismatch — the gap between what the agent can represent and the true preference structure.

---

Policy-Guided Search to Approximate arg max (Addresses Intractability 2)

The Thompson sampling steps in Algorithm 1 require computing $\arg\max_{b \in \mathcal{Y}} U(b)$ — finding the text response that maximizes some utility function $U$ over the space of all possible strings. This is fundamentally a discrete optimization over an exponentially large, unstructured space. The paper's solution is conceptually simple: replace global optimization with local search over a candidate set generated by the current policy.

Step 1: Generate candidates from the policy. For each prompt $x_t$, sample $M = 20$ responses from the current policy $\pi_{\theta_{t-1}}(\cdot | x_t)$ at temperature $\eta = 0.7$. These $M$ responses form the candidate set $\mathcal{S}_t = \{y_t^i\}_{i=1}^M$. The intuition: the policy $\pi_{\theta_{t-1}}$ has been trained (via DAP) to favor responses that have high reward under what it has learned so far. Therefore, sampling from it naturally produces candidates that are biased toward high-reward regions of the response space — which is exactly where $\arg\max$ wants to search.

Step 2: Greedy selection within the candidate set. Given a utility function $U$ (which could be a sampled reward $r_\phi$ from the ERM ensemble, or a variance-based acquisition function), simply evaluate $U$ on each candidate in $\mathcal{S}_t$ and pick the one with the highest value: $y_t = \arg\max_{y \in \mathcal{S}_t} U(y)$. This is equivalent to taking the temperature $\eta \to 0$ limit of the policy-guided distribution $\pi_{\text{prior}}(\cdot | x_t) \exp(U(\cdot) / \eta)$ — as $\eta$ shrinks, the distribution concentrates on the highest-utility candidate.

Why this works (and why it might fail). The paper frames this approximation through the lens of sampling from a policy-guided distribution. Formally, sampling from $\pi_{\text{prior}}(y|x) \exp(U(y)/\eta)$ is appropriate because "it favors responses $y$ that approximately maximize $U(y)$" (Section 4.2.2). When $\eta \to 0$, this becomes exact greedy selection.

The quality of this approximation depends critically on two factors:

1. The prior policy $\pi_{\text{prior}}$ must place probability mass on high-utility regions. If the policy never generates responses that are actually good, then even exhaustive search over $\mathcal{S}_t$ will not find the true $\arg\max$. This is why the LLM policy is updated continuously (Section 4.2.3) — as $\pi_\theta$ improves, the candidate set $\mathcal{S}_t$ shifts toward regions of higher true reward, and the approximation quality improves over time.

2. The candidate set size $M$ must be large enough to cover promising candidates. With $M = 20$, the search can only choose among 20 options. If the optimal response is rare under $\pi_\theta$ (e.g., appears with probability 0.001), then with $M = 20$, it will almost never appear in $\mathcal{S}_t$, and the search cannot find it regardless of how good $U$ is. The paper does not ablate $M$, but acknowledges this limitation implicitly: the approximation replaces $\arg\max_{b \in \mathcal{Y}}$ with $\arg\max_{b \in \mathcal{S}_t}$, where $\mathcal{S}_t$ is a small random subset.

Computational reuse. The paper notes an important practical optimization: "We also reuse the same $\mathcal{S}_t$ for different $U$ functions at time $t$ to save computation" (Section 4.2.2). At each round, the policy generates $M$ responses once. The two dueling responses $y_t$ and $y'_t$ are then selected from this same set by evaluating different utility functions (e.g., for BAI, $y_t$ uses $U(y) = r_\phi(x_t, y)$ with a sampled $\phi$, while $y'_t$ uses $U(y) = \mathbb{V}_\phi[\sigma(r_\phi(x_t, y_t) - r_\phi(x_t, y))]$). This halves the generation cost compared to sampling separate candidate sets for each response.

Why not use reinforcement learning or search-based methods? The paper does not discuss alternatives like PPO with a KL penalty (which would require the ERM to provide dense rewards) or tree search over tokens (which would be prohibitively expensive for the text generation length considered). The policy-guided sampling approach is computationally lightweight — it requires only $M$ autoregressive generations and $M$ ERM forward passes per round, both of which are parallelizable.

---

Online Policy Learning from Mixed Preferences (Addresses Intractability 3)

The final intractability is that Algorithm 1 outputs a reward posterior, not a generative policy. To convert the knowledge encoded in the ERM and the collected preference data into an improved text generation model, the paper uses direct alignment from preferences (DAP) trained online on mixed data. This component solves two problems simultaneously: it produces a generative policy (the end product), and it provides the candidate distribution $\pi_{\text{prior}}$ that makes the policy-guided search effective.

The DAP loss (Equation 9). The policy $\pi_{\theta_t}$ is updated by minimizing a direct preference optimization loss on a batch of preference data:

$\mathcal{L}_\pi(\theta_t | \mathcal{B}_t, \pi_{\text{ref}}, \mathcal{F}) = \mathbb{E}_{(x, y^+, y^-) \sim p_{\mathcal{B}_t}}\left[\mathcal{F}_{\theta_t}(x, y^+, y^-, \pi_{\text{ref}})\right]$

where $\mathcal{B}_t$ is a batch of preference triplets (prompt, winning response, losing response) produced at round $t$, $\pi_{\text{ref}} = \pi_{\text{sft}}$ is the initial supervised fine-tuned model used as a reference for KL regularization, and $\mathcal{F}$ is any DAP loss function.

What it computes: For each preference triplet in the batch, compute the chosen DAP loss $\mathcal{F}$, which measures how well the current policy $\pi_{\theta_t}$ ranks the winning response higher than the losing response (relative to the reference policy $\pi_{\text{ref}}$). Average over the batch.

Why this form: The abstraction over $\mathcal{F}$ is important — SEA is not coupled to a specific DAP loss. The paper experiments with three variants: DPO (Equation 10), IPO (Equation 11), and SLiC (Equation 12). This is a key differentiator from XPO and APL, which modify the DPO loss specifically. By treating $\mathcal{F}$ as a pluggable module, SEA can work with any direct optimizer.

The three specific DAP losses used:

DPO (Rafailov et al., 2023):

$\mathcal{F}_\theta(x, y^+, y^-, \pi_{\text{ref}}) = -\log \sigma\left(\beta \log \frac{\pi_\theta(y^+|x)}{\pi_{\text{ref}}(y^-|x)} \frac{\pi_{\text{ref}}(y^+|x)}{\pi_\theta(y^-|x)}\right)$

where $\beta = 0.1$ controls the deviation from the reference policy.

What it computes: Inside the sigmoid, compute the log-ratio of the policy's relative preference for $y^+$ over $y^-$ versus the reference policy's relative preference. If the policy assigns much higher probability to $y^+$ (relative to $y^-$) than the reference does, the argument to the sigmoid is large and positive, so $\sigma(\cdot) \approx 1$ and the negative log is near 0 (low loss). If the policy does the opposite, the argument is negative, $\sigma(\cdot) \approx 0$, and the negative log is large (high loss).

Why DPO: DPO eliminates the need for an explicit RL step by reparameterizing the reward function in terms of the policy and reference model. It is the most widely-used DAP method, making it an important baseline to include.

IPO (Azar et al., 2024):

$\mathcal{F}_\theta(x, y^+, y^-, \pi_{\text{ref}}) = \left(\log \frac{\pi_\theta(y^+|x)}{\pi_{\text{ref}}(y^-|x)} \frac{\pi_{\text{ref}}(y^+|x)}{\pi_\theta(y^-|x)} - \frac{1}{2\beta}\right)^2$

where $\beta$ is tuned from $\{0.2, 0.3, 0.5, 1.0\}$ across scales.

What it computes: Compute the same log-ratio as DPO, then compute the squared difference between this log-ratio and a target value $1/(2\beta)$. IPO penalizes the policy when the log-ratio deviates from this target, rather than only penalizing when the log-ratio is negative (as DPO does). This makes IPO more robust to overfitting because it doesn't push the log-ratio to infinity — it pushes it to a fixed finite target.

Why IPO: IPO addresses a known issue with DPO where the loss can drive the policy to assign arbitrarily high relative probability to winning responses, causing overfitting (Azar et al., 2024). The squared loss prevents this by having a finite target.

SLiC (Zhao et al., 2023):

$\mathcal{F}_\theta(x, y^+, y^-, \pi_{\text{ref}}) = \max\left(0, 1 - \beta \log \frac{\pi_\theta(y^+|x)}{\pi_{\text{ref}}(y^-|x)} \frac{\pi_{\text{ref}}(y^+|x)}{\pi_\theta(y^-|x)}\right)$

where $\beta = 0.2$.

What it computes: Compute the same log-ratio, multiply by $\beta$, and apply a hinge loss with margin $1$. If $\beta$ times the log-ratio exceeds 1, the loss is zero (the policy sufficiently favors the winning response). If it is less than 1, the policy incurs a linear penalty.

Why SLiC: The hinge loss provides a margin: once the policy's preference for the winner over the loser is large enough (above the margin), no further gradient is applied. This can prevent over-optimization.

The mixture distribution $\mathcal{B}^{\text{mix}}_t$ (Equation not explicitly numbered, but described in Section 4.2.3). The critical innovation in the policy update is that $\mathcal{B}_t$ in the loss above is not purely oracle-labeled data. Instead, it is a mixture:

$p_{\mathcal{B}^{\text{mix}}_t} = \gamma \, p_{\mathcal{B}_t} + (1 - \gamma) \, p_{\mathcal{B}^{\text{ERM}}_t}$

where $\gamma = 0.7$ (after a burn-in of 1,000 samples where $\gamma = 1$), $\mathcal{B}_t = \{x_i, y_i^+, y_i^-\}_{i=1}^b$ contains preference data labeled by the real oracle $P$, and $\mathcal{B}^{\text{ERM}}_t = \{x_i, \tilde{y}_i^+, \tilde{y}_i^-\}_{i=1}^b$ contains synthetic preference data labeled by randomly sampled individual ERM ensemble members $r_{\phi_k}$.

What it computes: For each batch of size $b$ used to update the policy, approximately $\gamma \cdot b$ examples come from real human (or oracle) labels, and the remaining $(1-\gamma) \cdot b$ come from the ERM's own predictions on the same dueling pairs. Both types of data use the same DAP loss $\mathcal{F}$. The only difference is the source of the $(y^+, y^-)$ labels.

Why this form — two reasons:

1. Data amplification: The ERM provides "free" pseudo-labels, effectively increasing the amount of training data the policy sees without additional oracle queries. This directly improves sample efficiency — the policy learns from 1/0.7 ≈ 1.43× more data than if only oracle labels were used. The paper notes this connects to model-based RL (Appendix B): "learning from mixed preferences further boosts sample efficiency because it utilizes the internal ERM to get pseudo labels instead of querying humans. This relates closely to model-based RL."

2. Alignment of policy proposals with ERM uncertainty: A subtle but critical point. If the policy $\pi_\theta$ were trained only on oracle-labeled data, it would learn to favor responses with high true reward $r^\star$ — which is good for the final policy but can actually hurt exploration. The reason: the policy's candidate set $\mathcal{S}_t$ would be biased toward high-$r^\star$ regions, which might be regions where the ERM has low uncertainty (all ensemble members agree they are good). But active exploration needs $\mathcal{S}_t$ to also contain responses in high-uncertainty regions, because the variance-maximizing second response selector (BAI) or the diverse TS samples (E&E) need candidates that the ERM disagrees about. Training on ERM pseudo-labels encourages the policy to also propose responses that individual ensemble members think are good (even if the true oracle might disagree), which expands the candidate set to cover regions of high epistemic uncertainty. The paper states this explicitly (Section 4.2.3):

"optimizing $\pi_{\theta_t}$ only with oracle data can average out the epistemic uncertainty of $R$, hindering the exploration efficiency. To mitigate this issue, we further align $\pi_{\theta_t}$ with $R_{\Phi_t}$ using the same direct optimizer to encourage $\pi_{\theta_t}$ to propose high-$r_{\phi_t^k}$ responses for individual $r_{\phi_t^k}$, leading to better approximation of $\arg\max_{b \in \mathcal{Y}} r(x, b)$ for any sampled $r$."

In other words: the policy-guided search works better when the policy's output distribution covers the support of the ERM's uncertainty, not just the support of the true reward. Mixed preference training achieves this by making the policy chase individual ERM members' preferences.

Why $\gamma = 0.7$ with a burn-in: The burn-in period of 1,000 samples where $\gamma = 1$ (only real oracle data) ensures the ERM has reasonable accuracy before its pseudo-labels are trusted. After the burn-in, $\gamma = 0.7$ strikes a balance: most training data is still real (maintaining fidelity to the true preference signal), but a substantial fraction is synthetic (providing the exploration-alignment benefit). The paper does not ablate $\gamma$, so this value is presumably chosen empirically.

Important detail: ERM pseudo-labels are NOT added to $\mathcal{D}_t$. The synthetic data $\mathcal{B}^{\text{ERM}}_t$ is used only for the policy update (Step 10 in Algorithm 2). It is explicitly excluded from $\mathcal{D}_t$, the experience buffer used to train the ERM itself (Step 9). This prevents a dangerous feedback loop: if the ERM trained on its own predictions, errors would compound and the uncertainty estimates would become meaningless. The ERM only ever sees real oracle labels, preserving the fidelity of its uncertainty signal.

---

The Full Practical Algorithm (Algorithm 2)

Algorithm 2 synthesizes the three components into a concrete procedure. I will walk through each numbered step, connecting it to the conceptual Algorithm 1 and the practical components described above.

Inputs:

• $\pi_{\text{ref}}$: reference policy (the SFT model, fixed throughout training).
• $\mathcal{F}$: DAP loss function (DPO, IPO, or SLiC).
• $p_\mathcal{X}$: prompt distribution (the TL;DR training prompts).
• $P$: preference oracle (simulated by Skywork-Reward-Llama-3.1-8B in main experiments; GPT-4o-mini in Section 6.4).
• $\gamma$: mixture ratio for real vs. synthetic labels (0.7 after burn-in).

Initialization (Line 1):

• Experience buffer $\mathcal{D}_0 \leftarrow \emptyset$ — stores all real oracle-labeled preference triplets.
• Policy $\pi_{\theta_0} \leftarrow \pi_{\text{ref}}$ — starts as the SFT model.
• ERM weights $\Phi_0 = \{\phi^0_k\}_{k=1}^K$ initialized randomly (each of the $K = 20$ MLP heads gets its own random initialization). The shared transformer backbone is loaded from the pretrained 0.4B model and frozen.

Per-round loop (Lines 2–11), iterating $t = 1, \dots, T$ where $T$ is the query budget (50,000 in experiments):

Line 3: Sample a prompt $x_t \sim p_\mathcal{X}$.

Line 4: Generate $M = 20$ candidate responses $y_t^i \sim \pi_{\theta_{t-1}}(\cdot | x_t)$ at temperature $\eta = 0.7$. These form $\mathcal{S}_t$.

Lines 5–7 (Duel Selection):

• Line 5 (common): Sample one ensemble member $\phi \sim \text{Uniform}(\Phi_{t-1})$ and set the first response $y_t \leftarrow \arg\max_{y \in \mathcal{S}_t} r_\phi(x_t, y)$. This is the practical approximation of Algorithm 1's Line 4.
• If E&E objective (Line 6): Resample $\phi \sim \text{Uniform}(\Phi_{t-1})$ repeatedly until the argmax produces a response $y'_t \neq y_t$. Both responses are Thompson samples — plausible best responses under different reward function draws.
• If BAI objective (Line 7): Set $y'_t \leftarrow \arg\max_{y \in \mathcal{S}_t} \mathbb{V}_\phi\left[\sigma(r_\phi(x_t, y_t) - r_\phi(x_t, y))\right]$, where the variance is computed across the $K$ ensemble members. This is the practical approximation of Algorithm 1's Line 6.

Line 8 (Oracle vs. ERM labeling):

• Sample a random gate $g \sim \text{Uniform}(0, 1)$.
• If $g < \gamma$: Query the real oracle $P$ to label the duel, obtaining $\mathcal{B}_t = \{x_t, y_t^+, y_t^-\}$. Append $\mathcal{B}_t$ to $\mathcal{D}_t \leftarrow \mathcal{D}_{t-1} \cup \mathcal{B}_t$.
• Else ($g \geq \gamma$): Use a randomly sampled ERM member $r_{\phi_k}$ to provide synthetic labels $\{\tilde{y}_t^+, \tilde{y}_t^-\}$, forming $\mathcal{B}_t = \{x_t, \tilde{y}_t^+, \tilde{y}_t^-\}$. Do NOT add this to $\mathcal{D}_t$.

Line 9 (ERM update):

• Update the ERM by taking $m = 5$ gradient steps: $\Phi_t \leftarrow \Phi_{t-1} - \alpha_R \nabla_\Phi \mathcal{L}_R(\Phi_{t-1} | \mathcal{D}_t)$.
• Each gradient step samples a random batch from $\mathcal{D}_t$ and updates all $K$ ensemble members according to Equation 8.
• The regularization coefficient $\lambda = 0.5$ is applied.

Line 10 (Policy update):

• Update the policy by taking one gradient step: $\theta_t \leftarrow \theta_{t-1} - \alpha_\pi \nabla_\theta \mathcal{L}_\pi(\theta_{t-1} | \mathcal{B}_t, \pi_{\text{ref}}, \mathcal{F})$.
• The batch $\mathcal{B}_t$ used here contains $\gamma$-weighted real data and $(1-\gamma)$-weighted ERM synthetic data, as described in the mixed preference learning section.
• The learning rate is $5 \times 10^{-7}$ with cosine scheduling (Appendix D). The optimizer is Adam with batch size 128. The DAP-specific $\beta$ parameters: $\beta = 0.1$ for DPO, $\beta = 0.2$ for SLiC, and $\beta$ tuned from $\{0.2, 0.3, 0.5, 1.0\}$ for IPO (with best values selected per scale).

Key hyperparameter summary table (from Section 5.2 and Appendix D):

---

The Distributed Learning System (Section 5.1)

The paper treats the distributed training infrastructure as a significant contribution in its own right, arguing that the absence of such a system has held back empirical research on online active alignment. The architecture is an Actor-Learner-Oracle design inspired by distributed deep RL systems (IMPALA by Espeholt et al., 2018; EnvPool by Weng et al., 2022).

The three workload types:

1. Actors (response generation): Use vLLM (Kwon et al., 2023) for memory-efficient autoregressive generation with PagedAttention. Actors receive prompts, generate $M = 20$ candidate responses from the current policy $\pi_\theta$, and send the candidate sets to the dueling selector (which runs inside the learner process). vLLM's optimized KV-cache management is critical for the throughput needed to sustain 50,000 rounds of online training.

2. Oracle (preference labeling): Wrapped as a service using Mosec (Yang et al., 2021b), which supports dynamic batching and parallel processing. The oracle RM (Skywork-Reward-Llama-3.1-8B) is hosted on a separate remote server with 16 A100 GPUs, accessible via a Kubernetes routing layer. This decoupling prevents the oracle's inference cost from competing with the learner's memory and compute. Mosec provides automatic batching — multiple concurrent labeling requests are aggregated into efficient batches for the GPU.

3. Learner (ERM and policy training): Uses DeepSpeed ZeRO (Rasley et al., 2020; Rajbhandari et al., 2020) for memory-efficient distributed training across multiple GPUs. The learner receives labeled duels from the oracle and updates both the ERM (5 gradient steps per round) and the policy (1 gradient step per round). Updated policy weights are broadcast to all actors after every optimizer step via NCCL (or GLOO for compatibility with newer vLLM versions).

Inter-process communication:

• Plasma (Philipp & Robert, 2017): An Apache Arrow-based shared-memory object store for efficient data transfer across process boundaries. Preference data flows from actors → oracle → learner through Plasma without serialization overhead.

• DeepMind Launchpad (Yang et al., 2021a): Composes all workloads into a distributed program, managing process lifecycle and network configuration.

• NCCL vs. GLOO for weight synchronization: NCCL is recommended for broadcasting updated policy weights from the learner master to all actors due to lower latency, but it requires older vLLM versions (before 0.4.3) and is incompatible with collocation configurations where the learner master must establish two separate process groups (one for DeepSpeed, one for weight sync). The paper notes this limitation in Appendix C.

Two deployment configurations (Figure 8, Appendix C):

• Config 1 (full collocation): All three workloads (actor, oracle, learner) run on every GPU. Eight vLLM instances and eight Mosec workers are spawned, one per GPU. After a batch of responses is generated and labeled, it is sent to the learner running across all eight GPUs with ZeRO-2. This maximizes GPU utilization but requires substantial GPU memory — it works only for 1B-scale models.

• Config 2 (half collocation): Actors and oracles run on four GPUs; the learner runs on the remaining four GPUs exclusively. This is necessary for 2.8B and 6.9B scales where the learner requires more memory. The tradeoff is idle time: GPUs allocated exclusively to the learner sit idle while waiting for new preference data, and actor GPUs sit idle while waiting for updated policies. The paper notes that this could be mitigated by asynchronous data collection (allowing minor staleness in the policy used for generation), citing large-scale RL systems like OpenAI Five (Berner et al., 2019), but leaves this optimization to future work.

Benchmarking results (Appendix C, Figure 9). Against HuggingFace's TRL online DPO trainer (which runs all stages sequentially on the same hardware), oat achieves:

• Config 1 (1B scale): 4.21s vs. 4.67s per batch of 128 samples (1.1× speedup).
• Config 2 (2.8B scale): 9.25s vs. 23.56s per batch (2.5× speedup).
• Config 2 (6.9B scale): 34.43s vs. 65.39s per batch (1.9× speedup).

The speedups come primarily from faster generation (vLLM vs. TRL's standard generation) and faster oracle inference (Mosec with dynamic batching vs. TRL's in-process inference, which at 6.9B scale requires ZeRO-3 sharding that severely slows inference). The learner itself is slower in oat Config 2 (using only half the GPUs) compared to TRL (using all GPUs), but this is outweighed by the generation and oracle gains.

Why this system matters beyond engineering. The paper argues that without such a system, it is impossible to properly evaluate online active alignment methods. Prior works that used only "a few iterations of batch learning" were forced into that design not by algorithmic choice but by computational constraints. The paper's ability to run 50,000 rounds of fully online interaction across three model scales and three seeds (9 experimental configurations, each requiring continuous actor-learner-oracle interaction) is what enables the empirical claims about sample efficiency. The open-sourcing of oat is positioned as enabling future research to build on this experimental foundation.

4. Key Insights and Innovations

Innovation 1: Adapting the Exploration Strategy to the Alignment Objective (E&E vs. BAI)

The paper's most intellectually distinctive move is not inventing a new exploration technique, but rather recognizing that LLM alignment contains two fundamentally different learning objectives that demand different exploration strategies, and providing a unified algorithm that gracefully handles both. Prior work either conflated these settings (Dwaracherla et al., 2024, applying an E&E algorithm to BAI evaluation) or ignored the distinction entirely by using the same passive strategy regardless of deployment context.

The conceptual contribution is the diagnosis of an objective mismatch in prior active exploration methods. In contextual dueling bandit theory, the explore-and-exploit (E&E) setting minimizes cumulative regret — every suboptimal duel incurs real cost because users see both responses. The best-arm identification (BAI) setting minimizes labeling cost to find the optimal policy, with no penalty for showing bad duels during training. These two objectives produce qualitatively different optimal exploration behavior: E&E should sample both duel responses to maximize immediate quality (both responses should be plausible best responses under the posterior), while BAI should sample the second response to maximize information gain about the preference relative to the first, even if that means showing a poor response.

"sub-optimal responses with confidently high rewards might be tried for a long time at the expense of not exploring other potentially better choices" (Section 4.1, explaining why E&E-style exploration fails for BAI).

This framing converts what could be dismissed as "just applying Thompson sampling" into a principled design choice: the two different second-response selection rules (Algorithm 1, Line 5 vs. Line 6) are not heuristics but direct consequences of which regret metric the agent optimizes. The practical upshot (Section 6.3, Figure 7) confirms the theory: BAI-TS achieves superior offline performance improvement over E&E-TS, while E&E-TS achieves superior online performance. The Uncertainty strategy — pure information maximization without anchoring to reward quality — performs worst of all for E&E (as expected, since it ignores user experience) and is also surpassed by BAI-TS for BAI, demonstrating that even when labeling cost is all that matters, balancing reward-seeking with information-seeking beats purely maximizing information.

This is a fundamental conceptual reframing, not an incremental improvement. It gives practitioners a clear protocol: if deploying an LLM to serve users, use E&E-TS (both responses look good under plausible reward functions); if hiring annotators, use BAI-TS (first response anchors quality, second response maximizes informativeness). The paper does not claim either strategy is new in isolation — Thompson sampling for E&E and uncertainty-driven selection for active learning are both established — but the articulation of the choice as a function of the deployment objective within the unified CDB framework is what changes how one thinks about designing alignment systems.

---

Innovation 2: The ERM as a Bootstrap Mechanism That Aligns Exploration with Policy Improvement

A subtle but powerful insight in the SEA design is that the epistemic reward model (ERM) serves two roles that reinforce each other: it guides which duels to show the oracle (exploration), AND it provides synthetic labels to train the policy toward regions of high epistemic uncertainty (via mixed preference learning). This dual role creates a virtuous cycle that prior active exploration methods missed.

The critical observation (Section 4.2.3) is that if the policy $\pi_\theta$ is trained only on oracle-labeled data, it learns to favor responses with high true reward $r^\star$. This makes the candidate set $\mathcal{S}_t$ (generated by sampling from $\pi_\theta$) biased toward regions where the ERM has low uncertainty — because both the policy and the ERM have converged on what the true preference signal indicates. But exploration needs the candidate set to also contain responses from high-uncertainty regions — responses where ensemble members disagree — because the variance-maximizing BAI second-response selector or the diverse E&E Thompson samples must have access to these regions to select informative duels.

The solution is mixed preference learning: train the policy on a blend of oracle labels and ERM pseudo-labels (from individual ensemble members). The ERM pseudo-labels encourage the policy to chase individual ensemble members' preferences, which diverge in high-uncertainty regions. This keeps the policy's output distribution spread wide enough to cover the support of epistemic uncertainty, while the oracle labels keep it anchored to true preference. The mechanism is self-reinforcing: as the policy generates candidates in uncertain regions, the ERM gets labeled data from those regions (via the oracle), uncertainty shrinks, the policy improves, and the cycle continues.

Prior active exploration methods for LLMs (Mehta et al., 2023; Das et al., 2024; Dwaracherla et al., 2024) kept the proposal policy fixed, meaning the ERM could only explore within a static candidate distribution. If that distribution had poor coverage of high-reward or high-uncertainty regions, exploration was inherently limited. Methods that did update the policy (Zhang et al., 2024a; Xie et al., 2024; Muldrew et al., 2024) used only oracle labels for policy training, which (as the paper argues) can average out epistemic uncertainty and shrink the exploration frontier.

The ablation in Figure 6 (Section 6.2) isolates this effect. Variant-2 (SEA without ERM sync) uses ERM-guided active exploration but trains the policy only on oracle data. Variant-3 (full SEA) adds mixed preference learning. The performance gap between them demonstrates that aligning the policy with ERM uncertainty is a distinct and measurable contributor to sample efficiency, not merely an implementation detail. This is a diagnostic insight — it identifies why prior active exploration + policy update combinations underperform: they fail to maintain the policy as a good proposal distribution for the ERM's uncertainty.

This innovation is incremental in mechanism (pseudo-labeling is standard in semi-supervised learning and model-based RL) but fundamental in framing: it recasts the relationship between the reward model and the policy from a one-way street (reward model guides policy) to a mutual alignment loop where the policy's coverage of uncertainty enables the reward model to explore effectively, and the reward model's exploration produces better data for the policy. This framing, and the empirical demonstration that the loop matters, is what distinguishes SEA from prior work that treated exploration and policy learning as separate stages.

---

Innovation 3: Verification That Active Exploration Provides Compound Gains Over Passive Online Learning — With a Surprising Sensitivity to Optimizer Choice

The paper's empirical results (Section 6.1, Figure 5) establish a clear performance hierarchy that had been hypothesized but not rigorously demonstrated at scale with fully online training: Offline < Passively Online < Actively Online (SEA). The magnitude of the gaps is striking — SEA achieves 2–5× better sample efficiency than passively online methods (Figure 1, right), and the relative improvements in win rate over reference responses range from +84% to +205% at convergence across model scales (Figure 1, left). These numbers are not marginal improvements on top of an already-good method; they represent a qualitatively different regime of data efficiency.

What makes this finding significant beyond the raw numbers is that it validates the CDB-theoretic prediction that Properties 1 and 2 are jointly necessary. Prior work had shown that online beats offline (Guo et al., 2024, validating Property 1), and that active exploration beats passive for fixed policies (Dwaracherla et al., 2024, validating Property 2). But no prior work had demonstrated, in a fully online setting with continuous policy updates at scale, that the combination yields compound gains rather than subadditive improvement. The fact that SEA substantially outperforms both passive online and fixed-policy active methods confirms that the two properties are complementary: online policy updates expand the reachable candidate space, and active exploration efficiently searches that expanding space.

A surprising and under-emphasized finding is the sensitivity of these gains to the choice of DAP optimizer. Figure 5 (first column, 1B scale) shows that Offline DPO, IPO, and SLiC all reach similar final performance (~70% win rate). But when active exploration is added via SEA, DPO benefits dramatically more than IPO or SLiC — the gap between SEA-DPO and Online-DPO is much larger than the corresponding gaps for SEA-IPO vs. Online-IPO or SEA-SLiC vs. Online-SLiC. The paper notes this briefly (Section 6.1):

"when incorporating active exploration, the SEA agent using DPO shows much larger improvement than the other two. This suggests that selecting the most suitable policy optimizer coupled with active exploration would yield the best agent."

This is a negative result with practical implications: the effectiveness of active exploration is not optimizer-agnostic, despite SEA being architecturally decoupled from the DAP loss. The paper does not diagnose why DPO benefits more — possibilities include DPO's implicit reward parameterization being better aligned with the ERM's reward estimates, or DPO's gradient structure being more responsive to the distribution of exploration data — but the finding itself is important because it warns practitioners against assuming that any direct optimizer will work equally well with active exploration. It also partially mitigates the criticism that methods like XPO and APL are "tightly coupled to DPO": if DPO is in fact the best optimizer for active exploration, that coupling may be less of a limitation than it first appears.

This innovation is primarily empirical validation that shifts the burden of proof. Before this paper, one could reasonably argue that passive online learning is "good enough" and that the complexity of active exploration isn't justified. After these results — 2–5× sample efficiency gains, consistent across three model scales and three seeds — the argument flips: the burden is now on passive methods to justify why they leave this efficiency on the table. The paper's contribution is not just the SEA algorithm but the demonstration that sample-efficient alignment is achievable with principled exploration, establishing a new baseline that future work must contend with.

---

Innovation 4: The Open-Source Distributed System as a Research Enabler (Not Just Engineering)

The paper's decision to treat the distributed learning system oat as a first-class contribution (Section 5.1, with detailed benchmarking in Appendix C) is itself an intellectual stance: the authors argue that the absence of such infrastructure has been a structural barrier to progress in online active alignment research. Many prior works (Muldrew et al., 2024; Dong et al., 2024; Chen et al., 2024; Zhang et al., 2024a; Xie et al., 2024) were forced into "a few iterations of batch learning" — not because their algorithms were designed for batch operation, but because running fully online training with continuous actor-learner-oracle interaction was computationally prohibitive without a purpose-built system.

"The absence of a performant open-source online alignment system has restricted many existing works to only a few iterations of batch learning... which creates a mismatch with their theories that typically require a large number of online interaction rounds." (Section 5.1)

This is a methodological insight dressed as systems engineering. The paper identifies a theory-practice gap in the online alignment literature: theoretical algorithms assume many rounds of online interaction, but experimental evaluations use pseudo-online batch setups that may not faithfully replicate the dynamics of true online learning. The authors argue that this gap cannot be closed by algorithmic innovation alone — it requires infrastructure that makes fully online experimentation tractable at the scale of modern LLMs (billions of parameters, tens of thousands of interaction rounds).

The specific design choices in oat — decoupling actors (vLLM), oracles (Mosec), and learners (DeepSpeed ZeRO) into independently scalable services — draw on distributed deep RL architectures (IMPALA, EnvPool) that the LLM alignment community had not widely adopted. The benchmarking (Figure 9) showing 2.5× latency reduction against HuggingFace's TRL is not just an engineering flex; it is evidence that the infrastructure gap was real and that closing it changes what experiments are feasible.

This contribution is infrastructural rather than algorithmic, but the paper treats it as enabling the algorithmic contributions. Without oat, the 50,000-round fully online experiments across three model scales and three seeds (9 experimental configurations, each requiring continuous interaction) would have been intractable. The open-sourcing is thus positioned as more than reproducibility — it is an invitation for the field to raise its empirical standards for online alignment research. The claim is that future work should not be accepted with batch-iterative approximations of online learning when fully online infrastructure exists. Whether the community adopts this standard remains to be seen, but the paper has shifted the Overton window by both arguing for and providing the tool.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the TL;DR summarization dataset (Stiennon et al., 2020), which consists of Reddit posts as prompts and requires the model to produce summaries aligned with human preferences. The paper fixes 50,000 prompts for training and limits the query budget to 50,000 preference labels as well, creating a setting where the number of available prompts equals the labeling budget.

• Base model(s). Experiments start from supervised fine-tuned Pythia models (Biderman et al., 2023) at three scales—1B, 2.8B, and 6.9B parameters—tuned on TL;DR, taken directly from the checkpoints released by Huang et al. (2024). These serve as both the initial policy $\pi_{\theta_0}$ and the fixed reference policy $\pi_{\text{ref}}$ throughout training. The Pythia family is chosen because it provides a controlled scaling ladder where architecture and training data are held constant across sizes, isolating the effect of model scale on alignment sample efficiency. Ablation experiments (Sections 6.2 and 6.3) use the 1B model "to save computation" (Appendix D).

• Metrics. The primary metric is win rate against reference responses judged by the preference oracle. In the BAI setting (the main experimental focus), the metric is "offline" win rate: periodically, the latest policy $\pi_{\theta_t}$ generates responses (using greedy decoding, temperature 0) for a fixed set of holdout prompts, and the oracle compares each generated response against the corresponding reference response from the dataset, reporting the fraction of wins. When comparing across model scales (Figure 1, left), absolute win rate against ground-truth reference responses is reported. When comparing within a single scale (Figure 5, all subplots), relative win rate against the initial SFT model is used. For the E&E setting (Section 6.3 only), the metric is "online" win rate: the running average of the oracle's preference for the agent's dueling responses during the experience collection process itself.

• Baselines. The paper includes several baselines organized by how they satisfy the two key properties:

  • Offline DAP (Property 1 ✗, Property 2 ✗): Standard direct alignment methods (DPO, IPO, SLiC) trained on a fixed dataset of preference pairs collected once from the reference policy $\pi_{\text{sft}}$. This is the standard DPO recipe from Rafailov et al. (2023), applied to the TL;DR training set.

  • Online (passive) DAP (Property 1 ✓, Property 2 ✗): Faithfully online direct alignment following the OAIF framework of Guo et al. (2024). At each round, both dueling responses are sampled from the current policy $\pi_{\theta_{t-1}}$, labeled by the oracle, and used to immediately update the policy. No active selection of which pairs to compare; exploration is passive.

  • APL (Active Preference Learning, Muldrew et al., 2024) (Property 1 ✓, Property 2 ✓): An active exploration method built on DPO that uses the implicit reward margin (the difference in DPO's implicit reward between two candidate responses) to select which response pairs to label. The method is tightly coupled to DPO—it cannot be used with IPO or SLiC. The paper reproduces APL in a fully online manner (the original work used a few batch iterations) with recommended hyperparameters.

  • XPO (Exploratory Preference Optimization, Xie et al., 2024) (Property 1 ✓, Property 2 ✓): An active exploration method that adds an optimistic bonus term to the DPO loss, derived from an epistemic uncertainty estimate. Like APL, it is coupled to DPO. The paper reproduces XPO online with recommended hyperparameters.

  • Variant-7 from the ablation (Section 6.2, Figure 6): This corresponds to Dwaracherla et al. (2024)—active exploration with an epistemic reward model but a fixed proposal policy. The policy $\pi_{\text{ref}}$ is never updated; only the ERM is learned. At inference time, Best-of-N sampling with the ERM selects responses. This baseline isolates the contribution of policy updates separate from active reward model learning.

  Note: SELM (Zhang et al., 2024a) is omitted because it shares "a very similar algorithmic design" with XPO (Appendix D).

• Generation budget / compute accounting. The paper's primary cost metric is the number of oracle queries—each duel labeled by the preference oracle (either the scalar RM or GPT-4o-mini) consumes one unit of the labeling budget. The training budget is fixed at 50,000 queries for all methods, and the x-axis of all learning curves is "query step" (number of oracle queries made so far). This is the natural metric for sample efficiency: methods are compared by how much policy improvement they extract per labeled comparison. Computational cost (FLOPs, wall-clock time) is reported separately in the system benchmarking (Appendix C, Figure 9) but is not the primary axis of comparison for sample efficiency—the paper's central claim is about reducing human labeling cost, not GPU cost. The ERM's training and the policy's gradient steps are not counted against the query budget; only oracle interactions are. Note that SEA uses $\gamma = 0.7$ after burn-in, meaning roughly 30% of policy training batches use synthetic ERM labels that cost zero oracle queries—this is the data amplification mechanism.

• Cross-validation / statistical protocol. The paper runs 3 independent trials with different random seeds for every experiment, and reports results with "mean and standard error to indicate their statistical significance" (Appendix D). The error bars are visible as shaded regions on the learning curves in Figures 5, 6, and 7. For hyperparameter tuning, $\beta$ for IPO is swept over $\{0.2, 0.3, 0.5, 1.0\}$ and the best value is selected per scale. The mixture ratio $\gamma$ is not swept—it is fixed at 0.7 after a 1,000-sample burn-in at $\gamma = 1$, and the ERM regularization $\lambda = 0.5$ is chosen after "a coarse hyperparameter search" (Appendix D). There is no held-out validation set used for early stopping or model selection during online training—the online nature means all data is used as it arrives, and performance is measured on holdout prompts throughout.

Main Quantitative Results

Overall Comparison: SEA vs. Baselines Across Scales and Optimizers (Figure 5, Figure 1)

The central result is that SEA consistently and substantially outperforms all baselines across three model scales and three direct optimizers, with the gains being largest for DPO and at smaller scales.

Across all nine (scale × optimizer) combinations: The full learning curves are in Figure 5. The consistent pattern is that Online (passive) improves over Offline, and SEA improves over Online, with the SEA curves rising faster (better sample efficiency) and reaching higher final win rates (better asymptotic performance). The paper states that across all settings, "Online agents consistently improve sample efficiency over their Offline counterparts, validating the necessity of Property 1" (Section 6.1), and "SEA outperforms both offline and online passive methods across all scales and all direct optimizers, confirming the critical role that Property 2 plays for sample-efficient alignment."

DPO-specific comparison including APL and XPO (Figure 5, first row; Figure 7, right; Figure 1): Since APL and XPO are only compatible with DPO, the most complete baseline comparison is in the DPO column. Specific findings:

• At 1B scale (Figure 5, top-left): XPO provides a small improvement in final performance over Online (passive) but falls substantially short of SEA. APL shows a "significant sample efficiency boost" at early query steps—rising faster than Online—but its advantage diminishes and its asymptotic performance is below SEA. The SEA-DPO curve rises fastest, reaches highest (~0.9 relative win rate vs. SFT, corresponding to ~90% absolute win rate based on Figure 1 left bar for 1B).

• At 2.8B scale (Figure 5, top-center): XPO "falls short" and performs similarly to Online. APL's advantage over Online "diminishes when scaling up." SEA-DPO maintains a clear lead throughout.

• At 6.9B scale (Figure 5, top-right): APL "performs almost the same as Online"—its active exploration advantage essentially vanishes at this scale. XPO similarly shows minimal benefit. SEA-DPO continues to outperform all baselines, though the gap narrows somewhat compared to smaller scales. The paper does not provide a detailed diagnosis for why APL's benefit disappears at scale, but one hypothesis is that larger models produce more diverse candidates naturally, making APL's margin-based selection less informative.

Optimizer sensitivity (Figure 5, first column—1B scale across DPO, IPO, SLiC): All three Offline agents reach comparable final performance at the 1B scale, but their responsiveness to active exploration differs dramatically:

• DPO + SEA shows the largest improvement over Online-DPO, with the SEA curve substantially above the Online curve across all 50,000 query steps.
• IPO + SEA and SLiC + SEA also improve over their Online counterparts, but the absolute improvement is smaller, and SLiC Online delivers "slightly less improvement than DPO and IPO Online agents" even in the passive case.

The paper's interpretation: "selecting the most suitable policy optimizer coupled with active exploration would yield the best agent." This is a practically significant finding—it means that adopting SEA requires not just adding active exploration but also choosing the right underlying optimizer, with DPO being the clear winner among those tested.

Sample efficiency quantification (Figure 1, right): The paper quantifies sample efficiency by asking: how many queries does each method need to reach a given win rate threshold? The bar chart in Figure 1 (right) shows the number of queries required by passive Online versus active methods (XPO, APL, SEA) to attain various win rate levels. The key number from the paper's abstract and Section 6.1: SEA achieves 2–5× better sample efficiency compared to the passively online method. Specifically, "SEA not only attains significantly improved final performance but also achieves 2–5× better sample efficiency." The exact multiplier depends on the win rate threshold and model scale, but the representative finding is that SEA reaches a given performance level with half to one-fifth the labeling budget.

Convergence improvements (Figure 1, left): At convergence (after the full 50,000 query budget), the win rate improvements of SEA-DPO over SFT are +84% (1B), +110% (2.8B), and +124% (6.9B), while Online-DPO achieves +89%, +113%, and +147% respectively—note that the relative improvement numbers in the figure caption are win rates against reference responses, and Online-DPO actually achieves higher relative improvement at 6.9B than SEA-DPO (+147% vs. +124%). This apparent reversal at 6.9B deserves scrutiny—I will return to it in the Critical Assessment. The corresponding numbers for Offline DPO are +113% (2.8B) and +142% (6.9B), and for SEA-DPO +205% (2.8B) and +176% (6.9B). The 1B bar shows Online-DPO at +89% and SEA-DPO at +84%, with Offline DPO not shown for 1B.

Ablation: Decomposing SEA's Components (Figure 6, Table 1)

The ablation study (Section 6.2) constructs seven agent variants by crossing three axes: inference method (policy sampling vs. Best-of-N), exploration strategy (passive vs. active), and learning components (policy only, policy + ERM without sync, policy + ERM with sync). All variants use DPO on the 1B scale.

Policy-based inference (Figure 6, left):

• Variant-1 (Online DAP, Guo et al., 2024): Passive exploration, policy $\pi_\theta$ used for both response generation and as the final model. This is the baseline Online DPO.
• Variant-2 (SEA without ERM sync): Active exploration using the ERM for duel selection, but policy trained only on oracle-labeled data (no mixed preference learning, $\gamma = 1$ throughout). This isolates the benefit of active duel selection alone.
• Variant-3 (Full SEA): Active exploration with ERM-guided duel selection AND mixed preference learning ($\gamma = 0.7$ after burn-in), with the policy aligned to ERM uncertainty. This is the complete SEA algorithm.

The results in the left plot show a clear ordering: Variant-3 > Variant-2 > Variant-1. The gap between Variant-2 and Variant-1 demonstrates that active duel selection alone provides meaningful gains. The further gap between Variant-3 and Variant-2 demonstrates that mixed preference learning—aligning the policy with the ERM's uncertainty—provides an additional, distinct improvement. The paper states: "It clearly shows the benefits of learning ERM for active exploration (Variant-2) and aligning $\pi_{\theta_t}$ with $R_{\Phi_t}$ (Variant-3)."

Best-of-N inference (Figure 6, right):

Because an ERM is learned within the agent, one can use it at inference time for Best-of-N (BoN) sampling: generate N responses from a policy, score them with the ERM, and pick the highest-scoring one. This enables a direct comparison with Dwaracherla et al. (2024), which learns a similar ERM but does not update the policy. The variants with BoN inference are:

• Variant-4: BoN from $\pi_\theta$, passive exploration, policy + ERM learned. Uses the ERM for BoN at test time but not for active exploration during training.
• Variant-5: BoN from $\pi_\theta$, active exploration (ERM-guided duels), policy + ERM learned (no sync, $\gamma = 1$).
• Variant-6: BoN from $\pi_\theta$, active exploration, policy + ERM with sync (full SEA with mixed preference learning).
• Variant-7 (Dwaracherla et al., 2024): BoN from $\pi_{\text{ref}}$ (fixed SFT policy, never updated), active exploration, ERM learned. This is the closest reproduction of Dwaracherla et al. (2024)'s method—active reward model learning without policy updates.

The results in the right plot show: Variant-6 > Variant-5 > Variant-4, mirroring the policy-based ordering. The critical finding is about Variant-7: it "ceases to improve after ERM converges due to the limited performance of its fixed policy." The flatlining of Variant-7 demonstrates the fundamental limitation identified in Section 3: without updating the proposal policy, even a perfect ERM cannot select responses better than the best response in the static candidate distribution.

Choice of Exploration Strategy: E&E vs. BAI (Figure 7, Left and Middle)

Section 6.3 compares three exploration strategies based on posterior sampling, evaluated on both online performance (cumulative regret, the E&E metric) and offline performance (anytime regret, the BAI metric). All strategies use the same underlying ERM and policy training, differing only in how the second dueling response $y'_t$ is selected:

1. Uncertainty (pure exploration): Select both $y_t$ and $y'_t$ to maximize epistemic uncertainty about the preference outcome. Implemented by choosing the pair whose logit difference has the largest variance across ERM ensemble members. This corresponds to the approach of Das et al. (2024).

2. E&E-TS: Standard Thompson sampling for both responses (Algorithm 1, Line 5). Both $y_t$ and $y'_t$ are argmax responses under independently sampled reward functions from the ERM ensemble, constrained to be different.

3. BAI-TS: Thompson sampling for the first response, variance maximization for the second (Algorithm 1, Line 6). $y_t$ maximizes a sampled reward; $y'_t$ maximizes the variance of the preference probability relative to $y_t$.

Online performance (E&E metric, Figure 7 left): E&E-TS achieves the best online performance across all query steps. Uncertainty performs worst—this is expected because pure exploration ignores response quality, so the dueling pairs shown during data collection are often poor, incurring high immediate regret. BAI-TS is intermediate: its second response is selected for information gain rather than quality, so its online performance suffers relative to E&E-TS. The gap between E&E-TS and BAI-TS quantifies the cost of exploration: BAI-TS sacrifices roughly 5–10 percentage points of online win rate to gain information more quickly.

Offline performance (BAI metric, Figure 7 middle): The ranking reverses. BAI-TS and Uncertainty both exhibit "more efficient offline performance improvement than E&E-TS." Specifically, BAI-TS achieves the fastest rise in offline win rate, followed closely by Uncertainty, with E&E-TS trailing. The paper explains: "exploration for uncertainty minimizing helps to identify more informative responses to train the LLM policy." BAI-TS ultimately reaches the highest offline win rate, demonstrating that "exploration with both reward and information maximization is better than exploration with only information maximization."

An additional insight the paper draws from the E&E-TS results: E&E-TS "always chooses two responses with similarly high quality to exploit." This leads to less informative training data because both responses are plausible best-responses under some reward function—the resulting preference labels have small DAP loss gradients, causing slower policy improvement. This is a more nuanced point than simply "E&E-TS explores less": the dueling pairs are less contrastive, making it harder for the policy to learn from them.

Aligning with a Human Simulator: GPT-4o-mini as Oracle (Figure 7, Right)

To test whether the results from the scalar reward model oracle generalize to a more realistic setting with stochastic, nuanced human-like feedback, Section 6.4 replaces the Skywork scalar RM with GPT-4o-mini (gpt-4o-mini-2024-07-18) used as an LLM judge, following the prompt template from Li et al. (2023). The LLM-as-a-judge paradigm (Zheng et al., 2023) uses a strong LLM to compare two responses and output a preference, which better captures the potential randomness and reasoning of real human annotators than a deterministic scalar reward model.

The results in Figure 7 (right) show several notable differences from the scalar RM experiments:

• Higher variance overall: The learning curves "generally exhibit higher variance, possibly due to the randomness introduced in the feedback process." The error bands are visibly wider than in Figure 5, reflecting that GPT-4o-mini's preferences are less consistent (or less well-modeled by the BT assumption) than the scalar RM's.

• APL's behavior changes: APL "learns fast initially but is eventually outperformed by Online." This contrasts with Figure 5 (1B DPO), where APL maintained a lead over Online throughout. The paper hypothesizes that APL's margin-based selection may be less reliable with noisier feedback, causing it to select suboptimal dueling pairs after the initial phase.

• XPO's behavior changes: XPO "improves over Online after stabilizing its training and delivers a better final performance." This is also a reversal from Figure 5, where XPO showed minimal benefit. The paper does not provide a specific explanation, but it suggests that XPO's optimistic bonus mechanism may be more robust to stochastic feedback than APL's margin-based approach.

• SEA remains the best: Despite the changes in relative ordering of APL and XPO, SEA "is shown to offer the best sample efficiency as well as asymptotic performance." This is the key robustness check—SEA's advantage is not an artifact of the deterministic scalar oracle. The paper states this "further validates the importance of online learning and well-designed active exploration mechanism."

The LLM-as-a-judge experiment is conducted only at the 1B scale with DPO, and only with 50,000 query steps. The paper does not report whether SEA's advantage persists at larger scales with the GPT-4o-mini oracle, which is a limitation.

Ablation Studies and Robustness Checks

ERM ensemble size and architecture: No ablation over $K$ (ensemble size) or MLP hidden layer dimensions is provided. The values $K = 20$ and "2 hidden layers of 128 nodes" are fixed choices without sensitivity analysis. This is a notable gap—the ERM's uncertainty estimates are central to the method, and if they are insensitive to $K$ (or if $K = 5$ works equally well), the computational cost of training 20 heads could be reduced. Conversely, if performance degrades sharply for $K < 20$, that would indicate sensitivity that practitioners need to account for.

Candidate set size $M$: No ablation over $M = 20$ (the number of candidate responses generated per prompt). This is a critical hyperparameter for the policy-guided search approximation—if $M$ is too small, the search cannot find good responses even if $\pi_\theta$ puts some probability on them; if $M$ is too large, generation cost increases. The paper does not report how sensitive performance is to $M$, nor whether the optimal $M$ changes with model scale or training stage.

Mixture ratio $\gamma$: The value $\gamma = 0.7$ (after a 1,000-sample burn-in at $\gamma = 1$) is stated without ablation. The mixed preference learning mechanism is central to SEA's design, but the paper provides no evidence for the chosen ratio versus alternatives (e.g., $\gamma = 0.5$, $\gamma = 0.9$, $\gamma$ annealing from 1 to some final value). It is unclear whether the 0.3 fraction of synthetic data is optimal or simply a reasonable default.

ERM regularization $\lambda$: The value $\lambda = 0.5$ is reported as the result of "a coarse hyperparameter search" (Appendix D) with no details on the search range, the metric used for selection, or sensitivity. Given that $\lambda$ is crucial for maintaining ensemble diversity (and thus the quality of uncertainty estimates), this is a significant omission.

Regex $\beta$ sensitivity: For IPO, $\beta$ is tuned from $\{0.2, 0.3, 0.5, 1.0\}$ across scales with "the best performing results" reported. For DPO ($\beta = 0.1$) and SLiC ($\beta = 0.2$), the paper states these values "are robust for all scales" without further tuning. No sensitivity curves are shown.

ERM gradient steps per round $m$: The paper uses $m = 5$ and states this "suffices to achieve reasonable accuracy" (Algorithm 2). No ablation over $m$ is provided, and the ERM's accuracy is never directly measured or reported—only the downstream effect on policy win rate is visible.

Burn-in period length: The burn-in of 1,000 samples where $\gamma = 1$ (no ERM pseudo-labels) is stated without ablation. The justification is intuitive (the ERM needs initial accuracy before its labels are trusted), but the sensitivity to this choice is unknown.

Effect of removing ERM sync from policy training (Figure 6): This is the most informative ablation in the paper. The gap between Variant-2 (active exploration, $\gamma = 1$) and Variant-3 (active exploration, $\gamma = 0.7$ with mixed preference learning) isolates the contribution of aligning the policy with ERM uncertainty. The gap is visible and consistent across query steps, confirming that mixed preference learning provides a distinct benefit beyond active duel selection alone. However, the magnitude of the gap is modest—perhaps 3–5 percentage points of win rate—suggesting that active duel selection is the dominant component, with mixed preference learning providing a smaller but reliable boost.

Effect of removing active exploration (Variants 1 vs. 2, Variants 4 vs. 5): Both comparisons show clear gaps, quantifying the benefit of ERM-guided duel selection over passive sampling from $\pi_\theta$. The gap is larger for policy-based inference (Figure 6 left) than for BoN inference (Figure 6 right), which makes sense: BoN already selects the best response from a set, partially compensating for passive exploration during training.

Policy-based inference vs. Best-of-N inference (Variants 1/2/3 vs. 4/5/6): The right plot of Figure 6 consistently shows higher win rates than the left plot for corresponding variants, because BoN provides an inference-time boost. However, the relative ordering of variants is preserved, indicating that the training-time benefits of active exploration and mixed preference learning translate to both inference methods.

Variant-7 flatlining (fixed proposal policy): This is the most striking negative result in the ablations. Variant-7 (Dwaracherla et al., 2024) initially improves as the ERM learns, but then plateaus well below all other variants. This validates the paper's central claim that active reward model learning without policy updates is fundamentally limited—the ERM can only select among responses the fixed policy can generate, and that ceiling is quickly reached. This ablation is the cleanest evidence that Properties 1 and 2 are complementary: active exploration (Property 2) alone, without online policy updates (Property 1), cannot achieve sample-efficient alignment.

Exploration strategy comparison under both metrics (Figure 7, left and middle): This is a well-designed robustness check showing that the optimal strategy is metric-dependent. E&E-TS wins on online performance; BAI-TS wins on offline performance. This directly validates the paper's theoretical framing that the two settings require different algorithms. The fact that Uncertainty (pure exploration) is not optimal for either metric—it loses to E&E-TS on online performance (as expected) AND to BAI-TS on offline performance—is a non-obvious finding that supports the paper's design choice to balance reward and information maximization.

LLM-as-a-judge oracle (Figure 7, right): This serves as a robustness check against oracle type. The qualitative patterns hold (SEA outperforms baselines), but the relative ordering of APL and XPO changes, indicating that exploration strategies are sensitive to the noise characteristics of the oracle. This is both a strength (SEA is robust) and a limitation (the paper does not explain why APL and XPO swap positions, making it unclear how to predict which method will work best with a new oracle).

Critical Assessment

Claim 1: "SEA achieves highly sample-efficient alignment, outperforming recent active exploration methods for LLMs" (Abstract and Section 6.1)

What was tested: Win rate learning curves for SEA vs. Offline, Online, APL, and XPO across three model scales and three optimizers (Figure 5), plus win rate vs. query budget comparisons (Figure 1).

Does the evidence support the claim? Partially, with important caveats about the baseline implementations and the scope of "recent active exploration methods."

The evidence clearly shows that SEA outperforms Offline and Online (passive) methods. The gap to Online is substantial and consistent—Properties 1+2 beat Property 1 alone.

The comparison to APL and XPO is more nuanced:

• APL and XPO were reproduced by the authors in fully online mode, whereas their original papers used batch-iterative setups. The authors state they "follow the recommended hyperparameters from their papers" (Appendix D). However, these hyperparameters were tuned for batch-iterative training, not for the fully online setting. It is possible that APL and XPO would perform better with hyperparameters re-tuned for the online regime. The paper does not report any re-tuning effort.

• APL and XPO are only compared with DPO because they are incompatible with IPO and SLiC. This means the claim "outperforms recent active exploration methods" rests on comparisons with only two methods, both using DPO, and at the 1B scale where SEA-DPO shows its largest advantage. At 2.8B and 6.9B, the gap between SEA and APL narrows (Figure 5), and at 6.9B with the GPT-4o-mini oracle (Figure 7 right), XPO actually achieves final performance comparable to or better than SEA. This suggests the claim of universal superiority may not hold at larger scales or with different oracles.

• The comparison to SELM (Zhang et al., 2024a) is omitted entirely because it is "very similar" to XPO. Even if the algorithms are similar, independent empirical validation would strengthen the claim. The reader cannot know whether SELM would show the same patterns as XPO.

What would strengthen this claim: (1) Re-tuning APL and XPO hyperparameters for the fully online setting to ensure the comparison is fair to those methods. (2) Extending the DPO-specific comparisons to 2.8B and 6.9B for the GPT-4o-mini oracle. (3) Including SELM as an additional baseline, or at minimum reporting results from attempting to reproduce it.

Claim 2: "SEA achieves 2–5× better sample efficiency compared to passively online methods" (Abstract, Section 6.1, Figure 1 right)

What was tested: The number of queries required by each method to reach various win rate thresholds (Figure 1 right).

Does the evidence support the claim? The claim is supported at the 1B scale with DPO, where Figure 1 (right) shows passive Online requiring 2–5× more queries than SEA to reach the same performance. However, the claim is stated as a general property of SEA, and the evidence for larger scales is less clear.

At 6.9B (Figure 5, top-right), the Online DPO curve rises quite rapidly and the gap to SEA is narrower than at 1B or 2.8B. The paper does not provide the query-ratio analysis for 2.8B and 6.9B separately—Figure 1 (right) appears to aggregate or focus on the 1B results. If the 2–5× factor only holds at 1B and shrinks to, say, 1.3× at 6.9B, the claim is misleading.

Furthermore, at 6.9B the convergence win rates in Figure 1 (left) show Online-DPO at +147% vs. SEA-DPO at +176%—a 29 percentage point difference. But the relative improvement numbers in the figure caption are confusing: for 2.8B, Offline DPO +113%, Online DPO +142%, SEA DPO +205%; for 6.9B, Offline DPO +142%, Online DPO +147%, SEA DPO +176%. The fact that Online-DPO at 6.9B achieves +147% (nearly matching SEA-DPO at 2.8B's +176%) while SEA-DPO at 6.9B is only +176% suggests that the benefit of active exploration may diminish with scale—larger models may explore sufficiently well passively that active selection provides less marginal value.

What would strengthen this claim: (1) Explicit query-ratio analysis for 2.8B and 6.9B scales. (2) Reporting whether the 2–5× factor is consistent across optimizers (DPO vs. IPO vs. SLiC). (3) A discussion of why the benefit appears to shrink with scale.

Claim 3: "The choice of exploration strategy should match the deployment objective (E&E vs. BAI)" (Sections 4.1 and 6.3)

What was tested: Online and offline win rates for three exploration strategies (Uncertainty, E&E-TS, BAI-TS) at the 1B scale with DPO (Figure 7, left and middle).

Does the evidence support the claim? Well-supported, with one caveat. The results cleanly show that E&E-TS wins on online performance and BAI-TS wins on offline performance, validating the theoretical prediction. The additional finding that BAI-TS outperforms Uncertainty on offline performance is a non-obvious result that strengthens the paper's argument for balancing reward and information maximization.

The caveat: the experiment uses only one model scale (1B) and one optimizer (DPO). The theoretical argument is general, but the empirical validation is narrow. It is plausible that at larger scales, where the model's initial policy is stronger, the relative ordering of strategies might change—for example, if the policy already produces good responses, the gap between E&E-TS and BAI-TS on offline performance might narrow because even E&E-TS duels are fairly informative.

What would strengthen this claim: Replication at 2.8B and/or 6.9B scale.

Claim 4 (implicit): "SEA is a practical algorithm that can work with different direct optimizers" (motivated by the pluggable $\mathcal{F}$ abstraction in Section 4.2.3)

What was tested: SEA with DPO, IPO, and SLiC at the 1B scale (Figure 5, first column), and for DPO only at 2.8B and 6.9B.

Does the evidence support the claim? Weakly. While SEA technically works with all three optimizers (all SEA curves are above their respective Online baselines), the benefit is substantially larger for DPO than for IPO or SLiC. The paper's own text acknowledges this: "the SEA agent using DPO shows much larger improvement than the other two" (Section 6.1). If the optimizer matters this much, then the abstraction is leaky—SEA is not truly optimizer-agnostic in practice, because the sample efficiency gains are optimizer-dependent.

Moreover, the IPO and SLiC results are only shown at the 1B scale. For 2.8B and 6.9B, only DPO is reported. This is a significant omission: if SEA-IPO at 6.9B performs only marginally better than Online-IPO, the claim of optimizer-agnosticity would be further weakened.

What would strengthen this claim: (1) Full scale sweep for IPO and SLiC. (2) Diagnosis of why DPO benefits more—is it DPO's implicit reward parameterization, gradient properties, or sensitivity to the $\beta$ hyperparameter? (3) Testing with additional optimizers beyond the three studied, such as SimPO (Meng et al., 2024) or KTO.

Claim 5 (implicit): "The distributed learning system is a significant enabler" (Section 5.1)

What was tested: Batch latency comparison between oat and HuggingFace's TRL online DPO trainer at three scales (Appendix C, Figure 9).

Does the evidence support the claim? The system benchmarking is thorough and the speedups (up to 2.5× at 2.8B) are meaningful. However, the claim that this "enables" research has a circular quality—the paper demonstrates that oat is faster, but does not demonstrate that TRL would be incapable of running the same experiments. Fifty thousand rounds at 23.56 seconds per batch (TRL at 2.8B) would take approximately 328 hours (13.7 days) per run; at 9.25 seconds (oat Config 2), it takes 128 hours (5.3 days). Both are feasible with sufficient compute; the difference is convenience and cost, not fundamental capability.

The stronger argument is that the decoupled architecture makes it practical to run multiple concurrent experiments (3 seeds × 3 scales × several methods), which would be prohibitively slow with TRL. But the paper does not explicitly make this case, nor does it report total experimental compute (only that "all experiments conducted for this research consume about 2 A100 GPU years" in Appendix D). Two GPU-years is substantial but not prohibitive for a well-resourced lab, even without a custom distributed system.

Missing Experiments That Would Strengthen the Paper

1. Ablation over $M$ (candidate set size). This is the most glaring omission. The entire policy-guided search approximation depends on $M$ being large enough to cover good candidates. If $M = 5$ performs nearly as well as $M = 20$, the method is more efficient than reported. If $M = 20$ is barely enough and $M = 10$ shows significant degradation, the method is fragile. Without this ablation, practitioners cannot make informed decisions about the generation budget.

2. Direct measurement of ERM accuracy and calibration. The paper's entire argument about uncertainty-guided exploration rests on the ERM providing meaningful epistemic uncertainty. But ERM accuracy is never directly evaluated—e.g., by measuring how well the ensemble's mean prediction correlates with the oracle's reward, or how well the ensemble's variance predicts actual prediction error. Showing that the ensemble is well-calibrated (variance ≈ expected squared error) would substantially strengthen the method's foundations. The only indirect evidence is that active exploration improves downstream win rate, which could be for reasons unrelated to uncertainty quality.

3. Comparison to a version of SEA where the ERM uses the same backbone scale as the policy. The ERM uses a 0.4B frozen transformer while the policy uses up to 6.9B. This is a deliberate design choice reflecting "the fact that human preferences can be more complex than what the agent can model" (Appendix D). But it introduces a confound: does SEA's advantage come from active exploration, or from having access to a separate (albeit smaller) model for reward estimation? A comparison where the ERM shares the policy's backbone (e.g., using the 1B Pythia as the ERM backbone when training the 1B policy) would isolate the exploration mechanism from the model capacity effect.

4. Experiments beyond TL;DR summarization. All results are on a single dataset and task. Summarization is a reasonable testbed, but the claims about sample efficiency are stated generally. Testing on dialogue (e.g., Anthropic Helpful-Harmless), instruction following, or code generation would establish whether the benefits transfer across domains. The paper acknowledges this implicitly by not claiming domain-generality, but the abstract and introduction are framed broadly ("aligning LLMs with human preferences").

5. Experiments with real human feedback, not just simulated oracles. The paper uses a scalar RM and GPT-4o-mini as preference oracles. While the GPT-4o-mini experiment (Section 6.4) adds realism, it is still an LLM judge, not a human. Real human preference data introduces noise patterns (annotation artifacts, position bias, length bias, inconsistent standards across annotators) that may interact differently with active exploration strategies. The paper's claims about "sample-efficient alignment with human preferences" would be substantially strengthened by even a small-scale human study.

6. Compute-matched comparison including the cost of candidate generation. The paper's primary metric is oracle queries, but SEA generates $M = 20$ candidate responses per prompt (some of which are used for ERM pseudo-labeling without oracle cost). The passive Online baseline also generates responses, but only the 2 used for the duel. If generation cost were accounted for, SEA would be more expensive per oracle query than Online. The paper does not report a FLOPs-matched or wall-clock-matched comparison, only a query-count-matched one. This is a legitimate choice (human labeling is the bottleneck, not GPU compute), but it means the sample efficiency claims should be understood as labeling efficiency, not computational efficiency.

Genuine Weaknesses

• The convergence win rates at 6.9B in Figure 1 (left) show Online-DPO at +147% and SEA-DPO at +176%. The fact that Online-DPO's relative improvement (+147%) actually exceeds SEA-DPO's at the same scale is confusing and not discussed in the main text. The paper only highlights the positive findings (+84%, +110%, +124%, +205%, etc.). This selective reporting weakens confidence in the universality of SEA's advantage.

• The optimizer sensitivity finding is buried. That SEA works much better with DPO than with IPO or SLiC is a practically important limitation, but the paper treats it as an aside ("This suggests that selecting the most suitable policy optimizer...") rather than as a central caveat. A practitioner reading only the abstract might assume SEA is equally effective with any DAP method, which is not what the data show.

• No confidence intervals on the query-ratio analysis. Figure 1 (right) reports point estimates for the number of queries required to reach various win rates, but with no error bars. Given that the learning curves (Figure 5) show non-trivial variance across 3 seeds, the query-ratio estimates are likely noisy, especially for higher win rate thresholds where fewer methods reach the target.

• The LLM-as-a-judge experiment is a single scale, single optimizer, single judge model. While it is a welcome robustness check, its generalizability is limited. Different judge models (GPT-4, Claude, Llama-3) might produce different preference patterns, and SEA's advantage might vary accordingly.

• The system benchmarking (Appendix C) compares oat to TRL, but TRL is not optimized for multi-GPU online training in the way oat is. TRL's online DPO trainer is a relatively new feature (the paper cites a specific GitHub link), and the comparison may not reflect what a well-engineered baseline would achieve. The 2.5× speedup is meaningful but not transformative enough to be a primary contribution on its own.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Numbers

The assumption or constraint. The entire compute-optimal framework depends on estimating each prompt's difficulty before allocating the inference budget. The paper's method for doing so is generating 2,048 samples per question and calculating either the pass@1 rate (oracle bins) or the average PRM final-answer score (predicted bins). As the authors acknowledge explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

Generating 2,048 samples per question costs more compute than the largest test-time budgets studied (256–512 generations). This means the total computational cost to run the compute-optimal policy on a single question is roughly 2,048 + N generations (for difficulty estimation plus strategy execution), where N is the budget allocated after difficulty is known. The ~4× efficiency gains over best-of-N reported in Figures 4 and 8 are computed after difficulty is known, without amortizing the cost of learning it.

The consequence. In a realistic deployment where difficulty is unknown upfront, the total cost would be dominated by the difficulty estimation step for most reasonable per-question budgets. For example, if the optimal strategy for an easy question allocates 16 generations (matching best-of-64, per the ~4× claim), the total generations would be 2,048 + 16 = 2,064 — actually worse than simply running best-of-512 on every question. The claimed efficiency gains are therefore an upper bound on achievable efficiency that cannot be realized without a cheaper difficulty estimation mechanism. The practical throughput of this method at deployment time is unclear.

What evidence exists in the paper. The gap is acknowledged in Section 3.2 ("we do not account for this cost") and flagged as "a key avenue for future work." However, no experiment measures the sensitivity of the compute-optimal policy to difficulty estimation quality, nor does any ablation show how the policy degrades if difficulty is estimated from a smaller number of samples (e.g., 32 or 64 rather than 2,048). The difficulty estimation itself is not included in any budget calculation in any figure.

Mitigation status. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but does not develop or evaluate such a model. No alternative difficulty estimation method (e.g., using the PRM score on a single greedy sample, or using a lightweight classifier) is tested. The limitation is identified but entirely unaddressed in the current work.

---

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability

The assumption or constraint. The paper's approach assumes the base model's proposal distribution already contains correct solutions at some non-trivial rate. The mechanism amplifies existing capability through search and revision but does not create new capability. The authors state this explicitly in the Section 7 takeaway box:

"test-time compute [cannot] compensate for fundamental capability gaps that larger pretraining would address"

The consequence. For the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero), no amount of test-time compute helps. Across all methods — search (Figure 3, right), revisions (Figure 7, right), and compute-optimal combinations (Figure 9) — bin 5 accuracy hovers at 1–3% regardless of budget up to 512 generations. The FLOPs-matched comparison (Figure 9) shows the scaling line for bin 5 as essentially flat near 0–5%, far below the ~14× larger model's greedy performance. This means the approach offers no path forward for problems that genuinely exceed the base model's training distribution — novel reasoning, out-of-distribution generalization, or tasks requiring knowledge the model did not acquire during pretraining. For such problems, pretraining remains the only viable path, and the paper provides no guidance on how to identify these problems without expensive difficulty estimation.

What evidence exists in the paper. The failure on bin 5 is stark and consistent: Figure 3 (right, bottom-most group) shows beam search and best-of-N both at ~1–3% for all budgets. Figure 7 (right, bin 5) shows all sequential-to-parallel ratios producing ~2–3%. Figure 9 shows the bin 5 scaling line flat and below the larger model's performance at all three R values. The paper does not report what fraction of the MATH test set falls into bin 5 — if this fraction is large, the method's real-world impact is limited.

Mitigation status. The paper does not attempt to mitigate this limitation. It is presented as a fundamental boundary condition: "test-time compute amplifies existing capability but does not create it from nothing." No mechanism is proposed for extending the base model's reach on hard problems (e.g., retrieval augmentation, tool use, multi-step decomposition). The limitation is discussed transparently but not addressed.

---

The ~14× Larger Model Baseline Is Potentially Weak — Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), which would scale both data and parameters equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding — no majority voting, no best-of-N, no test-time compute of its own.

The consequence. A Chinchilla-optimal model trained with ~14× more total FLOPs (scaling both parameters and data appropriately) would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining — e.g., +27.8% on easy questions at R ≪ 1 (Figure 1, top-right bar chart) — may shrink or reverse against a properly compute-optimal larger model. Furthermore, giving the larger model even a modest test-time compute budget (e.g., best-of-8 with the compute-optimal policy) would create a much stronger baseline that is never tested. The comparison is therefore not between "best pretraining" and "best inference-time compute" but between a specific non-optimal pretraining recipe and a tuned inference-time strategy — favoring the latter.

What evidence exists in the paper. The caveat is acknowledged in Section 7 but no ablation or sensitivity analysis quantifies how much the advantage changes if the larger model were Chinchilla-optimal. The paper does not report the ~14× larger model's pass@1 on the MATH difficulty bins, which would help the reader assess how much room for improvement exists from giving it test-time compute. The bars in Figure 1 and the stars in Figure 9 represent the greedy-only larger model, and no alternative baselines are shown.

Mitigation status. The paper frames this as future work and does not attempt to correct for it. A practitioner comparing pretraining vs. inference-time compute investments cannot use these numbers directly unless they also plan to scale parameters without scaling data (which is known to be suboptimal). The limitation is acknowledged but the magnitude of its effect is unknown.

---

Results Are from a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*) — Generalization Is Unverified

The assumption or constraint. All experiments — search, revisions, compute-optimal scaling, FLOPs-matched comparison — use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not tested.

The consequence. Several findings could be model- or domain-specific in ways that matter for deployment:

• The PRM's quality and over-optimization behavior: PaLM 2-S*'s output distribution determines when beam search over-optimizes the verifier signal. A model with different calibration properties (e.g., a Llama-3 or GPT-4 class model) might show different optimal strategy allocations per difficulty level.

• The revision model's training dynamics: The ability to learn from edit-distance-paired incorrect-correct trajectories depends on the base model's in-context learning capabilities and output structure, which vary across model families.

• MATH as a domain: MATH consists of competition-level symbolic reasoning problems with unambiguous ground-truth answers, enabling exact pass@1 computation for difficulty estimation and clean PRM training via Monte Carlo rollouts. Tasks without clean correctness signals — open-ended generation, dialogue, creative writing, multi-step planning — would require fundamentally different verifier training and difficulty estimation approaches that the paper does not develop.

The test set is 500 questions, split into five difficulty quintiles of ~100 each, further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample, and the selected strategies may not be robust to dataset shift.

What evidence exists in the paper. The paper provides no out-of-domain evaluation, no transfer to another model family, and no sensitivity analysis of the policy selection to the number of difficulty bins or the test set size. The confidence in the ~4× efficiency claim depends on the assumption that these 500 MATH questions are representative of a broader problem distribution, which is not established.

Mitigation status. The limitation is not explicitly discussed. The paper's scope is bounded to MATH with PaLM 2-S* in the experimental sections, but the abstract and introduction frame contributions broadly ("LLM alignment," "LLM capabilities"). The gap between the evaluated scope and the claimed scope is unaddressed.

---

The Revision Model Has a ~38% Correct-to-Incorrect Reversion Rate — and the Fix Is a Patch

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). At test time, the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly "revise" them to wrong answers. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. Any sequential revision chain has an inherent tension: later revisions can undo the progress of earlier ones. The paper mitigates this with majority voting or verifier-based selection across the chain (picking the best answer from any point rather than always taking the last revision), but this is an imperfect patch. It means:

• The revision chain cannot be trusted to monotonically improve — the last revision is not necessarily the best, so a selection mechanism external to the revision process is required.
• The selection mechanism itself consumes budget — keeping the best answer from a chain of length L requires either majority voting across chain positions (which may be ambiguous when L is small) or a verifier (which requires a learned reward model, adding complexity and potential over-optimization risk).
• The 38% reversion rate means that, in expectation, roughly 2 out of every 5 correct answers in a chain will be corrupted in the next step, wasting compute and potentially confusing the selection mechanism.

Additionally, the ReST$^{\text{EM}}$ experiment (Appendix K, Figure 16) shows that attempting to further optimize the revision model with RL-style training backfires: "additional sequential revisions substantially hurt performance." The paper hypothesizes this is due to "spurious correlations in revision data" caused by on-policy data collection. This suggests the revision training procedure is brittle and sensitive to data generation methodology.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1, derived from analyzing revision chains. The ReST$^{\text{EM}}$ failure is shown in Appendix K, Figure 16, with fully sequential performance dropping to ~33.5% at 256 generations compared to ~38.5% at the optimal ratio. The within-chain selection mechanism (majority or verifier) is described in Section 6.1 and Appendix I, but its effectiveness is not directly ablated against a "take final revision" baseline.

Mitigation status. The paper patches the problem with within-chain selection but does not solve it at the source. A more principled solution — training the model to recognize when no revision is needed, or including "correct stays correct" trajectories in training — is not explored. The ReST$^{\text{EM}}$ failure suggests the current training recipe is near a performance cliff, and improvements may not be straightforward.

---

Latency and Wall-Clock Time Are Not Discussed — Sequential Strategies Are Inherently Serial

The assumption or constraint. The paper measures test-time compute entirely in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial: each revision r_{i+1} depends on revision r_i, so generating a chain of length L takes L sequential autoregressive generation steps. Parallel best-of-N can execute all N generations simultaneously with sufficient hardware.

The consequence. A compute-optimal policy that allocates, for example, 128 generations as 64 sequential × 2 parallel chains (a ratio favored for medium-difficulty problems in Figure 7) takes roughly 64× longer wall-clock time than a best-of-128 strategy running 128 parallel samples simultaneously. For latency-sensitive applications — interactive assistants, real-time decision-making, user-facing chatbots — this may be prohibitive regardless of the accuracy advantage. The paper's primary metric of "generations" conceals this tradeoff, making it impossible for practitioners to assess whether the efficiency gains translate to real-time deployments or only to batch processing.

What evidence exists in the paper. None. The paper does not report wall-clock times for any strategy, does not discuss the latency-throughput tradeoff of sequential vs. parallel allocation, and does not provide latency-aware variants of the compute-optimal policy (e.g., penalizing sequential steps in the objective). All figures and claims use generation count as the sole cost metric.

Mitigation status. Not addressed. The limitation is absent from the paper's discussion. A latency-aware allocation policy — balancing generation count against wall-clock time, perhaps with a tunable parallelism constraint — is a natural extension but is neither developed nor suggested as future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM alignment from a static, offline data acquisition paradigm toward a continuous, online resource allocation problem where each human preference query is a costly decision to be optimized. Before this work, the field's default assumption—implicit in both RLHF pipelines (Christiano et al., 2017; Ouyang et al., 2022) and direct alignment methods like DPO (Rafailov et al., 2023)—was that preference data is collected once, in bulk, from a fixed behavior policy, and the only question is how to best learn from that static dataset. This paper argues that this assumption leaves enormous efficiency on the table: the agent can and should decide which comparisons to show annotators, and should immediately use the resulting labels to improve both its reward model and its generative policy in a tight loop.

The magnitude of this shift is a reframing, not a paradigm change. Thompson sampling, epistemic uncertainty estimation via ensembles, and active learning are all established techniques. The paper's contribution is not inventing these building blocks but rather synthesizing them into a coherent architecture for online alignment and demonstrating, through rigorous fully-online experiments at scale, that the synthesis yields compound gains beyond what any single mechanism provides. The key reframing is viewing alignment through the lens of contextual dueling bandits (CDB), which forces the designer to confront two questions that offline methods simply ignore: (1) what is the objective during data collection—minimizing cumulative regret (E&E) or identifying the best policy with minimum labeling cost (BAI)?; and (2) how should the exploration strategy change depending on the answer?

This reframing reconciles a tension that had been latent in the active exploration literature. Prior work on active reward model learning (Mehta et al., 2023; Das et al., 2024; Dwaracherla et al., 2024) focused on learning better reward models from fixed proposal distributions, implicitly targeting the BAI objective (find the best policy eventually, without caring about intermediate response quality). Simultaneously, work on online DAP (Guo et al., 2024) focused on improving the policy continuously with passive exploration, implicitly targeting the E&E objective (every response matters because users see them). These two lines of work appeared to be solving the same problem with different tools, but the CDB framing reveals they are solving different problems that map to different deployment scenarios. The paper's empirical demonstration that E&E-TS wins on online metrics while BAI-TS wins on offline metrics (Figure 7) provides the first clean evidence that this distinction is not academic—it matters for algorithm design.

The paper also redirects research attention in two ways. First, it demonstrates that active exploration without online policy updates is fundamentally capped (Variant-7 in Figure 6 flatlines once the ERM converges, because the fixed proposal policy cannot generate better responses than its initial distribution). This finding makes fixed-proposal active reward learning—a line of work represented by Dwaracherla et al. (2024) and Das et al. (2024)—less attractive as a standalone alignment strategy, since the ceiling is low. Future work in active exploration for alignment should include policy updates, or justify why they are excluded. Second, the paper's surprising finding that DPO benefits substantially more from active exploration than IPO or SLiC (Figure 5, first column; Section 6.1) suggests that the choice of optimizer is not independent of the exploration strategy. This opens a research question that did not previously exist: what properties of a DAP loss make it amenable to active exploration, and can we design losses that are specifically optimized for this setting?

Finally, the open-sourcing of oat as a distributed Actor-Learner-Oracle system changes the economics of empirical research in online alignment. Before this work, running fully online experiments over 50,000 interaction rounds with continuous policy updates was computationally prohibitive for most academic labs—hence the prevalence of batch-iterative approximations in prior work. The paper argues, and demonstrates through benchmarking (Appendix C, Figure 9), that the infrastructure gap was real. By releasing oat, the paper lowers the barrier to entry for rigorous online alignment research, which should raise the empirical standard the community expects from future work. Whether the community adopts this standard is an open question, but the paper has made it harder to justify evaluating an online algorithm with only "a few iterations of batch learning" when a performant open-source system exists.

Follow-Up Research This Work Enables

Characterize the failure modes of the policy-guided search approximation when the proposal distribution is misaligned with the ERM's uncertainty. The entire active exploration mechanism depends on the policy $\pi_\theta$ generating candidate sets $\mathcal{S}_t$ that cover both high-reward and high-uncertainty regions. Mixed preference learning (the ERM sync component, $\gamma = 0.7$) is designed to maintain this coverage, but the paper never directly measures whether it succeeds. A follow-up study could instrument the candidate generation process: for each prompt, compute (a) the oracle reward of the best candidate in $\mathcal{S}_t$, (b) the maximum epistemic uncertainty (variance of preference probability) among candidates in $\mathcal{S}_t$, and (c) how these quantities evolve over the course of training for SEA with and without mixed preference learning. If $\mathcal{S}_t$ collapses to low-uncertainty regions when $\gamma = 1$ (no ERM sync), that would directly validate the paper's mechanistic claim. If $\mathcal{S}_t$ already covers high-uncertainty regions even without mixed preference learning, then the benefit of ERM sync must come from elsewhere (e.g., regularization, data amplification), and the paper's explanation would need revision. This experiment requires no new infrastructure—only logging the ERM ensemble's predictions on candidates during training.

Test whether the 2–5× sample efficiency advantage persists with real human annotators, and diagnose how annotation noise interacts with exploration strategy. The paper's oracle is either a deterministic scalar RM or GPT-4o-mini used as an LLM judge. Real human preference annotations introduce noise patterns—position bias, length bias, annotator disagreement, inconsistent standards—that are not captured by these simulators. The BAI-TS strategy selects dueling pairs by maximizing preference variance across ensemble members, which assumes the dominant source of uncertainty is epistemic (lack of data). With noisy humans, there is also irreducible aleatoric uncertainty (inherent randomness in preferences), and it is unclear whether the variance-maximization criterion can distinguish between "I'm uncertain because I haven't seen enough data" and "I'm uncertain because humans genuinely disagree." A strong follow-up would replicate the exploration strategy comparison (Figure 7, left and middle) using a small set of real human annotations on TL;DR (e.g., 500–1,000 queries, with 3–5 annotators per comparison to measure inter-annotator agreement). The key measurement is whether BAI-TS still outperforms E&E-TS and Uncertainty on offline win rate when the oracle is noisy, or whether the information-maximization strategy starts selecting pairs that are noisy rather than informative. If BAI-TS degrades relative to passive exploration under high noise, that would establish a boundary condition on when active exploration is worth the complexity.

Develop a cheap difficulty estimator for the prompt distribution that determines when active exploration is most valuable, analogous to the difficulty-conditioned allocation in test-time compute scaling. The paper shows that SEA's benefit over passive Online is not uniform—it is largest at the 1B scale and shrinks at 6.9B (Figure 5, first row), and within the 1B scale, some queries contribute more to policy improvement than others (implicit in the BAI-TS selection criterion). This suggests that not all prompts benefit equally from active exploration. A follow-up could train a lightweight classifier to predict, from the prompt text alone, whether active exploration will provide a significant advantage over passive sampling for that prompt. The classifier would be trained on features like prompt length, uncertainty of the initial ERM ensemble on the prompt, and the history of which duels on similar prompts produced large policy gradient norms. At deployment, prompts predicted to benefit from exploration would be routed to SEA; prompts where passive sampling is sufficient would use the cheaper Online strategy. This mirrors the compute-optimal test-time scaling framework (the reference example in this analysis) but applied to labeling rather than inference budget. The concrete experiment would compare a two-tiered system (SEA for hard prompts, passive Online for easy prompts) against uniform SEA and uniform passive Online, measuring total labeling cost to reach a target win rate. The paper's existing difficulty binning (based on win rate against reference) provides a starting oracle for which prompts are "hard," but the key contribution would be predicting hardness from the prompt alone without generating responses.

Investigate why DPO benefits substantially more from active exploration than IPO or SLiC, and design an optimizer-aware exploration strategy. The paper's finding that SEA-DPO shows much larger improvement over Online-DPO than SEA-IPO over Online-IPO (Figure 5, first column) is reported but unexplained. This is a critical gap because it undermines the paper's claim that SEA is optimizer-agnostic. A diagnostic follow-up would measure, for each optimizer, (a) the distribution of per-example gradient norms on duels selected actively vs. passively, (b) the variance of the policy's implicit reward (for DPO) across the candidate set $\mathcal{S}_t$, and (c) whether the ERM's uncertainty estimates correlate more strongly with true reward error when the policy is trained with one optimizer versus another. A plausible hypothesis: DPO's implicit reward parameterization (where $r_\theta(x, y) = \beta \log(\pi_\theta(y|x) / \pi_{\text{ref}}(y|x))$) aligns naturally with the ERM's learned reward, so duels selected to maximize ERM uncertainty also tend to have large DPO loss gradients, creating a synergy. IPO's squared loss and SLiC's hinge loss may respond differently to the same duels. If the hypothesis holds, the implication is that active exploration strategies should be co-designed with the policy optimizer—for example, an IPO-specific BAI-TS variant that selects duels to maximize the variance of the log-ratio rather than the variance of the preference probability. If the hypothesis is disproven (DPO's advantage comes from something else, like hyperparameter sensitivity), that would still be valuable because it would guide practitioners toward debugging the interaction rather than designing new losses.

Scale the ERM to share the policy's backbone and measure whether model capacity mismatch explains any of SEA's advantage. The ERM uses a frozen 0.4B transformer while the policy uses up to 6.9B parameters. The paper justifies this as reflecting "the fact that human preferences can be more complex than what the agent can model" (Appendix D), but it introduces a confound: part of SEA's advantage over passive Online might come from having access to a separately trained reward model, not from active exploration per se. A controlled experiment would train SEA variants where the ERM uses the same Pythia backbone as the policy (1B, 2.8B, or 6.9B), with the ensemble of MLP heads on top, and compare against the 0.4B ERM at each scale. If the performance gap between SEA and Online shrinks when the ERM backbone matches the policy backbone, that would indicate that capacity mismatch (the ERM being a weaker model than the oracle) is a necessary condition for SEA's benefit—perhaps because it creates the epistemic uncertainty that active exploration exploits. If the gap remains, capacity mismatch is not the driver. This experiment also has practical implications: if a small ERM works as well as a large one for exploration, practitioners can save substantial compute by using a lightweight ERM (as the paper does). If a large ERM is significantly better, the paper's current design is leaving performance on the table.

Conduct a negative result study: when does active exploration hurt alignment, and can we detect these failure modes online? The paper's results are uniformly positive for SEA, but the ReST$^{\text{EM}}$ failure in Appendix K (where RL-style optimization of the revision model backfired severely) suggests that exploration strategies can be brittle. A systematic study of failure modes would deliberately stress-test SEA by (a) reducing the ERM ensemble size $K$ until uncertainty estimates become unreliable, (b) increasing the ERM pseudo-label fraction $1 - \gamma$ until the policy overfits to ERM errors, (c) using a deliberately miscalibrated ERM (e.g., by under-training it with $m = 1$ gradient step per round), and (d) testing on prompts where the BT model assumption is violated (e.g., by using an oracle that exhibits non-transitive preferences). The goal is to map the conditions under which SEA degrades to or below passive Online, and to develop online diagnostics (e.g., monitoring the ERM's validation loss, the variance of ensemble predictions, or the policy's KL divergence from $\pi_{\text{ref}}$) that could detect these failure modes early enough to fall back to passive exploration. This kind of negative result study is rarely published but is essential for practitioners deciding whether to adopt SEA in safety-critical alignment settings where a period of degraded performance during exploration could be unacceptable.

Practical Applications and Downstream Use Cases

Crowdsourced alignment with minimum labeling cost. The most direct application of SEA's BAI-TS variant is for organizations that hire annotators to provide preference labels for LLM alignment. In this setting, the relevant metric is labeling cost to reach a target policy quality, and the quality of dueling responses shown to annotators is irrelevant (they are paid either way). The paper's results (Figure 1, right; Figure 5) indicate that SEA-DPO at the 1B scale can match the performance of passive Online DPO using roughly 2–5× fewer preference labels. At typical commercial annotation costs ($0.50–$2.00 per comparison depending on task complexity and annotator expertise), reducing a 50,000-label budget to 10,000–25,000 labels saves $12,500–$80,000 per alignment run. The savings compound when alignment is performed repeatedly (e.g., for each model update, for each new domain or language, or for each customer deployment). The practical barrier to adoption is that SEA requires an initial ERM (the ensemble of reward models) to guide exploration, and training that ERM requires some labeled data upfront. The paper's burn-in period of 1,000 samples (where $\gamma = 1$ and all queries are passive) provides a natural warm-start: collect the first ~1,000 labels passively to train the initial ERM, then switch to active exploration for the remaining budget. A production system would need to determine the optimal burn-in length automatically (when does the ERM become accurate enough that its uncertainty estimates are trustworthy?), but the algorithmic template is directly applicable.

Online user-facing LLM systems that learn from preference feedback during serving. Commercial systems like ChatGPT already ask users to compare two responses (Figure 10 in Appendix E). Currently, these comparisons are used passively—both responses are sampled from the current policy, and the preference label is used to update the model. SEA's E&E-TS variant provides a drop-in improvement: instead of sampling both responses from the policy, sample one via Thompson sampling (maximizing a randomly drawn reward function from an ERM ensemble) and the other similarly, ensuring both responses are reasonable (maximizing some plausible reward) while naturally exploring uncertain regions of response space. The key benefit is improved sample efficiency during serving: each user comparison provides more information for policy improvement, meaning the model converges to better behavior with fewer user interactions. The paper's Figure 7 (left) shows that E&E-TS achieves better online performance than passive exploration, meaning users also see higher-quality responses on average during the exploration phase. The practical integration challenge is latency: SEA's candidate generation ($M = 20$ responses per prompt, ERM scoring of each) adds computational overhead to each user request. For systems where the bottleneck is user feedback volume (not GPU compute), this overhead is acceptable. For systems where inference latency is the primary constraint, a lighter-weight variant (e.g., smaller $M$, cached candidate sets across similar prompts) would need development.

Self-improvement pipelines where LLMs generate and label their own training data. A growing paradigm in LLM development is using the model itself (or a stronger model) to generate preference comparisons for alignment—e.g., LLM-as-a-judge for constitutional AI, or using a strong RM to label on-policy data. In these pipelines, the labeling cost is computational (GPU time) rather than financial (human annotator time), but the principle of sample efficiency still matters because labeling large batches of duels is expensive. SEA's mixed preference learning mechanism ($\gamma = 0.7$) is directly applicable: the agent can use a strong oracle RM (e.g., GPT-4 or a top-ranked RewardBench model) as the "human" for $\gamma$ of the duels, and use its own weaker ERM for the remaining $1 - \gamma$, effectively amplifying the data without additional oracle calls. The paper's results with the GPT-4o-mini oracle (Figure 7, right) demonstrate that SEA maintains its advantage when the oracle is an LLM judge rather than a scalar RM, suggesting the approach transfers. The practical application would be: given a budget of 10,000 GPT-4 API calls for preference labeling, use SEA to generate a stronger aligned policy than passive online DPO would produce with the same budget. At GPT-4o-mini pricing (~$0.15 per 1M input tokens), the API cost savings from 2–5× sample efficiency are modest ($50–$200), but the wall-clock time savings (fewer API calls to wait for, given rate limits) could be significant for rapid iteration cycles.

When to Prefer This Method

The paper articulates explicit tradeoffs between two deployment scenarios that map to different exploration strategies, and between active exploration (SEA) versus passive online and offline alternatives. These can be summarized as decision rules grounded in the paper's experimental results:

• Use SEA with BAI-TS when aligning LLMs through paid crowdsourcing, where the primary cost is annotation budget and the quality of dueling responses shown to annotators during training does not matter. The objective is to maximize policy quality per labeled comparison. The evidence: BAI-TS achieves the best offline win rate among exploration strategies (Figure 7, middle), and SEA with DPO achieves 2–5× better labeling efficiency than passive Online DPO (Figure 1, right). The practical requirement is that you can afford the 1,000-sample burn-in (passive labeling to train the initial ERM) before switching to active selection.

• Use SEA with E&E-TS when deploying an LLM system that asks real users for preference feedback during normal operation (e.g., ChatGPT-style comparison prompts). The objective is to improve the policy over time while maintaining reasonable response quality for users during the learning process. The evidence: E&E-TS achieves the best online win rate among exploration strategies (Figure 7, left), meaning users see higher-quality duels during data collection. The practical requirement is that the additional inference cost of candidate generation ($M = 20$ samples per prompt) and ERM scoring is acceptable relative to the latency budget per user request.

• Prefer SEA over passive Online DAP when (a) using DPO as the optimizer (since the benefit is largest for DPO; Figure 5, first column), (b) operating at smaller model scales (1B–2.8B) where the gap between active and passive exploration is widest (Figure 5, first row), and (c) the labeling budget is constrained enough that a 2× reduction in required queries matters economically. The evidence: SEA-DPO outperforms Online-DPO by substantial margins at 1B and 2.8B, with the gap narrowing at 6.9B (Figure 5). If using IPO or SLiC as the optimizer, the benefit of SEA over passive Online is positive but smaller (Figure 5, first column), so the added complexity of maintaining an ERM ensemble and implementing mixed preference learning may not be justified.

• Prefer passive Online DAP over SEA when (a) deploying at large scales (6.9B+) where the marginal benefit of active exploration is reduced (Figure 5, top-right shows APL performing nearly identically to Online; SEA maintains a lead but a narrower one), (b) using an optimizer other than DPO where SEA's advantage is modest and not yet well-characterized at scale (the paper only tests IPO and SLiC at 1B), (c) the engineering complexity of maintaining an ERM ensemble and implementing the distributed Actor-Learner-Oracle architecture is a barrier, and the available labeling budget is sufficient that passive online learning reaches acceptable quality, or (d) latency constraints make $M = 20$ candidate generations per query infeasible.

• Prefer offline DAP (standard DPO/IPO/SLiC on a static dataset) over any online method when (a) a high-quality static preference dataset already exists for the target domain and the cost of collecting new online labels exceeds the benefit, (b) the deployment environment does not support online interaction (e.g., safety-critical applications where showing exploratory responses to users is unacceptable regardless of the exploration strategy), or (c) the model scale is large enough that offline methods already achieve sufficient alignment quality (Figure 5 shows Offline DPO reaches ~70–80% win rate at all scales, so the residual improvement from online methods may not justify the infrastructure investment in some applications).