ArXiv: 2406.09760

🎯 Pitch

A DPO-tuned model’s own internal reward signal—normally just a training byproduct—can be harnessed to iteratively self-improve without any new human feedback, boosting win rates by over 8% on AlpacaEval 2. The key insight is that length-regularized reward shaping and experience replay make this bootstrapping stable, enabling a 8B model to surpass Gemini Pro using only its initial offline dataset.

1. Executive Summary

This paper introduces DICE (self-alignment with DPO ImpliCit rEwards), a bootstrapping method that iteratively improves a DPO-tuned language model by using the implicit reward model induced by DPO itself to label on-policy generations—constructing preference datasets for subsequent DPO rounds without any external feedback. Evaluated on AlpacaEval 2 and Arena-Hard with Zephyr-7B-beta and Llama-3-8B-DPO base models, DICE incorporates two refinements: length-regularized reward shaping to debias the constructed preference dataset against verbosity (penalizing response length during dataset construction rather than in the training objective) and experience replay (mixing generated on-policy data with the original offline preference dataset to prevent catastrophic forgetting). The method achieves more than 8% length-controlled win rate improvement over both base models—increasing Zephyr-based models from 12.69% to 20.71% and Llama3-based models from 18.20% to 27.55%—with the 8B Llama3 variant surpassing Gemini Pro 1.0 on the AlpacaEval 2 leaderboard, establishing that DPO implicit rewards can serve as effective self-supervision signals for continued alignment improvement without external annotators, reward models, or additional human feedback.

2. Context and Motivation

The Core Problem: How to Keep Improving an Already-Aligned Model Without New Human Feedback

The paper tackles a practical and strategically important question: once you have a DPO-tuned language model that has been aligned using a fixed offline preference dataset, can you extract further alignment improvements from it without acquiring any additional human-annotated data, external reward models, or AI feedback? This is the "self-alignment" or "bootstrapping" problem — can an aligned model serve as its own teacher, iteratively refining itself using only signals that are already contained within the model?

This matters for several reasons that the paper articulates (Section 1, Introduction):

  • Annotation bottleneck: Human preference data is expensive and slow to collect. The UltraFeedback dataset used in this paper contains 60,000 preference pairs, which already represents a significant annotation effort. If every round of alignment improvement requires fresh human labels, the scaling of alignment becomes linearly coupled to the availability of human annotators. A method that can extract multiple rounds of improvement from a single fixed dataset breaks this coupling.

  • Deployment efficiency: In production settings, organizations may have already invested in aligning a model (e.g., via DPO on proprietary preference data). The ability to continuously improve that model without commissioning new annotation campaigns represents substantial cost savings and faster iteration cycles.

  • Foundation for self-improving systems: A model that can self-evaluate and self-improve without external supervision is a stepping stone toward more autonomous alignment pipelines. This connects to broader research agendas around self-play fine-tuning (Chen et al., 2024b) and self-rewarding language models (Yuan et al., 2024).

  • Democratization of alignment: If self-alignment works robustly, organizations with limited resources — who may only have access to a single preference dataset and a moderate compute budget — can still produce highly aligned models competitive with those from well-resourced labs, which is exactly what the paper demonstrates by matching Gemini Pro with an 8B model (Table 2).

The Gap Left by Existing DPO: It's a One-Shot Affair

DPO (Rafailov et al., 2024b) was a breakthrough in simplifying RLHF by eliminating the need for a separately trained reward model and the complexities of online RL optimization (PPO). However, the paper identifies a critical limitation: DPO, as typically practiced, is a single-shot procedure. You take a fixed offline preference dataset, you run DPO once, and you get a single aligned model. Subsequent work revealed that continuing DPO training on the same fixed dataset for multiple rounds leads to performance degradation, not improvement (Section 4.2, Table 1). The authors report that "Offline DPO Iter 2" on the Zephyr backbone drops the AlpacaEval 2 LC win rate from 12.69% (base) to 10.17% after one round and all the way down to 1.89% after two rounds — a catastrophic collapse. Even with an updated reference model ("Offline DPO w/ new ref"), the second round drops to 5.15%.

This degradation is not a training instability — it reflects a fundamental mismatch between offline optimization and on-policy learning, which the paper addresses through theoretical analysis (Appendix A). The key insight is:

"If there exists a suboptimal response y−∈Sy^- \in S that lies in the high likelihood region of πθ(t)\pi_{\theta^{(t)}} ... and y−y^- is never sampled from π(t)=πμ\pi^{(t)} = \pi_\mu thus not optimized as yl(t)y^{(t)}_l during all tt rounds, we have πθ(t)(y∗∣x)≤1−p\pi_{\theta^{(t)}}(y^* | x) \leq 1 - p."

In plain language: when you train on a fixed dataset, there may be bad responses that the model has come to believe are good (high likelihood under the current policy), but those bad responses never appear in the offline data, so they never get penalized. The model keeps producing them with high probability, and the training objective has no mechanism to reduce that probability. The model cannot "unlearn" these bad responses because it never sees them in the preference pairs used for training.

This explains why simply running more DPO on the same data fails — it's optimizing a moving target (the model's own distribution) using stale positional data. Each round of DPO changes what responses the model produces, but the training data doesn't reflect this changed distribution. The gap between the training distribution (offline data) and the model's current output distribution (on-policy) grows with each round.

The Iterative DPO Framework and Why It Needs Preference Signals

The iterative DPO framework (Tran et al., 2023) was proposed to address exactly this staleness problem. The idea is simple: after each round of DPO, generate fresh responses from the updated model, obtain preference labels for those responses, and use this new on-policy preference dataset for the next round. This keeps the training data aligned with the model's evolving output distribution.

But this framework immediately encounters a chicken-and-egg problem: where do the preference labels for the newly generated responses come from? The paper identifies three possible sources of preference signals (Figure 1, left panel), and positions DICE relative to them:

Option 1: External scalar reward model (rϕr_\phi). This is the classical RLHF approach — train a separate reward model on the offline preference dataset, then use it to score new generations. This works (Dong et al., 2024; Xiong et al., 2024) but requires training and maintaining a separate model, which adds complexity. The paper doesn't pursue this path, excluding it as "beyond this work's scope."

Option 2: LLM-as-a-Judge, external (πϕ\pi_\phi). Prompt an external LLM (e.g., GPT-4) to judge which of two responses is better. This is conceptually clean but introduces dependency on an external API, incurs monetary costs, and may raise data privacy concerns. Also excluded from DICE's scope.

Option 3: LLM-as-a-Judge, self (πθ(t−1)\pi_{\theta^{(t-1)}}). Prompt the policy model itself to judge its own responses — the approach of self-rewarding LMs (Yuan et al., 2024). This is truly self-contained (no external dependency), and the paper implements it as a baseline. However, the paper identifies a critical weakness: the model's own judgments provide coarse preference signals. The LLM-as-a-Judge prompt used in the baseline (Appendix E, Figure 6) asks the model to assign a discrete score from 0 to 5. The authors hypothesize that these coarse rewards are "not able to provide effective preference signals when responses are of high quality" (Section 4.2), and the empirical results support this: LLM-as-a-Judge achieves only modest improvements on the Llama3 setting (LC win rate from 18.20% to 21.80% after two iterations), compared to DICE's jump to 27.55%.

DICE's positioning — Option 4: DPO implicit rewards. The paper's core insight is that there is a fourth option that has been overlooked: the DPO-tuned model itself contains a mathematical reward function as a byproduct of its training. Specifically, DPO training implicitly defines a reward model (Section 2.2, Eq. (4)):

r(x,y)=βlog⁡πθ(y∣x)πref(y∣x)r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}

This is not a separately trained model — it's a mathematical identity derived from the DPO objective. The paper's key move is recognizing that this implicit reward can serve as the preference signal for iterative DPO, enabling a fully self-contained bootstrapping loop: the model generates responses, its own implicit reward scores them, the highest and lowest are paired as preference data, and DPO is run again.

This positions DICE as a zero-cost, zero-dependency method for self-alignment. It requires nothing beyond what already exists after one round of DPO: the policy model, its reference model, and the original preference dataset. The implicit reward model has been hiding in plain sight since the original DPO paper.

Two Critical Practical Problems That Motivate the Refinements

The paper doesn't just propose using implicit rewards naively — it identifies two failure modes that would undermine the approach and motivates targeted refinements to address them.

Problem 1: Length exploitation (motivating length-regularized reward shaping). Preference-tuned models have a well-documented tendency to generate unnecessarily verbose responses (Park et al., 2024). This happens because human annotators (and GPT-4 evaluators) tend to prefer longer responses when content is otherwise similar, creating a spurious correlation between length and preference. In an iterative self-alignment loop, this problem compounds dangerously: longer responses get higher implicit rewards, get selected as winners, get reinforced by the next round of DPO, making the model produce even longer responses, and so on.

The paper provides concrete evidence of this in Figure 2 (top). When the vanilla DPO implicit reward (no length correction) is used to construct a preference dataset from on-policy generations, the distribution of the length difference between winning and losing responses (∣yw∣−∣yl∣|y_w| - |y_l|) is heavily skewed positive, with an average length difference of 1,031 characters. The winning responses are massively longer than the losing ones. In contrast, a high-quality offline dataset like UltraFeedback (bottom panel of Figure 2) shows an almost symmetrically distributed length difference centered near zero. This stark visual comparison makes the problem concrete: vanilla implicit rewards produce a dataset that systematically favors length over quality.

The paper's solution — length-regularized reward shaping — is motivated by this diagnostic. Rather than combatting length bias in the training objective (as Park et al., 2024 do, which requires expensive hyperparameter tuning of a regularization coefficient), DICE debiases the dataset itself by penalizing length in the reward function. This is framed as a reward shaping operation (Sutton & Barto, 2018) that subtracts a length penalty from the implicit reward:

rLR(x,y;α)=βlog⁡πθ(y∣x)πref(y∣x)−α∣y∣r_{\text{LR}}(x, y; \alpha) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} - \alpha |y|

The parameter α\alpha is optimized via a simple black-box search to make the resulting preference dataset approximately length-unbiased — specifically, to minimize the absolute average length difference between winners and losers (Eq. (6)). When optimized (α∗=0.023\alpha^* = 0.023 in the Zephyr setting), the length difference distribution becomes much more balanced, with the average dropping from 1,031 to -21 (essentially zero). This transforms the dataset from being dominated by length to being dominated by quality.

A critical design choice that the paper emphasizes: this length regularization happens at dataset construction time, not at training time. This avoids the expensive hyperparameter tuning loop required by Park et al. (2024), where different λ\lambda values must be tried, models trained, and evaluated. In DICE, you find α∗\alpha^* by analyzing the dataset (no training required), construct the debiased dataset, and proceed with standard DPO training using a fixed β\beta.

Problem 2: Catastrophic forgetting from over-reliance on imperfect rewards (motivating experience replay). The implicit reward model is not a perfect proxy for human preferences — it's derived from a model that was itself trained on a finite preference dataset. If self-alignment relies exclusively on this imperfect signal, two risks emerge:

  1. Reinforcement of errors: The implicit reward may systematically prefer certain types of responses that humans would not, and these biases get amplified across iterations.

  2. Catastrophic forgetting: The model may drift away from the knowledge and behavior that was encoded in the initial DPO-trained model (which was trained on verified human preference data). As the paper states: "Solely relying on the implicit reward model may result in forgetting the knowledge inbuilt in the initial policy at the first DPO stage" (Section 3.2).

The solution — experience replay — is motivated by continual learning theory (Rolnick et al., 2019). In each iteration, the preference dataset is constructed as a mixture: a fraction γ\gamma of the data comes from the original offline preference dataset (DofflineD_{\text{offline}}), and (1−γ)(1 - \gamma) comes from self-generated data labeled by the implicit reward. This ensures the model doesn't forget what it learned from human-verified preferences while still benefiting from on-policy data.

The paper explicitly draws an analogy to Deep Q-learning from Demonstrations (Hester et al., 2018), where mixing offline demonstration data with online RL experience accelerates learning. The insight is the same: offline data provides a stable, high-quality anchor, while on-policy data provides freshness and closer alignment to the current policy. Figure 3 (Section 4.5) empirically validates this motivation: γ=0\gamma = 0 (pure self-generated data) and γ=1\gamma = 1 (pure offline data) both underperform intermediate values, with γ=0.5\gamma = 0.5 providing the best balance in the Zephyr setting.

Reconciling Contradictory Prior Evidence

The paper is implicitly motivated by an emerging tension in the preference tuning literature that it helps resolve. On one hand, offline DPO works well for a single round (many papers show this). On the other hand, multiple rounds of offline DPO degrade performance (Table 1 shows this catastrophically). The theoretical analysis in Appendix A explains why: offline data becomes increasingly "stale" as the policy drifts. This creates a sampling distribution mismatch that no amount of additional training can fix — the model keeps producing responses that aren't in the training data and therefore never get corrected.

Separately, the iterative DPO literature had shown that on-policy sampling helps (Guo et al., 2024; Tajwar et al., 2024; Tang et al., 2024), but the existing methods for obtaining on-policy preference labels required external reward models. The paper's reconciliation is elegant: the DPO implicit reward resolves the staleness problem without requiring anything external, because the model already contains a reward function, and that reward function is naturally defined with respect to the current policy and its reference — making it inherently on-policy.

Finally, the paper positions itself relative to self-rewarding LMs (Yuan et al., 2024), which prompted the model to judge its own outputs. That work required an explicit supervised fine-tuning step on an evaluation dataset to teach the model to judge, plus a carefully designed prompt template that yields discrete scores (0–5). DICE's implicit reward approach requires neither: the reward signal emerges directly from the DPO training objective with no additional training, and it provides a continuous-valued signal (the log-ratio of policy and reference probabilities) rather than a coarse discrete score. The empirical comparison (Table 1) shows DICE consistently and substantially outperforming LLM-as-a-Judge across both backbone models and both benchmarks.

How DICE Fits into the Broader Self-Improvement Landscape

The paper positions DICE at the intersection of three active research threads:

  1. Self-improving fine-tuning (Huang et al., 2022; Li et al., 2023a; Sun et al., 2023; 2024): DICE extends this paradigm by showing that the implicit reward — not the model's prompted judgment — can serve as the self-improvement signal, and that this signal is more effective (Table 1) while requiring less infrastructure (no judge fine-tuning, no prompt engineering for evaluation).

  2. On-policy sampling in preference tuning (Guo et al., 2024; Tajwar et al., 2024; Tang et al., 2024): DICE is presented as a practical method for obtaining on-policy preference labels at zero additional cost, which the theoretical analysis (Appendix A) suggests is the key driver of improvement.

  3. DPO implicit rewards (Rafailov et al., 2024a; Zhong et al., 2024): Prior work explored implicit rewards for token-level credit assignment or as a standalone reward model for PPO training. DICE uses implicit rewards differently — as a bootstrapping signal for iterative self-alignment, closing the loop without any external components.

The paper's ambition is not to propose a fundamentally new training algorithm but to demonstrate that a resource already available in any DPO-trained model — its implicit reward function — can be repurposed as a reliable self-supervision signal for continued alignment improvement, when combined with careful dataset construction (length debiasing) and continual learning safeguards (experience replay).

3. Technical Approach

3.1 Reader Orientation

DICE is a self-contained bootstrapping procedure that takes a single DPO-tuned language model and a fixed offline preference dataset, and iteratively improves the model's alignment with human preferences without ever requiring external annotators, reward models, or AI judges. The problem it solves is the "staleness" of offline DPO data—once a model has been aligned once, continuing to train on the same preference pairs causes catastrophic degradation because the data no longer reflects what the model actually generates—and the shape of the solution is a closed feedback loop where the model generates its own training data and scores it using a mathematical reward function that emerged as a byproduct of the original DPO training, with two engineering safeguards (length debiasing and experience replay) that prevent the loop from spiraling into pathological behaviors.

3.2 Big-Picture Architecture (Diagram in Words)

The DICE system operates as an iterative cycle with five major components connected in a loop:

  1. The policy model ($\pi_{\theta^{(t-1)}}$) — the current DPO-tuned LLM that generates candidate responses. It serves dual roles: as the proposal distribution that samples new on-policy data, and as the "judge" whose implicit reward function evaluates those samples.

  2. The reference model ($\pi_{\theta^{(t-2)}}$ or $\pi_{\theta^{(-1)}}$ for the first iteration) — the frozen checkpoint from the previous DPO round, used to compute the implicit reward via the log-ratio $\beta \log \frac{\pi_{\theta^{(t-1)}}(y|x)}{\pi_{\text{ref}}(y|x)}$. This model provides the "baseline" against which the current policy's improvements are measured.

  3. The length-regularized implicit reward function ($r_{\text{LR}}$) — a scalar scoring function that takes a prompt-response pair and outputs a real value representing how good the response is relative to the reference model, minus a length penalty controlled by parameter $\alpha$. This function labels which generated responses are "winning" and "losing" for dataset construction.

  4. The dataset construction module — takes $K$ on-policy generations per prompt, scores them with the implicit reward, selects the highest and lowest as winning and losing responses respectively, finds the optimal length penalty $\alpha^*$ to make the dataset length-unbiased, and mixes the resulting self-generated preference pairs with the original offline preference dataset at a ratio controlled by $\gamma$.

  5. The DPO training loop — takes the mixed preference dataset $D_t$ and runs standard DPO optimization (Eq. (3)) to produce the next policy $\pi_{\theta^{(t)}}$, using $\pi_{\theta^{(t-1)}}$ as the reference model for this round, which then becomes the policy model for the next iteration.

The flow is strictly cyclic: a prompt is sampled from the prompt set $\mathcal{X}$ (extracted from the offline dataset $D_{\text{offline}}$) → $K = 16$ responses are generated from the current policy → each response is scored by the length-regularized implicit reward → the best and worst are paired into a preference example → this process repeats to build a full dataset $D(\alpha^*)$ → the dataset is mixed with $D_{\text{offline}}$ at ratio $\gamma$ → standard DPO trains the next policy → the cycle repeats. The loop runs for $t = 1, 2$ iterations (two rounds post-bootstrapping) and is fully enumerated in Algorithm 1.

3.3 Roadmap for the Deep Dive

  • First, the DPO implicit reward function itself—where it comes from, what it computes, and why it is legitimate as a preference signal. This is the foundation; everything else depends on understanding this mathematical identity.
  • Second, the length-regularized reward shaping mechanism—how the vanilla implicit reward is augmented with a length penalty, how the optimal penalty strength $\alpha^*$ is discovered via a black-box optimization that minimizes average absolute length difference, and why this transforms the dataset distribution.
  • Third, the dataset construction pipeline in full detail—how $K$ on-policy responses are generated, scored, ranked, and paired into preference examples, and how $\alpha^*$ is used to produce the debiased dataset $D(\alpha^*)$.
  • Fourth, the experience replay mechanism—how the self-generated dataset is mixed with the offline preference data at ratio $\gamma$, why this ratio matters, and the continual-learning motivation behind it.
  • Fifth, the DPO training step within each iteration—how the new policy $\pi_{\theta^{(t)}}$ is optimized on the mixed dataset, what reference model is used, and the hyperparameter landscape ($\beta$, learning rate, batch size, training steps).
  • Sixth, the initialization conditions and termination—what model and data the loop starts from, what constitutes the "base model," and how many iterations are productive before degradation sets in.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and systems paper whose core idea is that DPO-trained models contain a usable reward function—the implicit reward defined in Eq. (4)—and that this reward function is sufficiently reliable to serve as a self-supervision signal for constructing on-policy preference datasets, enabling iterative self-alignment without external feedback, provided that two engineering refinements (length-regularized reward shaping and experience replay) are applied to mitigate length exploitation and catastrophic forgetting respectively.

The DPO Implicit Reward: Where It Comes From and What It Computes

The DPO implicit reward is not a separately trained model—it is a mathematical identity that emerges from the relationship between the DPO objective and the standard RLHF objective. To understand it, we must trace the derivation from Rafailov et al. (2024b).

The standard RLHF objective (Eq. (2)) balances reward maximization against a KL-divergence penalty that prevents the policy from drifting too far from a reference distribution:

max⁡πθEx∼D,y∼πθ(y∣x)[rϕ(x,y)]−β⋅DKL[πθ(y∣x)∥πref(y∣x)]\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(y|x)}[r_\phi(x, y)] - \beta \cdot D_{\text{KL}}[\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x)]

where $r_\phi(x, y)$ is a separately trained reward model, $\pi_{\text{ref}}$ is the reference policy (typically the SFT model), and $\beta$ controls the strength of the KL regularizer.

Rafailov et al. (2024b) proved that this RLHF objective has an analytical closed-form solution. The optimal policy $\pi^*$ that maximizes Eq. (2) satisfies:

π∗(y∣x)=1Z(x)πref(y∣x)exp⁡(1βr(x,y))\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} r(x, y)\right)

where $Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp(r(x, y) / \beta)$ is the partition function (normalizing constant) that ensures $\pi^*$ sums to 1.

Taking the logarithm of both sides and rearranging, we can express the reward in terms of the optimal policy:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x)r(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)

This equation says: if you know the optimal policy $\pi^*$ and the reference policy $\pi_{\text{ref}}$, you can recover the underlying reward function up to an additive prompt-dependent constant $\beta \log Z(x)$.

The key insight of DPO is that when training with pairwise preferences under the Bradley-Terry model, the partition function $Z(x)$ cancels out because the preference probability $\sigma(r(x, y_w) - r(x, y_l))$ depends only on the difference of rewards for the same prompt $x$. The partition function is prompt-specific and drops out of the subtraction. This yields the DPO objective (Eq. (3)):

LDPO(πθ;πref)=−E(x,yw,yl)∼D[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right]

Observe that the term inside the sigmoid is exactly the difference of two expressions of the form $\beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$. This motivates the definition of the implicit reward (Eq. (4)):

r(x,y)=βlog⁡πθ(y∣x)πref(y∣x)r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}

where $\beta$ is the same KL-regularization hyperparameter from Eq. (2), $\pi_\theta$ is the DPO-trained policy, and $\pi_{\text{ref}}$ is the reference policy used during DPO training (typically the SFT model).

What it computes: for a given prompt $x$ and response $y$, the implicit reward outputs a real-valued scalar. It takes the log-probability assigned to $y$ by the current policy $\pi_\theta$, subtracts the log-probability assigned to $y$ by the reference policy $\pi_{\text{ref}}$, and scales the difference by $\beta$. Operationally, this is a single forward pass through both models: compute $\log \pi_\theta(y|x)$ (autoregressively: sum the log-probability of each token given previous tokens), compute $\log \pi_{\text{ref}}(y|x)$ identically, subtract, multiply by $\beta$. The result is a number that is positive when the current policy assigns higher probability to this response than the reference policy does (the response is "preferred" relative to the baseline), negative when it assigns lower probability (the response is "disfavored"), and near zero when the two policies agree.

Why this form works as a preference signal: the DPO training objective (Eq. (3)) optimizes the policy so that for human-preferred responses $y_w$, the implicit reward $\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}$ becomes larger (the model learns to increase probability mass on $y_w$ relative to the reference), and for dispreferred responses $y_l$, it becomes smaller (the model learns to decrease probability mass on $y_l$). This means that after DPO training, the implicit reward is correlated with human preference: higher implicit reward generally corresponds to higher-quality responses, lower implicit reward to lower-quality responses. The implicit reward is the signal the DPO optimizer used to distinguish good from bad—it is not an arbitrary function but the exact quantity that DPO training aligned with preference labels.

Why this is legitimate as a reward function: Theorem 1 in Rafailov et al. (2024b) proves that any reward function $r(x, y)$ can be represented in this form without loss of generality. There is no approximation—given any ground-truth reward model, there exists a policy $\pi^*$ such that Eq. (4) recovers that reward (up to the additive constant $Z(x)$). This means the implicit reward is theoretically well-founded, not an ad-hoc heuristic.

What it is not: the implicit reward is not a separately trained critic or value function. It does not require any additional training beyond the original DPO run. It is deterministic given the two models ($\pi_\theta$ and $\pi_{\text{ref}}$) and the hyperparameter $\beta$. It is per-response (outcome-level), not per-token, because Eq. (4) uses the full sequence probability under both models. This distinguishes DICE's usage from token-level implicit reward work (Rafailov et al., 2024a; Zhong et al., 2024) that decomposes the reward across individual tokens.

The role of $\beta$: the DPO hyperparameter $\beta$ controls the scale of the implicit reward. A small $\beta$ (e.g., 0.01) produces a compressed reward range—the log-ratio differences are squashed, making it harder to discriminate between responses. A large $\beta$ (e.g., 0.1) amplifies differences. In DICE, $\beta$ is inherited from the base model's DPO training and then swept (0.01 vs. 0.1) for each iteration to optimize downstream performance. Since the implicit reward's scale affects how strongly it differentiates winning from losing responses, the choice of $\beta$ matters for dataset construction quality. The paper hypertunes it per method and per base model, selecting based on AlpacaEval 2 performance (Appendix F).

Practical computation at inference time: to compute $r(x, y)$ for a generated response $y$, the paper uses the current policy $\pi_{\theta^{(t-1)}}$ as the "target policy" and the previous iteration's policy $\pi_{\theta^{(t-2)}}$ as the reference. For the very first bootstrapping iteration ($t = 1$), the reference is $\pi_{\theta^{(-1)}}$, which is the original SFT model used in the initial DPO training (Section 3, Algorithm 1 line 4: "For $t = 1$, we denote the reference policy in the implicit reward model as $\pi_{\theta^{(-1)}}$"). This is a crucial detail: the implicit reward always compares the current policy against its immediate predecessor, maintaining an on-policy character—it measures how the policy's assessment of a response has changed since the last round of adaptation.

Length-Regularized Reward Shaping: Why and How

The vanilla implicit reward defined above has a known pathology: it correlates with response length. Human annotators (and the AI evaluators used to construct training data) tend to prefer longer responses when content quality is similar, creating a spurious length-preference signal in the training data. The DPO-trained model absorbs this correlation, so its implicit reward systematically rates longer responses higher even when the extra length adds no substantive value.

In an iterative self-alignment loop, this creates a runaway length exploitation cycle. Concretely, in each iteration:

  1. The model generates $K$ responses per prompt.
  2. The implicit reward preferentially selects longer responses as "winning" and shorter responses as "losing."
  3. The constructed preference dataset is biased toward length, not quality.
  4. The next round of DPO further reinforces this bias, making the model more verbose.
  5. At the next iteration, the model generates even longer responses, and the cycle accelerates.

The paper makes this explicit and quantitative through Figure 2 (top panel). When the vanilla implicit reward ($\alpha = 0$) is used to construct a preference dataset from on-policy generations (Zephyr setting, first iteration), the distribution of length differences between winning and losing responses ($|y_w| - |y_l|$) has a mean of 1,031 characters and is visibly right-skewed. In plain language: the average winning response is over a thousand characters longer than the average losing response. This is not a subtle effect—it means the implicit reward is primarily selecting on verbosity, with content quality as a secondary factor.

To counteract this, the paper applies reward shaping, a technique from reinforcement learning (Sutton & Barto, 2018) where a potential-based shaping term is added to a reward function to guide learning without changing the optimal policy. The shaped reward (Eq. (5)) is:

rLR(x,y;α)=βlog⁡πθ(y∣x)πref(y∣x)−α∣y∣r_{\text{LR}}(x, y; \alpha) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} - \alpha |y|

where $\alpha \geq 0$ is a scalar hyperparameter controlling the strength of the length penalty, and $|y|$ is the string length of the response $y$ (number of characters).

What it computes: the length-regularized reward takes the vanilla implicit reward (the log-ratio term) and subtracts a linear penalty proportional to response length. For a given $\alpha$, longer responses receive a larger penalty, reducing their shaped reward. Shorter responses receive a smaller penalty, potentially making them competitive with (or preferred over) long-but-mediocre responses. The $\alpha$ parameter controls the tradeoff: $\alpha = 0$ recovers the vanilla implicit reward (no length penalty), while large $\alpha$ aggressively favors brevity regardless of content quality.

Why subtract, not divide or normalize: the additive penalty form $- \alpha |y|$ is a standard reward-shaping operation—it preserves the linear structure of the reward function and is easily optimized. A multiplicative correction (e.g., dividing by length) would fundamentally change the reward scale and could distort comparisons between responses of different lengths in unpredictable ways. The additive penalty has a clear interpretation: each additional character of length costs $\alpha$ units of reward. Since the reward ultimately determines which response is selected as "winning" and which as "losing," this linear penalty serves as a threshold: a response must be sufficiently better in quality (as measured by the implicit reward) to overcome the length penalty applied to it.

The optimization of $\alpha$ (Eq. (6)): rather than treating $\alpha$ as a hyperparameter to tune via expensive trial-and-error model training (as Park et al., 2024 do with their length-regularized DPO training objective), the paper proposes to optimize $\alpha$ directly on the dataset construction process without any model training. The objective is:

α∗=arg⁡min⁡α∣E(yw,yl)∼D(α)(∣yw∣−∣yl∣)∣\alpha^* = \arg\min_\alpha \left| \mathbb{E}_{(y_w, y_l) \sim \mathcal{D}(\alpha)} (|y_w| - |y_l|) \right|

where $\mathcal{D}(\alpha)$ is the preference dataset constructed by scoring the $K$ on-policy responses with $r_{\text{LR}}(\cdot, \cdot; \alpha)$ and selecting the highest-scoring as $y_w$ and lowest-scoring as $y_l$.

What this objective computes: for a given $\alpha$, construct the entire preference dataset, compute the length difference for each pair ($|y_w| - |y_l|$), average across all pairs, and take the absolute value. The $\alpha^*$ that minimizes this average absolute length difference produces a dataset where winning and losing responses are, on average, equally long. This is an unbiased dataset with respect to length: the preference signal is determined by content quality (as measured by the implicit reward's quality-sensitive component), not by verbosity.

Why absolute value, not squared error: minimizing the absolute value drives the mean length difference toward zero. Squared error would also do this but would additionally penalize high variance, which is not the goal—the goal is to eliminate systematic bias in one direction, not to force all preference pairs to have identical length differences. The absolute value directly targets the bias.

The optimization procedure: the paper uses Bayesian optimization via Gaussian process regression (specifically, the scikit-optimize gp_minimize function; Appendix C). This is a black-box optimizer suitable for non-differentiable, expensive-to-evaluate objectives. The objective landscape (Figure 4, left panel) shows that the absolute length difference is a smooth function of $\alpha$: it starts high at $\alpha = 0$ (heavy positive bias toward long winners), decreases rapidly as $\alpha$ increases, crosses through zero near $\alpha^* = 0.023$ for the Zephyr setting, and becomes large again (now negative bias—short winners) for larger $\alpha$. The optimizer finds the zero-crossing efficiently.

Why optimize on the dataset, not on model performance: this is the paper's key design choice distinguishing DICE from length-regularized DPO (Park et al., 2024). Park et al. add a length penalty term to the DPO training objective, which means the regularization coefficient $\lambda$ must be tuned by training models at each $\lambda$ value and evaluating them—an expensive loop involving model training, inference, and evaluation. DICE's reward-shaping approach decouples these: find $\alpha^*$ by analyzing the dataset (no training required, only forward passes to compute rewards), construct the debiased dataset, then run standard (unmodified) DPO training on it. This separates the length-debiasing problem from the training optimization problem, eliminating the costly hyperparameter tuning loop.

Empirical validation of the approach (Figure 2, top panel): with the optimized $\alpha^* = 0.023$ in the Zephyr setting, the length-difference distribution shifts from a mean of 1,031 (heavy positive skew) to a mean of -21 (effectively zero). The distribution (shown in orange) is much more symmetric and balanced, visually resembling the UltraFeedback offline dataset distribution (Figure 2, bottom panel), which is almost perfectly centered at zero. This is a clean diagnostic that the reward shaping is working as intended: the constructed dataset is now length-unbiased.

Interaction with $\beta$: the effective tradeoff between quality and length in $r_{\text{LR}}$ depends on both $\beta$ and $\alpha$. A larger $\beta$ amplifies the quality signal (the log-ratio term), making the length penalty relatively weaker; a smaller $\beta$ compresses the quality signal, making the length penalty relatively stronger. The paper sweeps $\beta \in \{0.01, 0.1\}$ per iteration and re-optimizes $\alpha^*$ for each $(t, \beta)$ combination. This ensures that the length debiasing adapts to the changing scale of the implicit reward as the policy evolves.

Comparison with LLM-as-a-Judge length bias (Appendix C, Figure 4, right panel): when the same length-debiasing optimization is applied to the LLM-as-a-Judge baseline (where the reward is a discrete score 0–5 from a prompted model), the objective landscape is different: the minimal average absolute length difference occurs at $\alpha = 0$. This means the LLM-as-a-Judge baseline does not exhibit systematic length bias in its judgments—at least, not in a way that the linear penalty model can detect. The paper therefore does not apply length regularization to the LLM-as-a-Judge baseline (it uses $\alpha = 0$). This is an interesting empirical finding: prompted model judgments appear to be more length-calibrated than DPO implicit rewards, possibly because the judge prompt template explicitly instructs the model to evaluate based on content quality criteria rather than length.

Dataset Construction Pipeline

With the length-regularized implicit reward defined and calibrated, the dataset construction process in each iteration $t$ proceeds as follows (Algorithm 1, lines 3–4):

Step 1: Response generation. Given a set of prompts $\mathcal{X}$ extracted from the offline preference dataset $D_{\text{offline}}$, the current policy $\pi_{\theta^{(t-1)}}$ generates $K = 16$ responses for each prompt $x$. Generation uses temperature sampling for diversity: the Zephyr setting uses temperature $T = 0.7$ and nucleus sampling with $p = 0.9$; the Llama3 setting uses temperature $T = 0.9$ and $p = 1.0$. The paper uses different random seeds to ensure the $K$ responses are genuinely diverse rather than near-duplicates. This produces $K$ candidate responses $\{y_1, y_2, \ldots, y_K\}$ per prompt.

Step 2: Reward computation. For each generated response $y_k$, the length-regularized implicit reward is computed:

rLR(x,yk;α)=βlog⁡πθ(t−1)(yk∣x)πref(yk∣x)−α∣yk∣r_{\text{LR}}(x, y_k; \alpha) = \beta \log \frac{\pi_{\theta^{(t-1)}}(y_k|x)}{\pi_{\text{ref}}(y_k|x)} - \alpha |y_k|

The target policy is the current model $\pi_{\theta^{(t-1)}}$, and the reference policy is the previous iteration's policy $\pi_{\theta^{(t-2)}}$ (or $\pi_{\theta^{(-1)}}$ for $t = 1$). The computation requires two forward passes per response (one through each model) to obtain the log-probabilities, plus a length calculation. For $K = 16$ responses across $|\mathcal{X}|$ prompts, this is $32 \times |\mathcal{X}|$ forward passes to compute all rewards.

Step 3: Preference pair construction for a trial $\alpha$. Given a candidate penalty strength $\alpha$, the highest-scoring response among the $K$ candidates (by $r_{\text{LR}}$) is labeled $y_w$ (winner), and the lowest-scoring is labeled $y_l$ (loser). This produces one preference pair $(x, y_w, y_l)$ per prompt. The full dataset $\mathcal{D}(\alpha)$ consists of $|\mathcal{X}|$ such pairs.

Step 4: Optimization of $\alpha$ via black-box search. The objective from Eq. (6) is evaluated for candidate $\alpha$ values: construct $\mathcal{D}(\alpha)$, compute the average absolute length difference $\mathbb{E}[|y_w| - |y_l|]$, and return this scalar to the optimizer. The Bayesian optimizer (Gaussian process-based, gp_minimize from scikit-optimize) proposes new $\alpha$ values, iterating until convergence. The search is cheap because it does not require any model training—only reward computation and dataset assembly, which are pure inference operations. For the Zephyr setting with $\gamma = 0$, the optimal found value is $\alpha^* = 0.023$.

Step 5: Final dataset construction at $\alpha^*$. Once $\alpha^*$ is found, the final self-generated preference dataset $\mathcal{D}(\alpha^*)$ is constructed using the same procedure: for each prompt, the response with the highest $r_{\text{LR}}(\cdot, \cdot; \alpha^*)$ becomes $y_w$, the lowest becomes $y_l$. This dataset has $|\mathcal{X}|$ preference pairs and is approximately length-unbiased.

Dataset size: the paper uses a prompt set of size that yields approximately 9,600 preference pairs ($9.6\text{k}$; Section 4.1: "In each round, we train the model for 300 steps on a preference dataset with 9.6k preference pairs"). This is the effective training dataset size per iteration.

Why highest and lowest, not random pairs or full ranking: the paper selects the extreme responses (highest and lowest reward) rather than random pairs or a full ranking of all $K$ candidates. This is a deliberate choice to maximize the signal-to-noise ratio in the preference data. The implicit reward is an imperfect proxy for true human preference—it has some noise. By selecting the extremes, the paper ensures that the winning response is clearly better (by the implicit reward's own metric) than the losing response, reducing the chance that the preference label is wrong due to reward noise. If the implicit reward's scores for two responses are nearly equal, labeling one as better is unreliable; by taking the max and min, the paper constructs pairs where the implicit reward has high confidence in the ordering.

Interaction between $K$ and dataset quality: with $K = 16$, the preference pair is constructed from the best and worst among 16 candidates. This provides a reasonably strong signal because the extremes of 16 samples tend to be genuinely different in quality. A smaller $K$ would produce weaker signals (less separation between best and worst). A much larger $K$ would increase the chance of finding outlier responses that game the implicit reward (over-optimization). The paper does not ablate $K$; 16 is fixed based on practical computational considerations (generating and scoring responses for thousands of prompts).

Experience Replay: Mixing Self-Generated and Offline Data

Even with length debiasing, relying exclusively on self-generated data labeled by the implicit reward carries a fundamental risk: the implicit reward is an imperfect approximation of human preferences, and iteratively optimizing against it can cause the model to drift away from behaviors that humans actually value (catastrophic forgetting) or to exploit blind spots in the reward (reward hacking).

The paper's solution is experience replay, a technique borrowed from continual learning (Rolnick et al., 2019) and deep reinforcement learning (Hester et al., 2018). In each iteration $t$, instead of training solely on the self-generated dataset $\mathcal{D}(\alpha^*)$, the training dataset $\mathcal{D}_t$ is constructed as a mixture (Algorithm 1, line 5):

Dt={(xi,ywi,yli)}i∈[N],sampled from pDt=(1−γ) pD(α∗)+γ pDoffline\mathcal{D}_t = \{(x_i, y_w^i, y_l^i)\}_{i \in [N]}, \quad \text{sampled from } p_{\mathcal{D}_t} = (1 - \gamma) \, p_{\mathcal{D}(\alpha^*)} + \gamma \, p_{\mathcal{D}_{\text{offline}}}

where $\gamma \in (0, 1)$ is the experience replay ratio—the fraction of training data drawn from the original offline preference dataset $\mathcal{D}_{\text{offline}}$ rather than from the self-generated dataset.

What this computes: for each preference pair in the training batch, with probability $\gamma$ it is sampled from the offline dataset (the same human- or AI-labeled preference pairs used for the initial DPO training), and with probability $(1 - \gamma)$ it is sampled from the self-generated dataset $\mathcal{D}(\alpha^*)$. The total dataset size is kept constant (approximately 9,600 pairs), so $\gamma$ controls the composition, not the total volume of data.

Why mixing helps: the offline dataset provides a stable anchor of verified human preferences. These preference pairs were labeled by humans (or, in the case of UltraFeedback, by a strong AI judge) and represent ground-truth quality signals. By keeping a fraction of this data in every training round, the model is continuously reminded of the behaviors that humans actually prefer, preventing catastrophic drift. The self-generated data provides on-policy freshness—it reflects the current policy's output distribution and allows the model to correct errors that weren't present in the offline data. The mixture balances these two desiderata.

The tradeoff at different $\gamma$ values (Figure 3):

  • $\gamma = 0$ (pure self-generated data): the model trains exclusively on its own outputs labeled by the implicit reward. This provides maximal on-policy information but risks reinforcing implicit reward biases and forgetting offline preferences. In the Zephyr setting, this achieves 17.87% LC win rate after two iterations—better than the base model (12.69%) but suboptimal compared to the $\gamma = 0.5$ setting (20.71%).

  • $\gamma = 1$ (pure offline data, equivalent to "Offline DPO w/ new ref"): the model trains on the same offline data used for initial DPO, but with an updated reference model. This provides stable, high-quality labels but does not introduce any new information about the model's current output distribution. Performance degrades to 13.40% after one iteration and 4.58% after two iterations—dramatic forgetting, as predicted by the theoretical analysis in Appendix A.

  • $\gamma = 0.5$ (equal mix, optimal in Zephyr setting): the model sees both the stable anchor of offline preferences and the fresh on-policy signal from self-generation. This achieves the best performance (20.71% LC win rate after two iterations), supporting the paper's hypothesis that the combination makes for "a good balance" (Section 3.2).

Why $\gamma$ varies by base model: the optimal ratio differs between the Zephyr backbone ($\gamma = 0.5$) and the Llama3 backbone ($\gamma = 0.1$; Appendix D, Figure 5). The paper attributes this to differing quality of self-generated data: "higher-quality generated data reduces the need for replayed experience" (Appendix F). The Llama3 base model starts from a higher-quality policy (18.20% vs. 12.69% LC win rate), so its self-generated responses are more reliable, requiring less offline anchor data. This is an intuitively sensible relationship: better models can trust their own judgments more.

Experience replay as a distribution-level intervention, not a loss-level intervention: the mixing happens at the data level—the training dataset itself is a mixture—not at the loss function level. The DPO training loss (Eq. (3)) is applied identically to both offline and self-generated preference pairs. This is simpler than alternative approaches such as adding a separate regularization term for offline data, and it naturally falls out of standard experience replay techniques.

Comparison to Deep Q-learning from Demonstrations (Hester et al., 2018): the paper explicitly draws this analogy (Section 3.2). In DQfD, an RL agent is trained on a mixture of offline demonstration data and online RL experience, with the demonstrations providing a stable learning signal that accelerates early learning and prevents catastrophic forgetting. DICE applies the same principle to preference optimization: offline preference pairs are the "demonstrations," and self-generated data labeled by the implicit reward is the "online experience."

DPO Training Within Each Iteration

Once the mixed dataset $\mathcal{D}_t$ is constructed, standard DPO training is performed to obtain the updated policy $\pi_{\theta^{(t)}}$ (Algorithm 1, line 6):

θ(t)←arg⁡min⁡θ −E(x,yw,yl)∼Dt[log⁡σ(βlog⁡πθ(yw∣x)πθ(t−1)(yw∣x)−βlog⁡πθ(yl∣x)πθ(t−1)(yl∣x))]\theta^{(t)} \leftarrow \arg\min_\theta \, -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_t} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\theta^{(t-1)}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\theta^{(t-1)}}(y_l|x)} \right) \right]

Key design choices in this training step:

Reference model selection: the reference model for this round is $\pi_{\theta^{(t-1)}}$—the policy from the previous iteration. This is a crucial detail that distinguishes iterative DPO from offline DPO. In offline DPO, the reference model is typically fixed (the initial SFT model), which means all training rounds optimize the policy relative to the same baseline. In DICE, the reference model shifts with each iteration: each round's DPO training teaches the model to improve relative to its previous self, not relative to the original SFT model. This creates a "ratcheting" effect—each round builds on the improvements of the previous round. The paper defines this clearly: "fine-tune the policy with DPO's objective (Eq. (3)) to obtain the updated policy $\pi_{\theta^{(t)}}$ with reference model $\pi^{(t)}_{\text{ref}} = \pi_{\theta^{(t-1)}}$" (Section 3).

Maintaining the reference model chain: for $t = 1$ (first bootstrapping iteration), the target policy is the base DPO-tuned model $\pi_{\theta^{(0)}}$ (e.g., zephyr-7B-beta or Llama-3-8B-DPO), and the reference policy is $\pi_{\theta^{(-1)}}$, which is the SFT model used in the original DPO training (e.g., mistral-7b-sft-beta for Zephyr). For $t = 2$, the target policy is $\pi_{\theta^{(1)}}$ (the output of iteration 1), and the reference policy is $\pi_{\theta^{(0)}}$. This chaining ensures the implicit reward used for dataset construction in iteration $t$ always compares the current policy against its immediate predecessor, keeping the reward signal local and on-policy.

Training hyperparameters: the paper specifies these in Section 4.1:

  • Training steps per iteration: 300 steps on a dataset of approximately 9,600 preference pairs.
  • Global batch size: 32 (meaning 32 preference pairs per gradient step).
  • Learning rate: $5 \times 10^{-7}$ with a constant schedule and a warm-up of 50 steps.
  • Optimizer: not explicitly named, but standard practice in DPO training (likely AdamW given the learning rate magnitude, but the paper does not specify the exact optimizer beyond referencing the training protocol).
  • $\beta$ hyperparameter: swept per iteration per method per model from $\{0.01, 0.1\}$, selected based on AlpacaEval 2 LC win rate.
  • Number of iterations: total of 2 DICE iterations (meaning the base DPO model is trained once, then DICE runs two additional rounds). The initial DPO training is considered "round 0." The paper reports results for Iter 1 and Iter 2.

Why 300 steps: the paper does not explicitly justify this number, but it is likely chosen to match the epoch budget of the original DPO training while keeping the total computation manageable (all experiments use 8 Nvidia A100 GPUs). With $N \approx 9,600$ pairs and batch size 32, 300 steps process approximately $300 \times 32 = 9,600$ pairs, which is exactly one epoch over the dataset. This suggests the paper trains for exactly one epoch per iteration—a deliberate choice to prevent overfitting to the self-generated data.

Why the constant learning rate with warm-up: the warm-up of 50 steps allows the optimizer to gradually increase the learning rate from zero to $5 \times 10^{-7}$, avoiding destabilizing large gradients at the start of training. The constant schedule thereafter maintains a steady optimization signal. This is a conservative training regimen appropriate for fine-tuning an already-aligned model, where large learning rates could cause catastrophic forgetting.

The role of $\beta$ in training vs. in reward construction: note that $\beta$ appears in two places—in the implicit reward computation (Eq. (5)) and in the DPO training loss (Eq. (3)). The paper hypertunes $\beta$ for each setting, but it is the same value used in both places. This is theoretically consistent: the implicit reward is defined in terms of the same $\beta$ that appears in the DPO objective, because Eq. (4) is derived from Eq. (2) and Eq. (3) with the same $\beta$ controlling the KL penalty. Changing $\beta$ between reward construction and training would be mathematically inconsistent—the reward would be scaled differently than the DPO loss expects.

Initialization Conditions and Termination

What constitutes the initial base model: DICE starts from a model that has already undergone one round of DPO training on a human-labeled preference dataset. Specifically:

  • Zephyr backbone: the base model is zephyr-7B-beta, which is DPO-trained from mistralai/Mistral-7B-v0.1 (SFT) on the full UltraFeedback dataset of 60,000 preference pairs following the Zephyr pipeline (Tunstall et al., 2023). The reference model used in this initial DPO was mistral-7b-sft-beta.
  • Llama3 backbone: the base model is princeton-nlp/Llama-3-Base-8B-SFT-DPO, which is DPO-trained from meta-llama/Meta-Llama-3-8B (SFT + DPO) developed by Meng et al. (2024), also on UltraFeedback.

The offline preference dataset used in DICE: the paper randomly samples a subset of "around 10k preference pairs" from UltraFeedback (Section 4.1: "We randomly sample a subset of around 10k preference pairs from UltraFeedback as the offline dataset $D_{\text{offline}}$ for our fine-tuning experiments"). This is approximately 1/6 of the full UltraFeedback dataset. The explicit goal is "to show how much the language model can improve from a DPO-tuned model and a subset of the preference dataset that was used to conduct the initial DPO training." This is a deliberate constraint: DICE is tested under conditions where the available data is strictly a subset of what was used for initial alignment—no new data, no external models.

Why 2 iterations and not more (Table 6, Appendix B): the paper experiments with a third iteration and finds that performance not only stops improving but degrades sharply. For Zephyr, DICE Iter 3 drops to 14.02% LC win rate (from 20.71% at Iter 2); for Llama3, it drops to 20.61% (from 27.55%). This mirrors the behavior observed with LLM-as-a-Judge (Appendix B, Table 6), and the paper acknowledges this as a known challenge: "we did not observe continuous improvement in our model beyond three iterations" (Section 5). The paper attributes this partly to using the same fixed prompt set in every iteration, hypothesizing that "using different prompt sets across iterations could help mitigate the performance drop." This limitation is important: DICE provides substantial short-term gains (two rounds) but does not enable indefinite iterative self-improvement.

The fixed prompt set issue: the prompts used for response generation are the same in every iteration (extracted from $D_{\text{offline}}$). This means the model repeatedly trains on preference data for the same set of prompts, potentially overfitting to those specific tasks and losing generalization to unseen prompts. Using a rotating or expanding prompt set across iterations would address this but would require additional prompt data not available in the constrained setting.

Summary of Design Choices and Their Justifications

  • Implicit reward over prompted self-judgment: the implicit reward is a continuous, mathematically grounded signal derived from the DPO objective, requiring no additional training or prompt engineering, and empirically outperforms the discrete 0–5 score from LLM-as-a-Judge (Table 1), especially on stronger base models where coarse scores fail to differentiate high-quality responses.
  • Length-regularized reward shaping over loss-regularized DPO: by operating at dataset construction time rather than training time, DICE decouples length debiasing from model training, avoiding the expensive hyperparameter tuning loop of Park et al. (2024) and enabling fast black-box optimization of $\alpha^*$ via dataset analysis alone.
  • $\alpha$ optimization via Bayesian search over grid search: the objective is non-differentiable (the dataset construction involves $\arg\max$ and $\arg\min$ over discrete responses), making gradient-based optimization impossible, while Bayesian optimization efficiently finds the zero-crossing of the length-difference curve (Figure 4).
  • Extreme pair selection (best vs. worst) over random pairs: selecting the highest and lowest implicit-reward responses maximizes the signal-to-noise ratio of the constructed preference labels, reducing the risk of incorrectly ordering pairs where the implicit reward cannot reliably distinguish quality.
  • Experience replay with $\gamma \in (0, 1)$ over pure self-generation ($\gamma = 0$) or pure offline data ($\gamma = 1$): mixing self-generated on-policy data with offline anchor data balances the benefits of distributional freshness (correcting errors in the current policy's output distribution) against the risk of catastrophic forgetting (drifting away from human-verified preferences), with the optimal $\gamma$ depending on the base model's quality (higher quality needs less anchor data).
  • Moving reference model ($\pi_{\theta^{(t-1)}}$) over fixed reference ($\pi_{\text{SFT}}$): shifting the reference model with each iteration creates a ratcheting effect where each round of DPO teaches the model to improve relative to its previous self, enabling cumulative gains rather than repeatedly optimizing relative to the same fixed baseline (which would quickly saturate).
  • One epoch per iteration over multiple epochs: training for exactly $N / (\text{batch size})$ steps ensures the model sees each self-generated preference pair exactly once, preventing overfitting to potentially noisy implicit-reward labels while still providing enough signal to shift the policy.
  • Two iterations over indefinite iteration: the observed performance collapse at iteration 3 (Table 6) suggests that the benefits of self-alignment saturate and eventually reverse, likely due to accumulating implicit-reward noise and fixed-prompt overfitting, making 2 rounds the practical sweet spot.

4. Key Insights and Innovations

Innovation 1: DPO Implicit Rewards as a Previously Overlooked, Zero-Cost Self-Supervision Signal for Iterative Alignment

The paper's most fundamental conceptual contribution is the recognition that the implicit reward function induced by DPO — a mathematical byproduct that had been noted in the original DPO paper but largely treated as a theoretical curiosity — is a practically usable, sufficiently reliable preference signal that can drive multiple rounds of continued alignment improvement without any external feedback. This is not an incremental algorithmic improvement over prior iterative DPO methods; it is a reframing of what resources are available for self-alignment.

The dominant assumption before this work: The iterative DPO framework (Tran et al., 2023) established that on-policy sampling improves alignment, but the field's default assumption was that obtaining preference labels for on-policy generations required either (a) a separately trained external reward model, (b) prompting an external LLM judge like GPT-4, or (c) training the model itself to be a judge via supervised fine-tuning on evaluation data (as in self-rewarding LMs, Yuan et al., 2024). Each of these paths introduces dependencies — on additional training data, external APIs, or separate model artifacts. The implicit reward was known to exist mathematically (Rafailov et al., 2024b, Eq. (4)) and had been used for token-level credit assignment (Rafailov et al., 2024a) and as a standalone reward for PPO training (Zhong et al., 2024), but no prior work had recognized it as a bootstrapping signal for iterative self-alignment. The paper is the first to close the loop: the model generates responses, its own implicit reward scores them, the extremes become preference pairs, and another round of DPO proceeds — all without a single additional model, API call, or training run beyond what already exists after the initial DPO alignment.

Why this is a fundamental shift, not an incremental tweak: This changes the economics of alignment improvement from "each round of improvement requires fresh annotation or external inference" to "each round of improvement requires only compute for on-policy generation and forward passes through two models you already have." The implicit reward has been hiding in plain sight since Rafailov et al. (2024b) — it is defined in Eq. (4) of that paper and validated theoretically by Theorem 1. But the field's attention had been on what the implicit reward represents theoretically (a reparameterization of the reward in the Bradley-Terry model) rather than on what it enables practically (a zero-dependency self-supervision loop). DICE's core move is recognizing that Theorem 1 isn't just a proof technique — it's an engineering primitive. Once DPO training completes, the implicit reward is a function you can evaluate at any (x, y) pair with two forward passes, and it is correlated with human preference by construction of the DPO objective. The paper demonstrates that this correlation is strong enough to serve as the sole preference signal for up to two additional rounds of improvement.

Evidence anchoring the claim: The empirical results in Table 1 are the cleanest demonstration. On both Zephyr-7B-beta and Llama-3-8B-DPO, DICE (which uses only implicit rewards for self-generated data labeling) substantially outperforms LLM-as-a-Judge (which prompts the model to produce discrete 0–5 quality scores). For the Llama3 backbone, DICE Iter 2 reaches 27.55% LC win rate versus 21.80% for LLM-as-a-Judge — a gap of nearly 6 percentage points. This is particularly striking because LLM-as-a-Judge uses the same model's own judgment capabilities (via prompting) and should, in principle, have access to the same underlying knowledge. The gap suggests that the implicit reward — a continuous, log-probability-ratio-based signal — captures preference information that the model cannot articulate when asked to produce an explicit quality score in natural language. The comparison with a separately trained internal reward model (Table 5) reinforces the point: the DPO implicit reward achieves 0.698 alignment rate with GPT-4o labels, compared to 0.624 for a scalar reward model trained on the same offline data and 0.656 for an external reward model trained on 9× more data. The implicit reward is not just convenient — it is competitive with purpose-built reward models on its own generated distribution.

A subtle but critical aspect of why this works: The implicit reward operates in probability space — it measures how much more (or less) the current policy favors a response compared to the reference policy. This means it is inherently on-policy: the signal reflects the current model's own assessment of quality relative to its own past state. When the model drifts during self-alignment, the implicit reward drifts with it, because it is computed from the current policy. In contrast, a separately trained reward model is frozen — its judgments become increasingly off-policy as the policy evolves. This property means the implicit reward may actually be more suitable for iterative self-alignment than a static reward model, even if the static reward model is more accurate in an absolute sense on a fixed test set. The paper doesn't fully develop this argument (it's implicit in the theoretical analysis of Appendix A), but it helps explain why a reward signal derived from DPO + on-policy generation can drive improvement over multiple rounds when offline data degrades so catastrophically.

Innovation 2: Dataset-Level Length Debiasing via Black-Box Optimization, Decoupling Length Regularization from Training

Prior work on length exploitation in preference tuning — notably Park et al. (2024) — addressed the problem by incorporating a length penalty into the DPO training objective itself. This couples the length regularization hyperparameter (λ) to the training process: each candidate value of λ requires training a full model and evaluating it, creating an expensive tuning loop. DICE's innovation is to move length debiasing upstream from training time to dataset construction time, making it a pre-processing step that can be optimized without any model training whatsoever.

The intellectual move: The paper reframes length exploitation not as a training pathology to be regularized away in the loss function, but as a dataset bias problem. The implicit reward systematically assigns higher scores to longer responses (Figure 2, top: mean length difference of +1,031 characters between winners and losers under vanilla implicit rewards). If you can debias the dataset — making the preference signal orthogonal to length — then standard unmodified DPO training will not amplify length bias, because it never receives a biased signal to learn from. This separation of concerns (debiasing the data vs. training the model) is the conceptual contribution.

What makes this more than an engineering convenience: The decoupling has theoretical significance. In the Park et al. (2024) approach, the length regularization term competes with the preference optimization term in the loss function — the trainer must balance two objectives (preference alignment vs. length penalty) via a scalar λ that has no natural scale or interpretation. Finding the right λ requires training multiple models and evaluating them on downstream benchmarks, which is both computationally expensive and vulnerable to overfitting the λ selection to the specific evaluation metric. In DICE's approach, the optimization target is directly interpretable: α* is the penalty strength that makes the average length difference between winning and losing responses equal to zero. This is a property of the dataset, not of the downstream task, so it can be evaluated purely by analyzing the constructed preference pairs — no model training, no benchmark evaluation. The optimizer (Bayesian search over a single scalar α) converges quickly because the objective landscape (Figure 4, left) is smooth and well-behaved.

The diagnostic move is itself an innovation: The paper's use of the length-difference distribution (Figure 2) as a dataset quality diagnostic is a conceptual contribution independent of the specific solution. By visualizing the distribution of |y_w| - |y_l| and comparing it against a known-good reference (UltraFeedback, which is nearly centered at zero), the paper provides a concrete, quantifiable criterion for whether a preference dataset is length-biased. This diagnostic can be applied to any preference dataset, regardless of how it was constructed, giving the community a tool for auditing dataset quality. The observation that LLM-as-a-Judge does not exhibit this bias (Figure 4, right: α* = 0 is optimal for that method) is an interesting finding in its own right — it suggests prompted model judgments may be more length-calibrated than DPO implicit rewards, possibly because the judge prompt template forces explicit evaluation along content criteria.

Evidence anchoring significance: The ablation in Table 4 demonstrates that the choice of α matters substantially and that the optimization procedure finds the right value. With γ = 0 (pure self-generated data), using α = 0 (no length debiasing) yields 13.32% LC win rate despite a high raw win rate of 15.37% — the length-controlled metric penalizes the model for its verbosity. Using the optimized α* = 0.023 jumps to 18.88% LC win rate, a gain of over 5.5 percentage points. Critically, doubling α to 0.046 over-corrects: the raw win rate collapses to 9.08% and the average response length drops to 876 characters (from 2,570 without debiasing), indicating the model is now punished for producing any substantive content. This U-shaped relationship between α and downstream performance validates both the existence of a length-bias problem and the correctness of the optimization procedure in finding the sweet spot. The matching result for γ = 0.5 (19.03% LC win rate at α* vs. 15.92% at α = 0) confirms that length debiasing is necessary regardless of the experience replay ratio — it addresses an orthogonal problem.

Innovation 3: Experience Replay as a First-Class Safeguard in Iterative Self-Alignment, Not Just a Continual Learning Afterthought

Experience replay is a well-known technique from continual learning (Rolnick et al., 2019) and deep RL (Mnih et al., 2015), and the idea of mixing offline and online data appears in prior RLHF-adjacent work (Hester et al., 2018). What makes DICE's use of experience replay a genuine contribution is the diagnosis of why it is necessary specifically in the iterative self-alignment setting and the empirical demonstration that the optimal replay ratio is non-trivial and model-dependent — it is not "include some offline data to be safe," but rather a tunable parameter that controls a fundamental tradeoff between on-policy freshness and offline stability.

The conceptual framing: The paper identifies that self-alignment with imperfect rewards faces a dual risk that differs from standard catastrophic forgetting. On one side, pure self-generation (γ = 0) risks reward hacking: the model may learn to exploit blind spots in the implicit reward, producing responses that score highly under its own reward function but are not actually preferred by humans. This is the "reinforcement of errors" problem. On the other side, pure offline data (γ = 1) risks staleness: the training data becomes increasingly off-policy as the model evolves, and the theoretical analysis in Appendix A shows this can lead to arbitrarily low probability on the optimal response even when the training loss is minimized to zero. The experience replay ratio γ controls the balance between these two failure modes. This framing — experience replay as a tunable defense against two distinct and opposing pathologies — is more nuanced than the standard continual learning narrative of "mixing old data prevents forgetting."

Why this is a finding, not just an engineering choice: The paper shows that the optimal γ varies substantially by base model: γ = 0.5 for Zephyr (LC win rate 20.71%) vs. γ = 0.1 for Llama3 (LC win rate 27.55%). This is not arbitrary — the paper provides a plausible mechanism: stronger base models produce higher-quality self-generated data, and therefore need less offline anchoring. The Llama3 base model starts at 18.20% LC win rate (vs. 12.69% for Zephyr), so its implicit rewards are more trustworthy, and its self-generated responses are less likely to contain systematic errors that need correction from offline data. This relationship suggests a meta-principle: the quality of the self-supervision signal (implicit reward accuracy on on-policy data) determines how much external anchoring is needed. This principle could guide the application of DICE-like methods to future models without requiring an expensive γ sweep — if your base model is strong, use less replay; if it's weak, use more.

Evidence anchoring significance: Figure 3 (and Appendix D, Figure 5) are the key empirical demonstrations. The U-shaped curve of performance vs. γ is not shallow — the difference between optimal and suboptimal γ is large. For Zephyr, the gap between the best (γ = 0.5, 20.71% LC) and the worst settings (γ = 1.0, 13.40% at Iter 1, collapsing to 4.58% at Iter 2) is over 16 percentage points. This is not a minor tuning detail; getting γ wrong can completely destroy progress. The fact that γ = 1.0 (pure offline data with updated reference) performs so poorly is the strongest evidence that experience replay is not merely a stabilizer — it is essential for the method to function. Without self-generated on-policy data, the model's performance degrades even faster than the base model, confirming the theoretical prediction that offline data becomes increasingly stale.

Broader implication beyond DICE: The experience replay finding has implications for any iterative self-improvement method. If the paper had found that γ = 0 (pure self-generation) worked best, the implication would be "self-alignment with implicit rewards is so reliable that offline anchoring is unnecessary." The actual finding — that intermediate γ is optimal and varies by model — is more nuanced and more useful. It means that future self-alignment methods should not treat offline data as disposable after the first round. The original human-verified preferences retain value throughout the iterative process, not as the primary training signal but as a form of regularization against reward model drift. This is a conceptual contribution independent of the specific mechanism (implicit rewards) used to generate the self-supervised data.

Innovation 4: Reconciling the Iterative DPO Degradation Puzzle Through the Lens of Distribution Staleness

The paper's theoretical analysis in Appendix A provides a clean, mechanistic explanation for a phenomenon that had been observed but not well-understood: why continuing DPO training on a fixed offline dataset causes catastrophic performance degradation (Table 1: Zephyr offline DPO Iter 2 collapses to 1.89% LC win rate), while iterative DPO with on-policy sampling improves performance. The explanation — that suboptimal responses in high-likelihood regions of the current policy may never appear in the fixed training data and therefore never get penalized — is elegant and reframes the problem from "overfitting to the training set" to "distribution mismatch between the policy's output distribution and the training data distribution."

Why this is a conceptual contribution, not just a proof: The field had observed the phenomenon. Guo et al. (2024) showed that offline DPO quickly overfits; Tajwar et al. (2024) demonstrated that on-policy sampling enhances performance. But the standard explanation was "overfitting" — a generic diagnosis that doesn't point to a specific mechanism or suggest a targeted solution beyond "use on-policy data." DICE's analysis identifies the precise failure mode: the offline dataset D_offline is a finite sample from a behavior distribution π_µ that may have zero probability mass on responses that the current policy π_{θ^{(t)}} assigns high probability to. If such a response is suboptimal (as measured by the true Bradley-Terry reward), the DPO objective — which only operates on pairs that appear in the training data — has no mechanism to reduce its probability. The model's probability mass on this response can remain arbitrarily high indefinitely, even as the training loss goes to zero.

What this explains that "overfitting" doesn't: The specific prediction of this analysis is that degradation should be non-uniform across prompts — it should be worst for prompts where the model's distribution has shifted furthest from the behavior distribution, and minimal for prompts where the two distributions still overlap. This is a testable prediction. The analysis also explains why updating the reference model (Offline DPO w/ new ref) doesn't help: the staleness is in the preference pairs themselves (which responses are compared as y_w and y_l), not in the reference model used in the DPO loss. Changing the reference model while keeping the same preference pairs still means the model never sees (and can never penalize) its own high-likelihood errors.

The implication for iterative methods more broadly: The analysis formalizes why on-policy sampling is not just beneficial but necessary for continued improvement. Any method that claims to iteratively improve a model using a fixed dataset of preferences — regardless of the specific algorithm (DPO, IPO, KTO) — will eventually hit this wall. The necessary condition for continued improvement is that the preference data reflects the current policy's output distribution, which in turn requires either (a) regenerating responses from the current policy and relabeling them (as DICE does) or (b) using a reward model that generalizes to the current policy's outputs (which a frozen scalar reward model may do if it's well-trained, though Table 5 suggests the implicit reward is competitive in this role).

Evidence anchoring significance beyond the proof: The empirical results in Table 1 provide the experimental confirmation. Offline DPO (fixed reference) and Offline DPO w/ new ref both collapse by Iter 2, with the degradation being worse for the method that continues to optimize against a stale dataset (offline DPO w/ new ref Iter 2 reaches 4.58% LC for Zephyr). In contrast, DICE — which regenerates responses from the current policy and relabels them with the implicit reward — continues improving, reaching 20.71% at Iter 2. This stark difference (4.58% vs. 20.71%) validates the theoretical claim: the mechanism of improvement is not just "more training" or "better hyperparameters" but fundamentally "training data that matches the current policy's distribution." The on-policy data construction in DICE is not a nice-to-have efficiency boost — it is the necessary condition for any improvement beyond the first round.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use the UltraFeedback dataset (Cui et al., 2023), which contains approximately 60,000 preference pairs. The paper randomly samples a subset of around 10,000 preference pairs as the offline dataset $D_{\text{offline}}$ for all fine-tuning experiments. The prompts used for on-policy generation in each DICE iteration are extracted from this same offline dataset. For evaluation, the primary benchmarks are AlpacaEval 2 (Li et al., 2023b) — which employs the AlpacaFarm prompt set of general human instructions and uses GPT-4-Turbo as both the reference response generator and the annotator — and Arena-Hard (Li et al., 2024), a recently released benchmark with 500 well-defined technical problem-solving queries, evaluated using Mistral-Large-Instruct-2407 (123B) as the judge model following official guidance.

  • Base model(s). Two base models are used, both sharing the property of having undergone exactly one round of DPO training on UltraFeedback prior to any DICE iteration: zephyr-7B-beta, which is DPO-trained from mistralai/Mistral-7B-v0.1 (SFT) on the full UltraFeedback dataset via the Zephyr pipeline (Tunstall et al., 2023); and Llama-3-8B-DPO (princeton-nlp/Llama-3-Base-8B-SFT-DPO), which is DPO-trained from meta-llama/Meta-Llama-3-8B by Meng et al. (2024). These represent two different model families (Mistral and Llama) at a comparable scale (7–8B parameters), chosen to demonstrate that DICE works across architectures and initial DPO training procedures. The reference models used for implicit reward computation are the corresponding SFT checkpoints: mistral-7b-sft-beta for Zephyr, and the initial SFT model for Llama3.

  • Metrics. The primary evaluation metric is length-controlled (LC) win rate on AlpacaEval 2, expressed as a percentage. The LC win rate corrects for the known bias of LLM-based evaluators toward longer responses by fitting a logistic regression model that predicts the annotator's preference from the length difference between the model's response and the reference response, then computing a counterfactual win rate at equal lengths (Dubois et al., 2024). The raw win rate (WR) is also reported for completeness, though the LC metric is the primary one since DICE's length-regularized reward shaping directly targets length exploitation. On Arena-Hard, the reported metric is the LC win rate (score with 95% confidence interval) from the Mistral-Large judge. For the reward model comparison (Section 4.4), the metric is alignment rate — the fraction of preference labels that match GPT-4o's judgments on 500 sampled (x, y_1, y_2) tuples from the generated dataset.

  • Baselines. The paper evaluates four baselines applicable to the self-alignment setting (Table 1): (a) Offline DPO — continuing DPO training with the same fixed offline preference dataset and the same fixed reference model as the initial DPO; (b) Offline DPO with new ref — similar to Offline DPO but with the trained policy after each round serving as the new reference model for the next round (this corresponds to $\gamma = 1$ in DICE's experience replay framework); (c) LLM-as-a-Judge — following the self-rewarding LM approach of Yuan et al. (2024), where the policy model itself is prompted (using the template in Appendix E, Figure 6) to assign a discrete 0–5 quality score to each generated response, and the highest- and lowest-scored responses form preference pairs for iterative DPO; (d) PPO with implicit reward (Section 4.5 / Table 8, mentioned as a brief comparison) — using the DPO implicit reward model as the reward signal for PPO training, following OpenRLHF (Hu et al., 2024) with recommended parameters. The LLM-as-a-Judge baseline is particularly important because it represents the direct alternative to implicit rewards for self-supervised preference labeling — it uses the same model's own judgment but accessed through prompting rather than through the DPO log-probability ratio.

  • Generation budget / compute accounting. In each DICE iteration, the current policy generates $K = 16$ responses per prompt using temperature sampling (Zephyr: $T = 0.7$, $p = 0.9$; Llama3: $T = 0.9$, $p = 1.0$). The prompt set is extracted from $D_{\text{offline}}$ and yields approximately 9,600 preference pairs. For the baseline methods, the same generation budget is used where applicable (LLM-as-a-Judge generates the same $K = 16$ responses per prompt). DPO training in each iteration runs for 300 steps at batch size 32, which corresponds to exactly one epoch over the 9,600-pair dataset. All experiments use 8 Nvidia A100 GPUs. The paper does not convert generation and training costs into a unified FLOPs budget for cross-method comparison, relying instead on the same generation budget (K, prompt set size) and training budget (steps, batch size) across methods to ensure approximate parity.

  • Cross-validation / statistical protocol. Hyperparameters ($\beta$ and $\gamma$) are selected based on performance on AlpacaEval 2 and then applied directly to Arena-Hard without further tuning. The paper validates this approach by treating the two benchmarks as proxies for validation and test sets: when $\gamma$ is selected based on AlpacaEval 2 performance, the same $\gamma$ achieves the best Arena-Hard performance for all four backbone–benchmark combinations (Tables 9 and 10 in Appendix F), and vice versa when Arena-Hard is used for selection. This cross-benchmark consistency provides evidence that the selected hyperparameters are not overfit to a single evaluation. The $\alpha^*$ optimization for length-regularized reward shaping uses Bayesian optimization with Gaussian process regression (gp_minimize from scikit-optimize) and is performed on the dataset construction process itself (no model training required), eliminating any evaluation-benchmark dependency for that parameter. The optimization landscape in Figure 4 confirms a smooth, well-behaved objective.

Main Quantitative Results

DICE Improves Alignment Substantially Over All Baselines Across Two Model Families

Table 1 presents the central comparison. On AlpacaEval 2 LC win rate, starting from the Zephyr base model at 12.69%, the results after two iterations are:

  • Offline DPO Iter 2: 1.89% (catastrophic degradation from 10.17% at Iter 1; the base model's 12.69% is better than any Offline DPO checkpoint)
  • Offline DPO w/ new ref Iter 2: 4.58% (also catastrophic, despite the updated reference model)
  • LLM-as-a-Judge Iter 2: 14.14% (modest improvement over base, but actually a decline from Iter 1's 15.36%)
  • DICE Iter 1: 19.03% (already a 6.34 percentage point gain over base, and 3.67 points above LLM-as-a-Judge's best iteration)
  • DICE Iter 2: 20.71% (an additional 1.68 point gain, totaling +8.02 points over base)

On the Llama3 base model starting at 18.20% LC win rate, the improvements are even larger:

  • Offline DPO Iter 2: 24.04% (modest improvement; Offline DPO does not collapse as catastrophically as with Zephyr, but still plateaus)
  • LLM-as-a-Judge Iter 2: 21.80% (only 3.6 points over base, and the improvement from Iter 1's 20.30% is minimal)
  • DICE Iter 1: 25.08% (+6.88 points over base)
  • DICE Iter 2: 27.55% (+9.35 points over base, and a substantial 5.75 points above LLM-as-a-Judge's best)

The pattern is consistent across both backbones: (1) Offline DPO fails to sustainably improve the model, collapsing catastrophically for Zephyr and plateauing for Llama3; (2) LLM-as-a-Judge provides modest gains but saturates quickly and can even degrade from Iter 1 to Iter 2; (3) DICE provides the largest gains, with improvement continuing from Iter 1 to Iter 2, and the gap over baselines is substantial (DICE Iter 2 beats LLM-as-a-Judge Iter 2 by 6.57 points on Zephyr and 5.75 points on Llama3).

The raw win rate (WR) column shows a complementary pattern: DICE Iter 2 achieves 20.16% (Zephyr) and 30.99% (Llama3), compared to 10.71% and 15.50% for the base models. The fact that the LC win rate also improves substantially — despite the LC metric specifically penalizing verbosity — confirms that the gains are not coming from length exploitation; they reflect genuine quality improvements that survive length debiasing.

Arena-Hard Results Show Consistent but Smaller Gains

Table 1 also reports Arena-Hard LC scores. For Zephyr, DICE Iter 2 reaches 18.45% (up from 10.17% base), compared to LLM-as-a-Judge at 13.47% and Offline DPO (w/ new ref) collapsing to 5.15%. For Llama3, DICE Iter 2 reaches 39.13% (up from 22.32% base), compared to LLM-as-a-Judge at 27.99% and Offline DPO at 23.55%. The Arena-Hard improvements are monotonically consistent with the AlpacaEval 2 rankings, though the absolute gains are somewhat smaller for Zephyr (8.28 vs. 8.02 on AlpacaEval 2) and notably larger for Llama3 (16.81 points on Arena-Hard vs. 9.35 on AlpacaEval 2 — the Arena-Hard improvement is nearly double the AlpacaEval 2 improvement in absolute terms for Llama3).

The 95% confidence intervals reported in Tables 9 and 10 (Appendix F) contextualize these results: for Zephyr with $\gamma = 0.5$ (the optimal setting), the Arena-Hard score is 18.4 with a CI of (-1.3, 1.6), meaning the true score could plausibly be anywhere from 17.1 to 20.0. The overlap between confidence intervals at different $\gamma$ values suggests that while the ranking is stable, the precise magnitudes should be interpreted cautiously. For Llama3 with $\gamma = 0.1$, the Arena-Hard score is 39.1 with a CI of (-2.0, 2.4), and the separation from the next-best $\gamma = 0.0$ (also 39.1) is essentially zero — the two settings are indistinguishable on Arena-Hard despite $\gamma = 0.1$ being clearly superior on AlpacaEval 2 (27.55% vs. 26.93%).

DICE on the AlpacaEval 2 Leaderboard: Competitive with Much Larger Models

Table 2 positions DICE-Llama3 8B Iter 2 (27.55% LC win rate) on the public AlpacaEval 2 leaderboard alongside both open-source and proprietary models. The key comparisons are:

  • DICE-Llama3 8B Iter 2 (27.55%) outperforms Llama 3 8B Instruct (22.92%) by 4.63 points — a nontrivial margin given both are 8B parameter Llama3 variants, with the difference being that DICE applies self-alignment on top of DPO rather than using Meta's official instruction tuning recipe.
  • DICE-Llama3 8B Iter 2 (27.55%) outperforms Gemini Pro (24.38%) by 3.17 points, despite Gemini Pro being a substantially larger proprietary model from a well-resourced frontier lab.
  • DICE-Llama3 8B Iter 2 (27.55%) is competitive with Snorkel (Mistral-PairRM-DPO) (26.39%) — a model trained with an external PairRM reward model on top of Mistral, suggesting DICE's implicit-reward approach can match the performance of approaches that use external reward models.
  • It remains behind Claude 2 (28.15%), Mistral Medium (28.61%), and GPT-4 0613 (30.18%) — all substantially larger or more extensively trained models.

DICE-Zephyr 7B Iter 2 (20.71%) outperforms Llama-3-8B-DPO (18.20%), Vicuna 33B v1.3 (17.58%), and GPT-3.5 Turbo 1106 (19.30%), placing a 7B model from a relatively small research lab ahead of several much larger or more established models.

These leaderboard results are important because they demonstrate that the self-alignment gains from DICE are not merely improvements over a weak baseline — they push models into competitive territory on the current landscape. However, it is noteworthy that DICE-Llama3 8B is compared against models that may not use the same evaluation pipeline or judge model, and the AlpacaEval 2 leaderboard has known sensitivity to the specific GPT-4-Turbo annotator version. The paper does not discuss annotator version pinning.

DICE Works with Multiple Direct Alignment Algorithms, Not Just DPO

Table 3 examines whether the preference dataset generated by DICE (using DPO implicit rewards with length-regularized reward shaping, first iteration, Zephyr setting) can improve alignment when used with alternative Direct Alignment from Preference (DAP) algorithms. The results compare training on the DICE-generated dataset vs. training on the offline dataset, for DPO (the default), IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), and Hinge loss (Zhao et al., 2023):

  • DPO: Offline 13.40% → DICE-generated 19.03% (gap of +5.63 LC, +6.57 WR)
  • IPO: Offline 14.83% → DICE-generated 18.51% (gap of +3.68 LC, +3.33 WR)
  • KTO: Offline 13.92% → DICE-generated 14.88% (gap of +0.96 LC, +1.51 WR)
  • Hinge: Offline 13.51% → DICE-generated 15.92% (gap of +2.41 LC, +3.12 WR)

All four algorithms benefit from the DICE-generated dataset over the offline dataset, but the magnitude of improvement varies substantially. DPO shows the largest gain (+5.63), followed by IPO (+3.68), Hinge (+2.41), and KTO (+0.96). This ordering is intuitive: DPO is the algorithm for which the implicit reward is defined (the implicit reward is the reparameterization that DPO's objective is optimized under), so the preference signal it produces is most naturally aligned with DPO's training objective. IPO, which also uses a pairwise preference loss but with a different functional form (squared loss on the implicit reward difference), still benefits substantially, suggesting the implicit reward signal is informative beyond DPO's specific logistic loss. KTO, which uses a unary preference signal (each response is labeled good or bad, not paired), benefits least — this is expected because the DICE dataset construction explicitly selects extreme pairs (best vs. worst), which provides a pairwise signal that KTO's unary loss cannot fully exploit.

The paper does not report whether DICE can be run iteratively with these alternative algorithms (the results in Table 3 are for a single round, and the algorithm-specific equivalents of DICE's experience replay and implicit reward chaining across iterations would require separate development). This is an acknowledged limitation — Section 5 notes that "DICE works best with DPO as it makes the iterative training possible."

DPO Implicit Rewards Are Competitive with Purpose-Trained Reward Models on Their Own Generated Distribution

Table 5 compares the implicit reward model against two scalar reward models on the task of predicting GPT-4o preference labels for 500 sampled response pairs from the first DICE iteration's generated dataset (Zephyr setting). The alignment rates (fraction of labels matching GPT-4o) are:

  • Internal Reward Model (trained on UltraFeedback, same base model mistral-7b-sft-beta, same training data as the DPO implicit reward): 0.624
  • ERM-555k (external reward model trained by OpenRLHF maintainers on 555k preference pairs — roughly 9× more data): 0.656
  • DPO Implicit Reward: 0.698

The DPO implicit reward achieves the highest alignment with GPT-4o, exceeding the internally trained reward model by 7.4 percentage points (0.698 vs. 0.624) and the 9×-larger-data external reward model by 4.2 percentage points (0.698 vs. 0.656). This is a striking result because the implicit reward has no additional training beyond the original DPO run — it is purely a mathematical function of the DPO-trained policy and its reference model — yet it outperforms purpose-built reward models trained explicitly on preference prediction.

The paper offers an interpretation: "the implicit reward model offers advantages when evaluating its own generated data." This is a form of on-policy advantage — the implicit reward is defined with respect to the current policy and reference model, so its assessments are calibrated to the distribution of responses that policy actually produces. A static reward model, even one trained on much more data, may be less calibrated on the specific distribution of responses from a model it wasn't trained to evaluate. However, the paper also notes that "scalar reward models in general excel on a wider range of tasks when the preference data is abundant," citing ArmoRM-Llama3-8B-v0.1 (trained on 1M preference pairs) as a counterexample — the implicit advantage may shrink or reverse when reward models are trained at larger scale.

It is worth noting that this comparison is limited to 500 examples from a single model's output distribution (Zephyr, first DICE iteration). The relative performance of implicit rewards vs. trained reward models may differ for other models, other iterations, or other tasks. The paper does not report alignment rates for the Llama3 setting or for later DICE iterations.

On-Policy Sampling via DICE Enables Continued Learning Beyond the First DPO Round

The theoretical analysis in Appendix A predicts that training on a fixed offline dataset should degrade as the policy drifts from the behavior distribution, because suboptimal responses in high-likelihood regions of the current policy may never appear in the offline data and therefore never get penalized. The empirical results in Table 1 provide strong confirmation: Offline DPO collapses by Iter 2 (Zephyr: 1.89%; Llama3: 24.04%, which is still below DICE Iter 1's 25.08%). Even Offline DPO w/ new ref — which updates the reference model to track the policy — degrades (Zephyr Iter 2: 4.58%; Llama3 Iter 2: 23.55%).

DICE, which regenerates on-policy responses and relabels them at each iteration, avoids this collapse and continues improving. The comparison is starkest for Zephyr: Offline DPO Iter 2 destroys the model (1.89% LC win rate, far below the base model's 12.69%), while DICE Iter 2 produces the best model (20.71%). This supports the paper's framing that on-policy data is not merely beneficial but is a necessary condition for improvement beyond the first DPO round.

The Llama3 results are less dramatic but still supportive: Offline DPO Iter 2 (24.04%) is above the base model (18.20%) but below DICE Iter 1 (25.08%) and far below DICE Iter 2 (27.55%). The fact that Offline DPO doesn't collapse for Llama3 to the same degree as Zephyr may reflect differences in the base model's output distribution — perhaps the Llama3 base model's outputs overlap more with the offline data distribution, or the offline data was collected from a behavior policy that is closer to the Llama3 distribution. The paper does not analyze this difference.

Ablation Studies and Robustness Checks

Length-regularized reward shaping ($\alpha$): choosing the right penalty strength is critical, and the $\alpha^\star$ optimization procedure finds the optimal value. Table 4 reports AlpacaEval 2 performance for the Zephyr setting (Iter 1) at two experience replay ratios ($\gamma = 0$ and $\gamma = 0.5$) and three length penalty values: no regularization ($\alpha = 0$), the optimized value ($\alpha^\star = 0.023$), and double the optimized value ($\alpha = 0.046$).

At $\gamma = 0$ (pure self-generated data): $\alpha = 0$ yields 13.32% LC win rate with an average response length of 2,570 characters — the model produces verbose outputs and is heavily penalized by the LC metric (despite a 15.37% raw win rate, the LC win rate is much lower). $\alpha^\star = 0.023$ yields 18.88% LC win rate with an average length of 2,109 characters — a substantial improvement of over 5.5 LC percentage points, with a moderate reduction in length. $\alpha = 0.046$ yields 14.40% LC win rate with an average length of only 876 characters — the raw win rate collapses to 9.08%, indicating the length penalty is now so strong that it suppresses quality along with length. This U-shaped relationship confirms both that length debiasing is necessary (substantial gain at $\alpha^\star$) and that over-regularization backfires (degradation at $2\alpha^\star$).

At $\gamma = 0.5$ (mixed data), the pattern is similar: $\alpha = 0$ yields 15.92% LC (length 2,600), $\alpha^\star$ yields 19.03% LC (length 1,848), and $\alpha = 0.046$ yields 14.91% LC (length 1,185). The optimal $\alpha^\star$ improves over no regularization by 3.11 LC points, and over double-$\alpha^\star$ by 4.12 points.

The interaction between $\alpha$ and $\gamma$ is noteworthy: the degradation from no regularization to $\alpha^\star$ is larger at $\gamma = 0$ (+5.56 LC points) than at $\gamma = 0.5$ (+3.11 LC points). This makes sense — when the dataset is pure self-generated data ($\gamma = 0$), the length bias of the implicit reward is the dominant factor shaping the training data, so correcting it yields larger gains. When offline data is mixed in ($\gamma = 0.5$), the offline data (which is approximately length-unbiased, as shown in Figure 2 bottom) already provides some length calibration, so the marginal benefit of reward shaping is smaller but still substantial.

Experience replay ratio ($\gamma$): intermediate values outperform extremes, with the optimum varying by base model quality. Figure 3 (Zephyr, AlpacaEval 2) shows the LC win rate across $\gamma \in \{0.0, 0.25, 0.5, 0.75, 1.0\} for two DICE iterations. At Iter 2, the optimal $\gamma = 0.5$ achieves 20.71% LC. Pure self-generated data ($\gamma = 0$) achieves 18.88% — a gap of 1.83 points. Pure offline data with updated reference ($\gamma = 1.0$) achieves only 13.40% at Iter 1 and collapses to 4.58% at Iter 2 (not shown directly on the figure for Iter 2 because the figure only extends to 5% at the bottom, but the Iter 2 point for $\gamma = 1.0$ would be far below the visible range — this is documented in Table 1).

Figure 5 (Appendix D, Llama3) shows a different pattern: the optimal ratio is $\gamma = 0.1$ (achieving 27.55% at Iter 2), with $\gamma = 0.0$ close behind at 26.93%. Higher ratios degrade more steeply: $\gamma = 0.5$ drops to 23.59% and $\gamma = 1.0$ drops to 22.50% at Iter 2. The paper attributes this difference to base model quality: "higher-quality generated data reduces the need for replayed experience" (Appendix F). The Llama3 base model (18.20% LC) is substantially stronger than the Zephyr base (12.69% LC), so its self-generated preference data is more reliable, and less offline anchoring is needed.

The Arena-Hard results (Tables 9 and 10) broadly confirm the same optimal $\gamma$ values, though with wider confidence intervals and less separation between adjacent $\gamma$ values. For Zephyr (Table 9), $\gamma = 0.5$ achieves 18.4 Arena-Hard LC with a 95% CI of (-1.3, 1.6), compared to $\gamma = 0.0$ at 17.6 (-1.4, 1.7) — the confidence intervals overlap substantially. For Llama3 (Table 10), $\gamma = 0.1$ and $\gamma = 0.0$ both achieve 39.1 Arena-Hard LC — effectively tied. This suggests that while the optimal $\gamma$ is stable across benchmarks, the precise value matters less when the base model is stronger (since even $\gamma = 0$ is close to optimal for Llama3).

Iterations beyond two: performance degrades sharply at Iter 3. Table 6 (Appendix B) extends both DICE and LLM-as-a-Judge to a third iteration. For Zephyr, DICE Iter 3 drops to 14.02% LC win rate (from 20.71% at Iter 2) — now only 1.33 points above the base model (12.69%). For Llama3, DICE Iter 3 drops to 20.61% (from 27.55% at Iter 2) — still above the base model's 18.20% but a substantial regression from the peak. LLM-as-a-Judge shows the same pattern: Iter 3 drops to 13.95% (Zephyr) and 16.86% (Llama3), the latter being below the base model.

The paper does not fully diagnose why Iter 3 degrades, but offers a hypothesis: the fixed prompt set may be a contributing factor ("using different prompt sets across iterations could help mitigate the performance drop observed in the third iteration"). This is plausible — by Iter 3, the model has been trained on preference pairs for the same prompts three times (once in the initial DPO, twice in DICE), which could lead to overfitting that harms generalization. Alternatively, the implicit reward may accumulate noise across iterations, or the model may begin to exploit degenerate patterns in the reward function that don't manifest as within-iteration performance degradation but surface on out-of-distribution evaluation.

The implicit reward can be used with PPO, but the gains are smaller than with DPO. Table 8 reports a brief experiment where the Zephyr implicit reward model is used as the reward signal for PPO training (using the OpenRLHF implementation, recommended parameters). PPO with implicit reward achieves 13.32% LC win rate (vs. 12.69% base), a slight improvement of 0.63 points. This is much smaller than DPO with the same implicit reward (19.03% at Iter 1, Table 1). The paper notes that "PPO's performance can potentially be enhanced with additional hyperparameter tuning" and leaves optimization to future work. This result suggests that the implicit reward's effectiveness is partly coupled to DPO as the training algorithm — the reward and the training objective are derived from the same mathematical framework, and using the reward with a different optimizer (PPO) loses some of the alignment between the reward signal and the policy update.

The LLM-as-a-Judge baseline does not exhibit length bias detectable by the $\alpha$-optimization procedure. Figure 4 (right panel, Appendix C) shows the optimization landscape for LLM-as-a-Judge rewards under the same $\alpha$-search procedure. Unlike the implicit reward landscape (left panel, which shows a clear minimum at $\alpha \approx 0.023$), the LLM-as-a-Judge landscape is monotonically increasing from $\alpha = 0$, meaning the optimal $\alpha^\star = 0$. The policy model's own prompted quality judgments do not systematically favor longer responses in a way the linear penalty can detect. The paper uses $\alpha = 0$ (no debiasing) for all LLM-as-a-Judge experiments. This is an interesting diagnostic finding — prompted self-evaluation may be more length-calibrated than DPO implicit rewards, perhaps because the judge prompt forces explicit evaluation along content criteria. However, as Table 1 shows, this length calibration comes at the cost of overall accuracy: the LLM-as-a-Judge preference signal is weaker overall (lower LC win rate gains).

Comparison between DICE's reward shaping and Park et al.'s (2024) length-regularized DPO. Table 7 directly compares two methods for mitigating length exploitation in the Zephyr setting with $\gamma = 0$. The "No mitigation" baseline (vanilla implicit rewards, $\alpha = 0$) achieves 13.32% LC win rate with 2,570 average characters. Regularized DPO (Park et al., 2024) with $\lambda = 0.02$ achieves 16.30% LC with 2,629 characters — some improvement in LC but length is actually slightly higher. With $\lambda = 0.05$, Regularized DPO achieves 16.03% LC with 2,030 characters — length drops but quality (LC win rate) actually decreases slightly from $\lambda = 0.02$. DICE's reward shaping ($\alpha^\star = 0.023$) achieves 18.88% LC with 2,109 characters — both higher quality and lower length than any Regularized DPO setting. The paper frames this as evidence that dataset-level debiasing (DICE) is more effective than loss-level regularization (Park et al.), and notes the practical advantage that $\alpha^\star$ is found without training models at multiple $\lambda$ values.

Critical Assessment

Does DICE demonstrate that DPO implicit rewards enable effective iterative self-alignment, as claimed?

The experiments strongly support this claim for the specific setting tested: two DPO-tuned 7-8B models, UltraFeedback as the offline dataset, K=16 generations per prompt, and exactly two bootstrapping iterations. The evidence is multi-faceted: DICE substantially outperforms all baselines (Table 1), the gains are consistent across two model families (Zephyr +6.3 to +8.0 LC points; Llama3 +6.9 to +9.4 LC points), the gains survive length debiasing (Table 4), and the gains transfer to a different benchmark (Arena-Hard, Table 1).

However, there are genuine limitations to the generality of this claim that the experiments do not address:

Single preference dataset, single task domain. All experiments use UltraFeedback, which consists of general instruction-following tasks. The paper does not test whether DICE works on domain-specific preference data (code generation, safety, summarization, mathematics). The length exploitation problem that motivates length-regularized reward shaping is particularly acute for open-ended text generation — it may be less relevant or even harmful for tasks where longer responses are genuinely better (e.g., chain-of-thought reasoning where more steps equals more accuracy). The $\alpha^\star$ optimization would blindly debias length even in domains where length and quality are genuinely correlated, potentially degrading performance. The paper does not discuss this domain-dependence caveat.

Single scale regime (~7-8B parameters). Both base models are in the 7-8B range. The paper does not test whether DICE works for smaller models (where implicit rewards may be less reliable) or larger models (where self-generated data quality may be higher, potentially changing the optimal $\gamma$). The finding that $\gamma^\star$ varies from 0.5 (Zephyr) to 0.1 (Llama3) suggests scale dependence, but without more data points, the relationship between model capability and optimal $\gamma$ is suggestive rather than established.

Two iterations only, with known degradation at three. Table 6 shows DICE performance collapsing at Iter 3 for both backbones. The paper's central claim is about the viability of iterative self-alignment, but "iterative" here means exactly two rounds — beyond which the method breaks. This is a significant qualification. The paper presents DICE as an ongoing self-improvement loop, but the empirical evidence supports "self-alignment works for one additional round beyond the initial DPO, and a second round provides a small additional boost, but the margin is shrinking and reverses at the third round." A reader expecting indefinite iterative improvement would be misled. The paper acknowledges this in Section 5 but frames it as an open question rather than a fundamental limitation — which is fair but important to note.

Does DICE demonstrate that length-regularized reward shaping effectively debiases the constructed preference dataset without impairing quality?

Supported with clear diagnostics and ablation evidence. Figure 2 provides a crisp visual: the vanilla implicit reward produces a heavily right-skewed length-difference distribution (mean +1,031 characters), while the regularized reward produces a nearly symmetric distribution centered at -21 characters. The ablation in Table 4 confirms that $\alpha^\star$ achieves the best LC win rate, that $\alpha = 0$ underperforms due to length exploitation (high raw win rate but penalized LC win rate), and that $\alpha = 2\alpha^\star$ over-corrects, collapsing quality.

One concern: the $\alpha^\star$ optimization minimizes the absolute average length difference, which produces an unbiased dataset with respect to average length. However, this does not guarantee that the dataset is unbiased in all respects — it only addresses the first moment of the length-difference distribution. If length exploitation also affects higher moments (e.g., the variance or the tail behavior), the optimization objective would not detect or correct it. The paper does not examine whether other distributional properties of the constructed dataset match the offline reference.

A more subtle concern: the paper implicitly assumes that a length-unbiased dataset is optimal for downstream alignment quality. This is plausible but not proven. It's possible that some length bias is actually beneficial — perhaps better responses are genuinely somewhat longer on average, and completely debiasing length removes a weak but real quality signal. The paper's evidence (Table 4) shows that $\alpha^\star$ (zero mean length difference) outperforms both $\alpha = 0$ (positive bias) and $\alpha = 2\alpha^\star$ (negative bias), which supports the optimality of zero bias for this setting, but this is a single data point and may not generalize.

Does DICE demonstrate that experience replay prevents catastrophic forgetting during iterative self-alignment?

Supported, but the evidence is more correlational than mechanistic. Figure 3 shows that intermediate $\gamma$ values outperform extremes, which is consistent with the claim that mixing offline data prevents forgetting while still allowing on-policy improvement. However, the experiment does not directly measure "forgetting" — it measures downstream LC win rate, which conflates prevention of forgetting with improvement from new data. A cleaner demonstration would involve: (a) showing that the model at $\gamma = 0$ loses capabilities that were present in the base model (e.g., performance on specific task categories degrades) while $\gamma = 0.5$ preserves them; or (b) directly measuring the KL divergence from the base model at different $\gamma$ values to quantify drift. Without such analyses, the claim that experience replay prevents catastrophic forgetting is supported by the aggregate performance pattern but not by direct evidence of what specific knowledge is being preserved.

Additionally, the paper frames experience replay as mixing offline preference pairs (Section 3.2: "a mixture of the generated data and the offline preference dataset"). But the mixing is at the data level — the same prompts appear in both the offline and generated datasets. This means the "experience replay" is not replaying arbitrary old data; it's replaying preference pairs for the same prompts that are in the self-generated data. This is a form of label ensembling rather than classical experience replay (which typically replays state-action trajectories from a buffer). The distinction matters because if the prompts were different, the benefit of experience replay might change — the offline data might provide generalization to unseen prompts rather than stabilizing labels for seen prompts.

Does DICE demonstrate that the improvement comes from implicit rewards specifically, rather than from on-policy sampling combined with any reasonable preference signal?

The comparison with LLM-as-a-Judge (Table 1) is the critical test here, and the results are mixed in their support. For Llama3, DICE substantially outperforms LLM-as-a-Judge (27.55% vs. 21.80% at Iter 2), providing clear evidence that the implicit reward provides a stronger signal than prompted self-evaluation on the same on-policy data. For Zephyr, the gap is also substantial (20.71% vs. 14.14%). This supports the claim that the implicit reward specifically — not just on-policy data — drives the improvement.

However, the LLM-as-a-Judge baseline uses a specific prompt template (Figure 6) that asks for a discrete 0–5 score. It's possible that a different prompt design (e.g., asking for a continuous score, asking for a direct pairwise comparison rather than absolute scores) would produce a stronger signal. The paper's hypothesis that "coarse rewards are not able to provide effective preference signals when responses are of high quality" (Section 4.2) is plausible — the implicit reward is continuous and can make fine-grained distinctions that a 0–5 scale cannot. But this hypothesis is not directly tested (e.g., by comparing 5-point vs. 10-point vs. continuous prompted scales). The implicit reward's advantage over LLM-as-a-Judge might be partly an artifact of the specific prompt template's coarseness, not a fundamental property of implicit rewards vs. prompted self-evaluation.

Does the reward model comparison (Table 5) demonstrate that implicit rewards are genuinely competitive with trained reward models?

The alignment rate comparison on 500 examples from the Zephyr-generated distribution is a useful diagnostic, but it should not be over-interpreted. The implicit reward (0.698) outperforms the internally trained reward model (0.624) and the 9×-larger-data external model (0.656), but the comparison has a key asymmetry: the implicit reward is evaluated on responses generated by the model it was implicitly trained to evaluate (the Zephyr policy), while the scalar reward models were trained on UltraFeedback and then evaluated on these specific model-generated responses. This is a home-field advantage for the implicit reward. The scalar reward models might outperform the implicit reward on a broader distribution of responses (e.g., responses from other models, or from later DICE iterations where the distribution has shifted further). The paper acknowledges this: "scalar reward models in general excel on a wider range of tasks when the preference data is abundant" (Section 4.4). So the claim that "implicit rewards are competitive with trained reward models" holds specifically for evaluating on-policy responses from the model that generated the implicit reward — which is exactly the use case in DICE, but should not be generalized to broader reward modeling applications.

Experiments that would have strengthened the paper

Scaling to more than two iterations with prompt-set rotation. The collapse at Iter 3 is well-documented (Table 6), but the paper only hypothesizes that a fixed prompt set causes the problem. Testing DICE with a different prompt set in each iteration would distinguish between "DICE fundamentally stops working after 2 rounds" and "DICE works for more rounds if you avoid overfitting to a fixed prompt set." This is a feasible experiment (UltraFeedback has more prompts than the ~10k subset used) that would substantially clarify the method's scalability.

Ablation of K (number of generations per prompt). The paper fixes K=16 without justification or ablation. K controls the signal-to-noise ratio of the constructed preference pairs: larger K means the best and worst responses are more extreme and the preference label is more likely to be correct, but also increases generation cost and may increase over-optimization risk. Understanding how K affects DICE's performance would provide practical guidance for deployment and would reveal whether the method is robust to smaller K (which matters for compute-constrained settings).

Direct measurement of catastrophic forgetting. The paper claims that experience replay prevents catastrophic forgetting, but never directly measures forgetting (e.g., by tracking performance on specific task categories, or by measuring how much the model's outputs change on prompts outside the training set). A simple diagnostic would be to evaluate the model on a held-out set of prompts that are never used in training, before and after each DICE iteration. If experience replay reduces the performance drop on these held-out prompts, that directly supports the forgetting-prevention claim.

Comparison against an oracle reward signal. The implicit reward is an imperfect proxy for human preferences. How much better would DICE perform if it had access to ground-truth preference labels for the on-policy generations? This would provide an upper bound on what self-alignment can achieve and would reveal how much of the remaining performance gap is due to implicit reward noise vs. fundamental limitations of the iterative DPO framework.

Tests on non-instruction-following tasks. All evaluation is on instruction-following benchmarks (AlpacaEval 2, Arena-Hard). It is unclear whether DICE's improvements transfer to other capabilities (reasoning, factual accuracy, safety). A model that improves on instruction following might degrade on factual accuracy (a known tension in alignment), and the paper provides no evidence either way.

In summary, the experiments strongly support the paper's central claim that DICE — using DPO implicit rewards, length-regularized reward shaping, and experience replay — enables substantial self-alignment improvements (over 8% LC win rate on AlpacaEval 2) for two rounds of bootstrapping across two moderate-scale model families on the UltraFeedback domain. The evidence for each component (length debiasing, experience replay) is solid within the tested range. The primary limitations are: the method's collapse after two iterations, the single-dataset/single-task evaluation, the lack of mechanistic evidence for experience replay's hypothesized forgetting-prevention role, and the absence of scaling analysis across model sizes, generation budgets (K), or data domains. These do not undermine the paper's contributions — an 8+ LC point improvement on a standard benchmark is genuinely significant — but they bound its claimed generality.

6. Limitations and Trade-offs

The Method Collapses After Two Bootstrapping Iterations, Not Enabling Indefinite Self-Improvement

The constraint: DICE is presented as an iterative bootstrapping procedure, but the empirical evidence shows that performance degrades sharply at the third iteration. Table 6 (Appendix B) documents this clearly: DICE-Zephyr drops from 20.71% LC win rate at Iter 2 to 14.02% at Iter 3 (barely above the base model's 12.69%), and DICE-Llama3 drops from 27.55% to 20.61%. The paper acknowledges this directly in Section 5:

"we did not observe continuous improvement in our model beyond three iterations. This issue highlights an open question within this field regarding the iterative enhancement of policy models."

The consequence: For a practitioner, DICE cannot be deployed as an ongoing self-improvement loop — there is a hard ceiling at two additional rounds, and running a third round actively destroys alignment quality. This means the headline 8%+ improvement is a one-time gain per base model, not a compounding capability. Any deployment that schedules regular retraining cycles would need to carefully monitor iteration count and stop before the cliff. The paper hypothesizes that a fixed prompt set across iterations may be the culprit (Section 5: "using different prompt sets across iterations could help mitigate the performance drop"), but this is untested — a practitioner who rotates prompts has no guarantee the method won't still collapse.

Evidence in the paper: Table 6 provides the iteration-3 results for both DICE and the LLM-as-a-Judge baseline. Both methods degrade at Iter 3, suggesting the limitation is not specific to DICE but reflects a more general challenge with iterative self-alignment. The Arena-Hard results in Table 1 are reported only through Iter 2; Iter 3 Arena-Hard scores are not provided, so the degradation's impact on that benchmark is unmeasured.

Mitigation status: Unresolved. The paper identifies the fixed prompt set as a suspected cause and suggests prompt-set rotation as future work, but no experiments test this. The degradation at Iter 3 is presented as an open question.

Length-Regularized Reward Shaping Assumes a Linear Length Penalty and an Unbiased-Dataset Objective Without Proving Either Is Optimal

The constraint: The length de biasing mechanism uses a linear penalty -α|y| applied to the implicit reward (Eq. (5)), and the optimization objective (Eq. (6)) seeks to drive the average length difference between winning and losing responses to exactly zero. Both choices are pragmatic heuristics, not derived from a principled theory of what constitutes an optimal preference dataset. The paper states this objective in Section 3.1:

"To find the most suitable α such that D(α) is (approximately) unbiased, we optimize α with the objective to minimize the average absolute difference in response length."

The consequence: A zero-mean length difference removes first-order length bias but says nothing about higher-order effects. If longer responses are genuinely better in some domains (e.g., chain-of-thought reasoning where additional steps improve accuracy), the α* optimization blindly penalizes length, potentially suppressing genuinely valuable verbosity. Conversely, if length exploitation manifests in the variance of the length-difference distribution (long-winner pairs are rare but extremely long), the optimization objective would miss this entirely. The linear penalty form -α|y| further assumes that each additional character contributes equally to bias — an assumption that may fail for multi-turn dialogue, structured outputs, or non-English text where character count is a poor proxy for perceived verbosity.

Evidence in the paper: Table 4 and Figure 2 demonstrate that the objective works well for the tested setting — α* produces a nearly symmetric length-difference distribution and yields the best LC win rate. However, the paper does not test alternative debiasing objectives (e.g., minimizing the KL divergence between the constructed dataset's length distribution and UltraFeedback's), alternative penalty forms (e.g., log-length, token count rather than character count, a nonlinear penalty that saturates for very long responses), or domain-transfer scenarios where length is genuinely correlated with quality. The comparison with Park et al.'s (2024) length-regularized DPO in Table 7 shows DICE's approach is better within the tested range, but the comparison is between two linear-penalty methods — neither is tested against a non-parametric or learned debiasing approach.

Mitigation status: None. The paper presents α* optimization as a feature (fast, training-free), not a limitation, but the underlying assumption that zero average length difference is the right target across all domains is never questioned. A practitioner deploying DICE in a domain where longer responses are genuinely better would need to modify or disable the length-regularized reward shaping, but the paper provides no guidance on how to detect such domains or adapt accordingly.

The Difficulty Estimation Cost for Experience Replay Ratio γ Is Hidden in Hyperparameter Tuning and May Not Transfer Across Deployments

The constraint: The experience replay ratio γ is a critical hyperparameter — Figure 3 shows that getting it wrong (e.g., γ=1.0 for Zephyr) can destroy all gains and produce a model worse than the starting point. The paper selects γ by sweeping {0.0, 0.25, 0.5, 0.75, 1.0} (Zephyr) or {0.0, 0.1, 0.25, 0.5, 0.75, 1.0} (Llama3) and selecting the best value based on AlpacaEval 2 performance (Appendix F). The optimal γ varies substantially by base model (0.5 for Zephyr vs. 0.1 for Llama3), and the paper attributes this to base model quality:

"higher-quality generated data reduces the need for replayed experience" (Appendix F).

The consequence: In a real deployment, a practitioner cannot sweep γ by running full DICE iterations and evaluating on AlpacaEval 2 for every new base model — this would require exactly the compute and external evaluation that DICE is designed to avoid. The optimal γ is also likely sensitive to the offline dataset size, the prompt distribution, and the number of iterations. The paper provides no cheaper proxy for selecting γ (e.g., a statistic of the self-generated data that predicts the optimal ratio, or a rule of thumb relating γ to base model performance). A practitioner who guesses γ based on the Zephyr/Llama3 results (e.g., using γ=0.3 as a middle ground) risks substantially suboptimal performance. The finding that γ=0.0 and γ=0.1 are nearly tied for Llama3 on Arena-Hard (both 39.1 LC, Table 10) while showing a clear gap on AlpacaEval 2 (27.55% vs. 26.93%, Figure 5) further complicates selection — the "optimal" γ depends on which benchmark is prioritized.

Evidence in the paper: Tables 9 and 10 (Appendix F) attempt to address transferability by showing that γ selected on one benchmark (AlpacaEval 2) transfers to the other (Arena-Hard). The optimal AlpacaEval 2 γ does achieve the best Arena-Hard score for both backbones, which is reassuring but limited to these two benchmarks. The paper does not test whether γ selected on a small validation subset of AlpacaEval 2 or Arena-Hard transfers to the full benchmark, nor whether γ selected for one model size transfers to another.

Mitigation status: Partial. The cross-benchmark validation in Tables 9 and 10 provides some evidence of stability, but the paper does not propose a method for selecting γ without expensive evaluation. The relationship between base model quality and optimal γ is described qualitatively but not formalized into a predictive rule.

All Experiments Use a Single Preference Dataset (UltraFeedback) and Instruction-Following Benchmarks; Transfer to Other Domains Is Unmeasured

The constraint: Every experiment — base model training, DICE bootstrapping, and evaluation — uses the UltraFeedback dataset and instruction-following benchmarks (AlpacaEval 2, Arena-Hard). The paper does not test DICE on domain-specific preference data (e.g., safety, summarization, code generation, mathematics) or evaluate on non-instruction-following capabilities (reasoning, factual accuracy, toxicity). The base models (Zephyr-7B-beta, Llama-3-8B-DPO) are both trained on UltraFeedback, and the offline dataset used in DICE is a random 10k subset of the exact same data. The paper's scope is stated in Section 4.1:

"Our experiments aim to show how much the language model can improve from a DPO-tuned model and a subset of the preference dataset that was used to conduct the initial DPO training."

The consequence: The paper's findings are valid specifically for general instruction-following alignment on UltraFeedback-like data distributions. A practitioner applying DICE to safety alignment (where the preference signal distinguishes safe from unsafe responses) or code generation (where length is correlated with correctness in non-obvious ways) faces several unknowns: (1) the implicit reward's correlation with domain-specific human preferences may be weaker or differently structured; (2) the length-regularized reward shaping that works for open-ended text may be actively harmful for tasks where longer responses are genuinely better; (3) the experience replay dynamics may change if the offline dataset has different coverage properties; (4) catastrophic forgetting may manifest differently — for instance, self-alignment on instruction following could degrade safety capabilities that were present in the base model, and the paper provides no safety evaluation to detect this.

Evidence in the paper: There is none — the paper does not discuss domain transfer, evaluate on non-instruction tasks, or measure capability preservation in dimensions other than the primary benchmarks. The "Ethics Statement" section is a single sentence acknowledging "many potential societal consequences" without specifying or testing any. This is not a flaw in the experimental design given the paper's stated scope, but it is a significant limitation for practitioners considering deployment.

Mitigation status: None. The paper does not claim cross-domain generality, but it also does not flag this as a limitation. The "Future Work" section (Section 5) focuses on DPO variants and theoretical understanding, not on domain-transfer studies.

The Method Requires Full Access to Both the Policy Model and Its Reference Model, Excluding Deployments Where the Reference Model Is Unavailable

The constraint: The DPO implicit reward (Eq. (4)) is defined as β log π_θ(y|x) / π_ref(y|x) — it requires computing log-probabilities under both the current policy and its reference model. In DICE, the reference model for iteration t is π_{θ^{(t-1)}} (the previous iteration's policy), with t=1 using the original SFT model π_{θ^{(-1)}}. This means DICE requires storing and loading at least two full copies of the model (current policy + reference) throughout training. For Zephyr-7B-beta and Llama-3-8B, this is manageable (two 7-8B models fit on 8 A100 GPUs). For larger models (70B, 405B), the memory and I/O overhead of maintaining both models becomes substantial. Additionally, the implicit reward computation requires two forward passes per scored response, doubling the inference cost of dataset construction compared to methods that use a single reward model or prompted judgment.

The consequence: DICE's cost advantage (no external reward model, no API calls) relies on already having the reference model. For models fine-tuned from public checkpoints (like Zephyr and the Llama3-DPO variant used in the paper), the reference model is available. But for proprietary or API-only models where the SFT checkpoint or intermediate DPO checkpoints are not released, DICE cannot be applied. Even when checkpoints are available, the storage cost of maintaining a chain of reference models across iterations may become prohibitive for very large models. The paper does not discuss whether a distilled or compressed reference model could substitute, or whether the implicit reward remains effective when computed with a different (e.g., smaller, older, or publicly available) reference model.

Evidence in the paper: The paper uses the official reference models for both base models (mistral-7b-sft-beta for Zephyr, the initial SFT model for Llama3) and stores the previous iteration's policy as the reference for each subsequent round. The computational cost is not quantified — the paper reports using 8 Nvidia A100 GPUs but does not specify GPU memory usage, training time per iteration, or the memory overhead of maintaining two models. The inference cost of the implicit reward (two forward passes per response, for K=16 responses per prompt across ~10k prompts) is not separately reported.

Mitigation status: Unacknowledged. The paper presents DICE as a "zero-cost, zero-dependency" method (Section 2 framing), but the dependency on the reference model is a real constraint. The paper does not discuss scenarios where the reference model is unavailable, nor does it ablate whether the implicit reward computed with a different reference (e.g., a smaller model, or a publicly available base model rather than the exact SFT checkpoint) would still be effective.

The Reuse of the Same Offline Preference Dataset for Both Initial DPO and DICE Bootstrapping Raises Questions About the Source of Gains and Data Contamination

The constraint: DICE uses a random subset of approximately 10k preference pairs from UltraFeedback as the offline dataset D_offline (Section 4.1). These same preference pairs — or pairs drawn from the same underlying dataset — were used to train the initial base model (Zephyr-7B-beta and Llama-3-8B-DPO were both DPO-trained on UltraFeedback). This means DICE never introduces genuinely new human preference information; the self-generated data is labeled by the implicit reward, which was itself trained on UltraFeedback, and the experience replay mixes in the same UltraFeedback pairs again. The paper's framing acknowledges this explicitly:

"We randomly sample a subset of around 10k preference pairs from UltraFeedback as the offline dataset D_offline for our fine-tuning experiments. Our experiments aim to show how much the language model can improve from a DPO-tuned model and a subset of the preference dataset that was used to conduct the initial DPO training."

The consequence: The 8%+ LC win rate improvement comes entirely from reusing and rearranging information already present in the initial DPO training data — not from accessing new human preferences. This is simultaneously a strength (no new annotation needed) and a limitation. It means DICE cannot teach the model new preference distinctions that were missing from the original training data. If the original DPO training left certain preference distinctions unresolved or certain types of responses incorrectly ranked, DICE may amplify rather than correct those errors, because the implicit reward inherits the original DPO training's blind spots. The paper does not characterize what types of improvements DICE produces — are the gains concentrated on prompts where the base model was already somewhat capable but inconsistent, or does DICE genuinely teach new capabilities? Without this characterization, a practitioner cannot predict whether DICE will help on their specific distribution of prompts or whether it will primarily make an already-good model more consistent on prompts similar to the training distribution.

Evidence in the paper: The paper provides no analysis of where the gains come from — no breakdown by prompt difficulty, prompt category, or base model confidence. The UltraFeedback dataset is not stratified or categorized in the paper's experiments, so there is no way to know whether DICE's improvements are uniform or concentrated. The experience replay experiment (Figure 3) shows that γ=0 (no offline data in the mix) underperforms γ=0.5, which suggests the offline data contributes something beyond what self-generation alone provides — but what that "something" is (correction of specific errors? stabilization of specific capability dimensions?) is unmeasured.

Mitigation status: Unacknowledged. The paper treats the reuse of UltraFeedback as a feature (no new data needed), which is valid, but does not discuss the implications for the nature of the improvements or the risk of amplifying existing biases in the training data.

7. Implications and Future Directions

How This Work Changes the Landscape

DICE introduces a conceptual shift in how the field thinks about the resources available for language model alignment: it demonstrates that a DPO-trained model is not merely a product of alignment but also a tool for further alignment, because its implicit reward function—a mathematical byproduct of DPO training that had been noted but never operationally exploited for self-improvement—is a sufficiently reliable preference signal to drive multiple rounds of continued improvement without any external feedback. This is not a paradigm shift on the scale of DPO itself (which eliminated the need for a separately trained reward model), nor is it an incremental refinement of an existing self-alignment recipe. It is best characterized as a reframing of existing resources: the paper identifies a capability that already exists in every DPO-trained model and shows how to put it to work.

The magnitude of the shift lies in what it makes unnecessary. Before DICE, a practitioner who had aligned a model via DPO and wanted further improvement faced a set of costly options: commission new human preference annotations, train a separate reward model on existing data and deploy it for iterative DPO, or prompt a larger external model as a judge. Each of these introduced dependencies—on annotators, on additional model artifacts, on external APIs. DICE shows that these dependencies are optional. The model's own implicit reward, computed as two forward passes through models the practitioner already possesses, can serve as the preference signal for two additional rounds of DPO, yielding gains of 8%+ LC win rate on AlpacaEval 2 (Table 1: Zephyr +8.02, Llama3 +9.35). The field's default assumption—that self-alignment requires either new data or an external judge—is now falsified for the DPO setting.

The paper also reconciles a tension that had been accumulating in the preference-tuning literature. On one hand, offline DPO works well for a single round but degrades catastrophically when repeated on the same data (Table 1: Zephyr Offline DPO Iter 2 collapses to 1.89% LC win rate). On the other hand, iterative DPO with on-policy sampling improves performance, but existing methods for labeling on-policy data required external reward models. The tension was: "iterative improvement requires on-policy data, but on-policy data requires external infrastructure." DICE resolves this by showing that the DPO implicit reward is an on-policy labeling mechanism that requires no external infrastructure—it is defined with respect to the current policy and its immediate predecessor, making it inherently on-policy. The theoretical analysis in Appendix A formalizes why offline data becomes stale (suboptimal responses in high-likelihood regions of the current policy may never appear in the training set and therefore never get penalized), and the empirical results in Table 1 provide the clean demonstration: simply regenerating responses from the current policy and relabeling them with the implicit reward prevents the collapse that offline-only training suffers.

This reframing redirects research attention toward several questions that were previously less salient:

  • Verifier robustness for self-alignment becomes a first-class problem. The implicit reward is not a perfect proxy for human preferences, and the paper's experience replay results (Figure 3: intermediate γ outperforms γ=0) show that relying on it exclusively leads to suboptimal performance. This suggests a research program around improving the implicit reward's reliability—through better DPO training, ensemble methods, or calibration techniques—specifically for the self-alignment use case, distinct from the general reward modeling literature.

  • Dataset construction as an optimization problem, not just a collection process. The paper's length-regularized reward shaping (Section 3.1) treats dataset construction as an optimization problem with a directly interpretable objective (minimize average absolute length difference between winners and losers). This decouples dataset debiasing from model training, enabling fast black-box optimization of the debiasing parameter α* without any model training. This approach could generalize: other known biases in preference data (e.g., position bias, sycophancy, style preferences) might be addressable through analogous dataset-level optimization objectives rather than loss-level regularizers.

  • The ceiling on iterative self-improvement is a measurable empirical phenomenon, not a theoretical inevitability. The paper documents that DICE stops improving after two iterations and degrades at three (Table 6). This is not presented as a fundamental limit—the paper hypothesizes that a fixed prompt set across iterations may be responsible—but it establishes that self-alignment loops have finite productive lifespans, and understanding what controls that lifespan becomes a tractable empirical question.

The paper also makes certain research directions less attractive, or at least shifts the burden of proof. Methods that require training a separate reward model for iterative DPO now face a stronger baseline: practitioners can reasonably ask whether the additional complexity, training cost, and maintenance burden of a separate reward model is justified given that the DPO implicit reward already provides a competitive signal (Table 5: 0.698 alignment rate with GPT-4o, exceeding both the internally trained reward model at 0.624 and the 9×-larger-data external model at 0.656, on its own generated distribution). Similarly, self-rewarding language model approaches that fine-tune the model to act as an explicit judge (Yuan et al., 2024) must now contend with the evidence that prompted self-judgment (LLM-as-a-Judge) substantially underperforms DPO implicit rewards (Table 1: DICE Iter 2 beats LLM-as-a-Judge Iter 2 by 6.57 LC points on Zephyr and 5.75 on Llama3), while requiring additional supervised fine-tuning on an evaluation dataset that DICE avoids entirely.

Follow-Up Research This Work Enables

Characterizing where DICE improves alignment and what capabilities it preserves or degrades. The paper reports aggregate LC win rate improvements but provides no breakdown by prompt category, difficulty, or capability dimension. A natural follow-up would annotate a subset of AlpacaEval 2 prompts by category (e.g., factual accuracy, reasoning, creativity, safety, instruction-following precision) and measure whether DICE's gains are uniform or concentrated. Concretely: does DICE primarily improve consistency on prompts where the base model already sometimes succeeds, or does it enable success on prompts where the base model almost always fails? Does it improve factual accuracy at the cost of verbosity or vice versa? Are safety-related capabilities preserved (a critical question since the experience replay mechanism mixes in offline data that may contain safety preferences, but the paper provides no safety evaluation)? This characterization would directly inform practitioners about when DICE is worth deploying and would reveal whether the method introduces capability tradeoffs that aggregate metrics conceal.

Testing whether prompt-set rotation extends DICE beyond two productive iterations. The paper documents collapse at Iter 3 (Table 6) and hypothesizes that reusing the same fixed prompt set across iterations may cause overfitting. A direct test would run DICE identically to the paper's setup but draw a different random 10k-prompt subset from UltraFeedback in each iteration (keeping the offline preference data fixed). If prompt-set rotation enables productive Iter 3 and beyond, the fixed-prompt hypothesis is supported and the method's scalability improves substantially. If Iter 3 still degrades despite fresh prompts, the limitation is deeper—perhaps the implicit reward's noise accumulates across iterations independent of prompt overfitting, or the DPO training dynamics themselves saturate. A negative result here would be equally informative, redirecting attention toward implicit reward calibration rather than prompt-set engineering.

Developing a cheap proxy for the optimal experience replay ratio γ without full evaluation sweeps. The paper shows that optimal γ varies substantially by base model (0.5 for Zephyr, 0.1 for Llama3) and attributes this to base model quality: stronger models need less offline anchoring (Appendix F). A practical follow-up would test whether a simple statistic—such as the base model's LC win rate, its average implicit reward on self-generated responses, or the alignment rate between its implicit reward and a held-out validation set—predicts the optimal γ. Concretely: run DICE with multiple base models spanning a range of initial capabilities (e.g., smaller DPO-tuned models at 1B, 3B, 7B, 13B scales), sweep γ for each, and regress γ* against candidate predictor statistics. A reliable predictor would eliminate the need for expensive γ sweeps in deployment and would validate (or refute) the paper's hypothesis that base model quality drives the optimal replay ratio.

Applying DICE's dataset-level debiasing framework to other known preference dataset biases. The paper's core methodological move—defining a directly interpretable dataset-level optimization objective and solving it without model training—could generalize beyond length bias. Position bias (annotators prefer the first response shown), sycophancy bias (models produce responses that agree with the user's stated views), and style bias (preference for confident or authoritative tone over accurate content) are all known preference dataset pathologies. For each, the follow-up question is: can you define a scalar statistic of the constructed dataset that captures the bias (analogous to average absolute length difference in Eq. (6)), and can you introduce a reward-shaping term that makes that statistic optimizable? For position bias, the statistic might be the correlation between the order of presentation during generation and the likelihood of being selected as winner. For sycophancy bias, the statistic might measure agreement between the user's stated position and the winning response's position. Successfully applying the framework to even one additional bias would establish dataset-level debiasing as a general method rather than a one-off length fix.

Combining DICE with online DPO algorithms that interleave generation and training. DICE operates in a batch-iterative mode: generate all responses, construct the full dataset, then train for one epoch. Recent work on online DPO (Guo et al., 2024) interleaves generation and training at a finer granularity, potentially enabling the model to learn from its own outputs more efficiently. A natural follow-up would replace DICE's batch dataset construction with an online loop: after each training step, generate a small batch of responses from the current (partially updated) policy, score them with the implicit reward, construct preference pairs, and immediately train on them. This would test whether the benefits of on-policy sampling can be realized at finer temporal granularity, potentially enabling more rounds of improvement before degradation sets in. The comparison would also reveal whether DICE's success comes primarily from on-policy data (in which case online DPO should match or exceed it) or also from the specific dataset construction choices (extreme pair selection, length debiasing), which would be harder to replicate in an online setting.

Stress-testing DICE on non-instruction-following domains and measuring capability preservation along critical dimensions. The paper evaluates exclusively on instruction-following benchmarks (AlpacaEval 2, Arena-Hard) using models trained on UltraFeedback (general instruction-following preferences). A comprehensive stress-test would: (1) apply DICE to a base model DPO-tuned for a different domain—safety (e.g., on Anthropic's harmlessness data), code generation, or mathematical reasoning—and evaluate on domain-appropriate benchmarks; (2) for each domain, measure not only the primary domain metric but also hold-out dimensions: does self-alignment on safety degrade instruction-following? Does self-alignment on instruction-following degrade safety (a critical question completely unaddressed in the paper)? (3) test whether the α* optimization (which finds a zero-mean length difference) is appropriate for domains where response length genuinely correlates with quality (e.g., chain-of-thought reasoning where more steps improve accuracy). A negative result on domain transfer—if DICE fails to improve or actively degrades performance outside instruction-following—would bound the method's generality and direct attention toward domain-specific adaptations of the reward shaping and experience replay components.

Practical Applications and Downstream Use Cases

Cost-efficient improvement of deployed DPO-tuned models without new annotation. An organization that has already aligned a 7-8B model via DPO on a proprietary preference dataset (e.g., a customer support chatbot fine-tuned on company-specific interaction preferences) can run DICE for two additional rounds using only the existing model artifacts and a subset of the original preference data, gaining a meaningful alignment improvement. The paper's numbers provide a concrete estimate of the benefit: starting from a Zephyr-level base model at ~13% LC win rate, two DICE iterations yield +8 LC points (to ~21%), a 62% relative improvement in alignment quality, without a single new human label, API call to an external judge, or separately trained reward model. The computational cost—generating 16 responses per prompt for ~10k prompts, computing implicit rewards via two forward passes per response, and running two rounds of DPO training at 300 steps each on 8 A100 GPUs—is modest relative to the initial DPO training cost. This use case is directly supported by the paper's experimental setup (Section 4.1: "a subset of the preference dataset that was used to conduct the initial DPO training").

Bootstrapping stronger open-source aligned models from existing public checkpoints. The open-source community has produced numerous DPO-tuned models (Zephyr, Starling, Tulu, various Llama-DPO variants) using public preference datasets like UltraFeedback. DICE provides a recipe for taking any such model and producing a meaningfully stronger variant without requiring access to the original training pipeline or additional preference data. The paper's leaderboard result (Table 2: DICE-Llama3 8B at 27.55% LC win rate, outperforming the official Llama 3 8B Instruct at 22.92% and surpassing Gemini Pro at 24.38%) demonstrates that this produces models competitive with much larger or more expensively trained alternatives. For organizations that lack the resources to conduct large-scale RLHF from scratch but can afford inference-time generation and a few rounds of fine-tuning, DICE offers a path to state-of-the-art alignment quality from publicly available base checkpoints. The key practical consideration is that the implicit reward computation requires the reference model used in the original DPO training—for public checkpoints where this reference model is also released (as with Zephyr), the pipeline is self-contained; for checkpoints where it is not, alternative reference models would need to be tested.

Improving the data efficiency of self-improving fine-tuning pipelines. Several research directions (SPIN, self-rewarding LMs, SPIN) construct iterative self-improvement loops where a model generates its own training data and is subsequently fine-tuned on it. DICE's experience replay technique—mixing a fraction of the original high-quality human-labeled data into each self-improvement round—is a drop-in addition to any such pipeline. The finding that optimal γ varies by base model quality (0.5 for Zephyr, 0.1 for stronger Llama3; Figures 3 and 5) provides a practical starting point for tuning this ratio. For stronger base models, minimal replay may suffice (γ ~ 0.1); for weaker models, more anchoring is needed (γ ~ 0.5). The paper's evidence that γ=0 (pure self-generated data) underperforms intermediate values for both backbones suggests that completely discarding original human labels in iterative self-improvement pipelines leaves alignment quality on the table—a concrete, actionable lesson for practitioners building self-improving systems.

Rapid iterative alignment in low-resource or privacy-constrained settings. Organizations operating under data privacy constraints that prevent sending user queries to external APIs (for LLM-as-a-Judge labeling) or training external reward models on sensitive data can apply DICE entirely on-premises. The method requires only the DPO-tuned model, its reference model, and the original preference dataset—all artifacts that already exist in the organization's infrastructure. No external API calls, no new annotation, no data leaving the premises. The paper demonstrates this in a simulated low-resource setting by using only a 10k subset of UltraFeedback (1/6 of the full dataset), which is intentionally constrained to show "how much the language model can improve from a DPO-tuned model and a subset of the preference dataset" (Section 4.1). Organizations with even smaller preference datasets could apply DICE, though the optimal γ and the productive number of iterations might shift with dataset size—an open practical question that the paper does not address.