Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis

ArXiv: 2409.20059

🎯 Pitch

Aligning an LLM-based translation model on a single neural quality metric can sharply degrade performance on standard lexical measures—chrF dropped 1.8 points in one setting—revealing a severe metric-instability trap that supervised fine-tuning avoids. The paper shows this instability is not inherent to alignment itself but stems from using translations from multiple external systems when building preference data, and that relying solely on the base model’s own candidates both matches multi-system gains and preserves metric consistency.

1. Executive Summary

This paper studies the effects of preference-based alignment on machine translation quality, conducting extensive experiments with the ALMA-13B-LoRA model on the WMT’22 and WMT’23 test sets using the FLORES-200-based preference dataset. The analysis focuses on Contrastive Preference Optimization (CPO) — a technique that combines a preference loss term with a supervised likelihood term — and compares it against Supervised Fine-Tuning (SFT) on high-quality data under both multi-system (candidates from the base model, GPT-4, and gold references) and mono-system (candidates from only the base model) preference data generation settings. While CPO consistently outperforms SFT on the alignment metric (e.g., improving xCOMET-QE scores by ~2.1 points over the base model in the multi-system setting), it introduces instability across downstream evaluation metrics, with neural-lexical metric consistency degrading sharply — for example, out-of-English chrF dropping from 49.49 to 47.69 when aligning on xCOMET-QE — an adverse effect that SFT does not produce. The paper establishes that using only the base model's own translations for preference data generation achieves performance comparable to relying on multiple external systems while substantially mitigating metric-side inconsistencies, demonstrating that alignment sensitivity is a function of candidate system choice rather than an inherent limitation of preference optimization itself.

2. Context and Motivation

The Core Problem: We Don't Know What Preference-Based Alignment Actually Does to Translation Quality

The central question this paper tackles is deceptively straightforward: when we use preference-based alignment techniques to optimize a translation model against a quality metric, what exactly happens to the model's behavior across different evaluation criteria? This matters because alignment methods represent a fundamental shift in how quality information is injected into translation systems — they optimize model weights directly on preferences induced by quality estimators, rather than selecting among already-generated candidates at inference time. While this approach has produced impressive leaderboard results, the community has lacked systematic understanding of its side effects, failure modes, and sensitivity to experimental design choices.

This gap is significant for several reasons the paper identifies across its introduction and background sections:

Deployment reliability: Translation systems deployed in production are evaluated on multiple fronts — accuracy (neural metrics like COMET or xCOMET-QE), fluency, and lexical fidelity (chrF, BLEU). If aligning on one metric inadvertently degrades performance on others, practitioners may ship systems that look excellent according to their optimization target but produce surprisingly poor outputs on dimensions they stopped measuring. This is not hypothetical — the paper provides concrete evidence that CPO-based alignment on neural metrics can cause substantial drops in lexical scores (Section 4.1, Table 1), meaning a model optimized for semantic adequacy might silently regress on surface-level faithfulness.
Resource allocation decisions: Building preference datasets requires candidate translations from multiple systems. The dominant paradigm, exemplified by Xu et al. (2024a), uses three distinct sources: the base model being aligned, a strong external system like GPT-4, and human-written gold references. Access to high-quality external systems may be expensive or impossible in many practical scenarios (low-resource languages, proprietary domains, cost-constrained environments). If alignment could work effectively using only the base model's own outputs — without external system access — the barrier to entry for preference-based fine-tuning would drop dramatically. The mono-system experiments in Section 5 directly test whether this is possible.
Scientific understanding of alignment dynamics: Preference optimization methods like DPO (Rafailov et al., 2024) and CPO (Xu et al., 2024a) have been largely validated through aggregate benchmark scores — "does the average COMET score go up?" But this black-box evaluation conceals what is actually happening: Is the model genuinely improving its translation capability, or is it learning to exploit quirks of the alignment metric? Are the observed gains robust to the choice of candidate systems, or do they depend on specific properties of the preference data? The paper's systematic ablation experiments — removing individual candidate systems (Section 4.2), fixing the chosen system to a single source (Section 4.3), and varying the quality gap between chosen and rejected pairs (Section 5.2) — are designed to open this black box and characterize the dynamics of preference-based alignment for the first time.

How Quality-Informed Translation Has Evolved

To understand why this paper's investigation is necessary, we need to trace how the field has progressively integrated quality signals into translation systems. The background section (Section 2) organizes this evolution into three generations of approaches.

First generation: lexical metrics as post-hoc evaluation. For decades, MT systems were evaluated using surface-level metrics like BLEU (Papineni et al., 2002), chrF (Popović, 2015), and METEOR (Banerjee and Lavie, 2005). These metrics count n-gram overlaps between system outputs and human-written references. They are computationally cheap and language-independent, but they suffer from a well-documented problem: poor correlation with human judgments. Mathur et al. (2020), Kocmi et al. (2021), and Freitag et al. (2022b) have all demonstrated that lexical metrics frequently disagree with human evaluators, particularly when translations are semantically correct but lexically divergent from the reference (a common scenario when translating between linguistically distant languages or when multiple valid translations exist). This means a system optimized for BLEU is not necessarily optimized for human preference — a fundamental limitation that motivated the move to neural metrics.

Second generation: neural metrics as evaluation and inference-time guidance. The introduction of learned metrics like BLEURT (Sellam et al., 2020), COMET (Rei et al., 2020, 2022a), and Metric-X (Juraska et al., 2023) represented a step change. These models are trained on human judgment data — typically source-translation-reference triplets annotated with quality scores — and learn to predict human preferences directly rather than counting surface matches. They achieve much higher correlation with human judgments and have become the de facto standard for MT evaluation (Freitag et al., 2022b; Kocmi et al., 2024). They also come in two variants: reference-based metrics that require a gold reference translation as input, and reference-free metrics (like xCOMET-QE and CometKiwi) that only need the source sentence and the generated translation — a critical practical advantage since gold references are unavailable at deployment time.

Once neural metrics became reliable enough to trust as quality signals, researchers began using them not just for evaluation but to actively improve translation output through quality-informed decoding. The core idea: sample multiple candidate translations from the model, score each with a neural metric, and select the best-scoring one. Two prominent instantiations are:

Minimum Bayes Risk (MBR) decoding (Kumar and Byrne, 2002, 2004; Eikema and Aziz, 2020; Fernandes et al., 2022; Freitag et al., 2022a): Sample $N$ candidate translations, compute the expected utility (quality score) of each candidate relative to all others under the metric, and select the candidate with the highest expected utility. This is the reference-based setting, since MBR requires scoring pairs of candidates against references.
N-best reranking (Fernandes et al., 2022; Freitag et al., 2022a): Generate $N$ candidates, score each with a reference-free metric, and select the highest-scoring one. This is simpler and works at deployment time without references.

These decoding strategies consistently outperform standard likelihood-based decoding (beam search, greedy decoding) because they directly optimize the metric of interest rather than the model's own probability distribution — which may not align with translation quality (Eikema and Aziz, 2020; Koehn and Knowles, 2017; Ott et al., 2018). However, they have a fundamental limitation: the quality information is only injected at inference time. The model's weights remain unchanged. This means every time you want a high-quality translation, you must generate and score multiple candidates — a computationally expensive process that multiplies inference cost by $N$ . For production systems serving millions of requests, this is often prohibitive.

Third generation: quality-informed fine-tuning. With the rise of decoder-only LLMs applied to MT (Zhu et al., 2023; Jiao et al., 2023; Hendy et al., 2023; Kocmi et al., 2023; Xu et al., 2023, 2024a; Alves et al., 2024), researchers have sought to distill quality information into model weights during training, so that the model produces high-quality translations with simple greedy decoding at inference time. This shifts the computational burden from inference (where it is paid on every request) to training (where it is paid once). Several approaches have emerged:

SFT on high-quality data (Alves et al., 2024): Score translation pairs from parallel corpora with a neural metric, filter to keep only high-scoring pairs, and fine-tune the model on this filtered data. This is straightforward but limited — it only teaches the model what good translations look like, without any signal about what bad translations look like and why they are bad.
Distillation from MBR (Finkelstein et al., 2024): Use MBR decoding to select the best candidate for each source sentence, then fine-tune the model to directly generate those MBR-selected translations. This compresses the benefits of expensive MBR decoding into model weights.
Preference-based alignment (Simianer, 2018; Rafailov et al., 2024; Xu et al., 2024a; Yang et al., 2023; Xu et al., 2024b; Wu et al., 2024): Rather than only learning from positive examples (high-quality translations), these methods learn from comparative preferences between chosen (higher-quality) and rejected (lower-quality) translations. The intuition is that a model that understands why one translation is better than another will learn a more robust quality function than one that only sees good examples.

It is this third approach — preference-based alignment — that the paper investigates.

Where Existing Understanding Falls Short

Despite the growing popularity of preference-based alignment for translation, the paper identifies several critical gaps in our understanding:

Lack of systematic characterization of what alignment actually changes. Prior work on CPO (Xu et al., 2024a) and related methods reported impressive aggregate gains on neural metrics but did not deeply investigate the cross-metric dynamics — what happens to non-optimized metrics when you align on a specific one? This is not just academic curiosity; it is a practical concern because neural and lexical metrics can behave very differently. The Kocmi et al. (2024) recommendation to "employ caution when using the same metric for evaluation that was used during training" exists precisely because metric-specific optimization can lead to metric-specific overfitting. But how severe is this effect? Does it affect all alignment methods equally? Does it depend on the candidate systems used? These questions were unanswered.

No analysis of sensitivity to the preference data construction. Xu et al. (2024a) established a specific recipe for CPO-based translation alignment: use three candidate systems (ALMA-13B-LoRA base model, GPT-4, and gold reference), score with xCOMET-QE, and select the highest and lowest as chosen and rejected pairs. This recipe works, but it leaves open crucial questions: Is GPT-4 necessary, or would the base model's own outputs suffice? Would alignment still work if the gold reference — which by definition has perfect lexical overlap — were removed from the candidate pool? What if the chosen system were always the same source (e.g., always GPT-4)? These are not hypothetical scenarios — in practice, practitioners may not have access to GPT-4 for generating preference data, and gold references are unavailable for new translation tasks. Without understanding the sensitivity of alignment to these choices, the community cannot adapt the approach to resource-constrained settings.

The multi-system assumption is expensive and potentially unnecessary. The standard multi-system paradigm requires maintaining and querying multiple translation systems — some of which (like GPT-4) are proprietary, costly, and may have usage restrictions. Some prior work has suggested that models can be effectively aligned using only their own outputs (Yang et al., 2023; Yuan et al., 2024; Dubey et al., 2024), but this has not been systematically compared against multi-system alignment for translation tasks. If mono-system alignment works comparably, it would dramatically lower the barrier to entry. If it doesn't, we need to understand exactly why external systems matter and which ones are essential.

No investigation of which properties of preference data drive alignment success. Even if we know that alignment works, we don't know why particular preference datasets work better than others. Is it the absolute quality of the chosen translations? The absolute quality of the rejected ones? The gap between them? The paper's grid experiment in Section 5.2 — systematically varying chosen and rejected qualities across nine preference datasets — is designed to answer exactly these questions. This level of controlled analysis had not been done before.

How This Paper Positions Itself

The paper frames itself as a diagnostic investigation, not a method proposal. It does not introduce a new alignment algorithm or claim to beat state-of-the-art translation scores. Instead, it takes the existing CPO approach — which has demonstrated strong results — and systematically probes its behavior, failure modes, and sensitivity to experimental design choices. The goal is to bridge a gap between the empirical success of preference-based alignment (which has been demonstrated) and the mechanistic understanding of how and why it works (which has been lacking).

The paper's positioning is explicitly analytical in several key ways:

Comparative rather than absolute: The paper does not ask "does CPO improve translation quality?" (that has been established by Xu et al., 2024a). It asks "does CPO improve translation quality more than SFT on the same data, and what are the side effects?" This shifts the evaluation from absolute performance to relative effectiveness and robustness.
Systematic ablation over recipe-following: Rather than accepting the multi-system setup as given, the paper systematically removes components (no GPT-4, no reference, no base model) and measures the impact. This reveals which components are essential and which are incidental — information that is directly actionable for practitioners.
Controlled construction of preference data: The mono-system experiments in Section 5 use the base model's own sampled translations, with explicit control over the quality gap between chosen and rejected pairs. This enables the paper to isolate the effect of candidate system diversity (multi-system vs. mono-system) from the effect of preference data quality properties.
Multi-metric evaluation as a diagnostic tool: By reporting not just the alignment metric but also other neural and lexical metrics, the paper uses metric divergence as a signal of alignment behavior. When CPO causes neural metrics to rise while lexical metrics fall (Table 1), this is not just reported as a problem — it is used as evidence that the alignment process is doing something fundamentally different from SFT, and the paper investigates what that something is (Section 4.2: removing references from the candidate pool eliminates the lexical drop, implicating reference inclusion as the cause).

The paper also implicitly positions itself as a practical guide for practitioners. The findings are structured to answer concrete questions someone building a preference-based alignment pipeline would ask: "Do I need GPT-4?" (Section 4.2: removing it hurts, so yes, if you can get it). "Can I just use my own model's outputs?" (Section 5: yes, with comparable performance and better metric consistency, if you control the quality gap properly). "What quality gap should I use?" (Section 5.2: the rejected option should be moderately lower quality, and the chosen option should be as high as possible — the extreme of maximal gap is actually suboptimal). These are actionable takeaways, not just scientific observations.

Finally, the paper connects to a broader concern in the MT evaluation community about metric gaming and overfitting. The finding that neural-metric-aligned CPO models can regress on lexical metrics — while SFT on the same data does not — resonates with warnings from Kocmi et al. (2024) and others about the dangers of optimizing against the same metric used for evaluation. By documenting exactly how this manifests for preference-based methods specifically, the paper provides concrete evidence that this concern is well-founded and identifies specific design choices (candidate system selection, use of rejection term) that either exacerbate or mitigate the problem.

3. Technical Approach

3.1 Reader Orientation

This paper is a diagnostic empirical study, not a new method proposal. The "system" being analyzed is a pipeline for fine-tuning a machine translation LLM using preference-based alignment — specifically, the Contrastive Preference Optimization (CPO) objective applied to the ALMA-13B-LoRA model. The core question it investigates is: what actually happens to a translation model's behavior across different evaluation metrics when you optimize it against preferences induced by a single quality metric, and how sensitive is this process to the way preference data is constructed? The solution takes the form of a controlled experimental framework that systematically varies (a) the candidate translation systems used to build preference pairs, (b) the learning objective (CPO vs. vanilla SFT on chosen translations), (c) the alignment metric (xCOMET-QE, CometKiwi, or chrF), and (d) the quality properties of chosen and rejected translations, while measuring downstream performance on multiple neural and lexical metrics to reveal alignment dynamics that aggregate scores conceal.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework has five major components organized into a training-evaluation pipeline:

Base Model (ALMA-13B-LoRA) — a decoder-only LLM fine-tuned for translation, serving as both the starting point for alignment and the source of candidate translations in the mono-system setting. It uses LoRA adapters with rank 16 and is the same architecture used by Xu et al. (2024a).
Preference Data Constructor — a configurable pipeline that takes source sentences, generates candidate translations from one or more systems (GPT-4, base model, gold references, or only the base model with top-p sampling), scores them with a chosen alignment metric (xCOMET-QE, CometKiwi, or chrF), and selects chosen-rejected pairs according to the experimental configuration (multi-system: pick highest/lowest among available systems; mono-system: select candidates at specific offset positions relative to the base model's greedy output).
Alignment Trainer — applies either SFT (negative log-likelihood on chosen translations only) or CPO (combining an SFT term with a preference-based sigmoid loss on the log-ratio of chosen to rejected probabilities) to LoRA-fine-tune the base model for one epoch with learning rate $10^{-4}$ , inverse square root decay, batch size 128, and CPO parameter $\beta = 0.1$ .
Greedy Decoder — generates translations at inference time using greedy decoding via the vLLM library, producing deterministic outputs for all evaluation.
Multi-Metric Evaluator — scores generated translations on WMT'22 and WMT'23 test sets using three evaluation metrics (xCOMET-QE, CometKiwi, chrF) plus additional metrics (Metric-X, BLEU) in the appendix, reporting results per language pair and aggregated into into-English (xx-en) and out-of-English (en-xx) categories.

Information flows as follows: source sentences enter the preference data constructor → candidate translations are generated and scored → chosen-rejected pairs are selected → the alignment trainer fine-tunes the base model → the fine-tuned model generates translations via greedy decoding → the multi-metric evaluator scores outputs on all metrics.

3.3 Roadmap for the Deep Dive

First, the CPO loss function (Section 3.2, "Learning objective"), since it is the core mechanism distinguishing preference-based alignment from standard SFT. Understanding its mathematical form — how it combines a preference term with a likelihood term — is prerequisite to interpreting every experimental result.
Second, the preference data construction pipeline (Section 3.1), covering both the multi-system approach (candidates from base model, GPT-4, and gold reference) and the mono-system approach (candidates from the base model alone, with controlled quality offsets). This is the experimental knobs that the paper systematically varies.
Third, the alignment metrics (xCOMET-QE, CometKiwi, chrF), since the choice of which metric induces preferences determines which quality signal the model learns from. Understanding what these metrics measure is essential to interpreting the cross-metric dynamics observed in the results.
Fourth, the training configuration (Section 3.2), including hyperparameters, the LoRA setup, and the SFT baseline — establishing the controlled comparison that isolates the effect of the CPO objective.
Fifth, the evaluation framework (Section 3.3), covering the test datasets, inference setup, and evaluation metrics. This is where the paper's multi-metric diagnostic approach is instrumented, and understanding the evaluation design is critical to appreciating the paper's claims about metric consistency and instability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a controlled experimental analysis paper whose core idea is that preference-based alignment, specifically CPO, exhibits complex dynamics that are invisible in aggregate metric scores — including sensitivity to candidate system choice, adverse cross-metric effects, and dependence on the quality properties of preference pairs — and that using only the base model's own translations for preference data generation can achieve comparable alignment gains while substantially improving metric consistency.

The CPO Loss Function

The Contrastive Preference Optimization (CPO) objective, introduced by Xu et al. (2024a), is the learning signal that drives preference-based alignment. It is a composite loss that combines two terms: a preference term that encourages the model to increase the relative likelihood of chosen translations over rejected ones, and a supervised term that directly increases the absolute likelihood of chosen translations. The empirical loss over $N$ training samples is:

$L_{\text{CPO}} = -\frac{1}{N} \sum_{i=1}^{N} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta (y_i^c \mid x_i)}{\pi_\theta (y_i^r \mid x_i)} \right) \right] + L_{\text{SFT}}$

where $L_{\text{SFT}} = -\frac{1}{N} \sum_{i=1}^{N} \left[ \log \pi_\theta (y_i^c \mid x_i) \right]$ is the standard negative log-likelihood loss computed only on the chosen translations $y_i^c$ , $\pi_\theta$ is the model being fine-tuned (parameterized by LoRA adapters $\theta$ ), $x_i$ is the source sentence, $y_i^c$ is the chosen (higher-quality) translation for source $x_i$ , $y_i^r$ is the rejected (lower-quality) translation, $\sigma$ is the sigmoid function, and $\beta$ is a scalar hyperparameter set to $0.1$ .

What this equation computes: for each training sample, the model computes the log-probability it assigns to the chosen translation, $\log\pi_\theta(y_i^c \mid x_i)$ , and the log-probability it assigns to the rejected translation, $\log\pi_\theta(y_i^r \mid x_i)$ . It takes the difference, scales it by $\beta$ , passes the result through the sigmoid function $\sigma(u) = 1 / (1 + e^{-u})$ , and takes the negative log. The sigmoid maps the scaled log-ratio to a value in $(0, 1)$ , and the negative log penalizes the model when this value is low — that is, when the model assigns roughly equal or higher probability to the rejected translation compared to the chosen one. The SFT term $L_{\text{SFT}}$ simultaneously pushes the model to assign high probability to the chosen translation in absolute terms, not just relative to the rejected one. The total loss $L_{\text{CPO}}$ is the sum of these two terms, averaged over the batch.

Why this form: the CPO loss is a modification of the Direct Preference Optimization (DPO) loss from Rafailov et al. (2024). The key difference is the removal of the reference policy term. In standard DPO, the loss includes a term $\beta \log \frac{\pi_\theta(y_i^c \mid x_i)}{\pi_{\text{ref}}(y_i^c \mid x_i)}$ that anchors the optimized policy to a frozen reference model, preventing the policy from diverging too far from its starting point. CPO drops this reference term entirely, which has two consequences: (1) it improves training efficiency by eliminating the need to maintain and query a reference model during training, reducing memory requirements and computation; (2) it incorporates the SFT term $L_{\text{SFT}}$ directly into the objective, which provides an absolute likelihood signal that the original DPO loss lacks. The authors of CPO (Xu et al., 2024a) argue that this combination — relative preference signal from the sigmoid term plus absolute quality signal from the SFT term — yields stronger alignment than either DPO or SFT alone. The paper under analysis uses this exact objective and investigates its empirical properties, not its mathematical derivation.

The $\beta$ parameter controls the strength of the preference signal relative to the SFT signal. When $\beta$ is large, the sigmoid term heavily penalizes cases where the chosen translation is not strongly preferred over the rejected one, pushing the model to maximize the probability gap. When $\beta$ is small, the preference term is weak and the loss approximates pure SFT. The value $\beta = 0.1$ is inherited from the original DPO paper (Rafailov et al., 2024) and appears to work well empirically, though the paper under analysis does not sweep this hyperparameter — it uses the fixed value and investigates other axes of variation (candidate systems, quality gaps, alignment metrics).

Preference Data Construction

The preference dataset $\mathcal{D}_{\text{pref}}$ consists of triples $(x_i, y_i^r, y_i^c)$ where $x_i$ is a source sentence, $y_i^c$ is the chosen translation (higher quality according to a metric $m$ ), and $y_i^r$ is the rejected translation (lower quality). The paper studies two fundamentally different approaches to constructing these triples: multi-system and mono-system.

Source dataset. All preference data is built on the FLORES-200-based dataset from Xu et al. (2024a). This dataset contains over 20,000 translation pairs spanning six languages — English (en), Czech (cs), German (de), Icelandic (is), Russian (ru), and Chinese (zh) — covering ten language directions: five into-English (cs→en, de→en, is→en, ru→en, zh→en) and five out-of-English (en→cs, en→de, en→is, en→ru, en→zh). The language distribution is not uniform: English is heavily represented as either source or target in all pairs, with each of the five non-English languages appearing in approximately 9.8% of the into-English pairs and 9.8–11.1% of the out-of-English pairs (detailed in Appendix B.2). The dataset provides the source sentences $x_i$ for which candidate translations will be generated and scored.

Multi-System Preference Data

Candidate generation. In the multi-system setting — which replicates the approach of Xu et al. (2024a) — candidate translations are obtained from three distinct systems for each source sentence:

Base: ALMA-13B-LoRA, the model being aligned. This provides the baseline translation quality against which improvements are measured.
GPT-4: OpenAI's large proprietary model, accessed via API. GPT-4 translations represent a strong external quality signal — Xu et al. (2024a) show that GPT-4 achieves higher xCOMET-QE scores than ALMA-13B-LoRA on these language pairs, making it the highest-quality system in the candidate pool on average.
Ref: The human-written gold reference translation from the FLORES-200 dataset. By definition, the reference has perfect lexical overlap with itself (chrF = 100 when scored against the reference) and high neural metric scores, though — importantly — it may not always be the highest-scoring option under neural metrics, since neural metrics can sometimes prefer translations that differ from the reference if they are judged more adequate or fluent.

Formally, for each sample $i$ , the candidate pool is $\mathcal{Y}_i^{\text{multi}} = \{y_i^{\text{Ref}}, y_i^{\text{Base}}, y_i^{\text{GPT-4}}\}$ .

Pair selection. Each of the three candidate translations is scored using the alignment metric $m$ — which can be xCOMET-QE, CometKiwi, or chrF (see next subsection for metric details). The chosen translation $y_i^c$ is the candidate with the highest score under $m$ , and the rejected translation $y_i^r$ is the candidate with the lowest score:

$y_i^c = \arg \max_{y \in \mathcal{Y}_i^{\text{multi}}} m(y) \quad \text{and} \quad y_i^r = \arg \min_{y \in \mathcal{Y}_i^{\text{multi}}} m(y)$

What this produces: a dataset where, for each source sentence, the model sees the best and worst available translations and learns to prefer the former over the latter. The quality gap between chosen and rejected — and which systems contribute the chosen and rejected translations — is determined entirely by the metric $m$ and the relative quality of the three candidate systems. Notably, the rejected translation is not necessarily a bad translation in absolute terms; it is simply the worst among the three available options, which could still be quite good if all three systems produce high-quality outputs.

Why this setup: using multiple systems with different quality profiles (base model, strong proprietary model, and human reference) ensures a meaningful quality gap in the preference pairs. If only the base model were used, the highest and lowest candidates might be too similar for the model to learn a clear preference signal. The inclusion of GPT-4 and references guarantees that the chosen translation is substantially better than the base model's output (on average), and that the rejected translation is substantially worse — providing a strong training signal. The cost is that this approach requires access to external systems that may be unavailable or expensive in practice.

Ablation experiments. The paper systematically investigates which candidate systems matter by creating variants of the multi-system dataset where one system is removed:

No Base: candidates are only GPT-4 and Reference. The base model's own translations are absent from the preference pairs entirely.
No Ref: candidates are only Base and GPT-4. Gold references are excluded.
No GPT-4: candidates are only Base and Reference. The strong external model is excluded.

For each ablated candidate pool, the same selection rule applies: chosen = highest score, rejected = lowest score among the remaining systems.

Fixed-chosen experiments. Going further, the paper creates datasets where a specific system is imposed as the chosen translation regardless of its relative score. Specifically, three datasets are created where:

Chosen = Base: the base model's translation is always the chosen option. If any remaining system has a lower score, that becomes the rejected; otherwise, the sample is discarded.
Chosen = Ref: the gold reference is always the chosen option, with analogous selection for the rejected.
Chosen = GPT-4: GPT-4's translation is always chosen.

Why these experiments: fixing the chosen system tests whether CPO can learn a meaningful preference signal when the "correct" answer is always from a single source. If CPO works well when GPT-4 is always chosen (learning to imitate GPT-4), but fails when the base model is always chosen (learning to imitate itself, which SFT already does), this reveals that the preference signal's value comes from having a diverse range of quality, not just from the contrastive objective itself. The results in Section 4.3 confirm this: CPO fails to beat SFT when the chosen system is fixed to a single source, revealing a lack of robustness compared to SFT.

Mono-System Preference Data

Candidate generation. In the mono-system setting, all candidate translations are generated by the base model itself — no external systems are used. For each source sentence $x_i$ , the base model generates $K = 50$ candidate translations using top-p sampling with $p = 0.6$ and temperature $\tau = 0.9$ . These parameters are the default settings from the ALMA paper (Xu et al., 2023, 2024a) and produce diverse but controlled outputs: top-p sampling restricts generation to the smallest set of tokens whose cumulative probability exceeds 0.6, and temperature 0.9 introduces moderate randomness while staying close to the model's most likely outputs.

Scoring and ranking. Each of the 50 candidates is scored with the alignment metric $m$ (xCOMET-QE, unless otherwise specified). The candidates are sorted in increasing order of quality:

$\mathcal{Y}_i^{\text{mono}} = \{y_i^1, \dots, y_i^K\} \quad \text{where} \quad y_i^1 \preceq \cdots \preceq y_i^K$

Here, $\preceq$ denotes "is of lower or equal quality than" according to metric $m$ . The base model's own greedy-decoded translation $y_i^{\text{Base}}$ is scored and its rank $b_i$ in the sorted list is identified. By construction, since the sorted list is increasing in quality:

$y_i^1 \preceq \cdots \preceq y_i^{b_i-1} \preceq y_i^{\text{Base}} \preceq y_i^{b_i} \preceq \cdots \preceq y_i^K$

The base translation sits somewhere in the middle of the quality distribution. Candidates to its left are worse; candidates to its right are better.

Pair selection with quality offsets. This is the key design choice that gives the mono-system approach its flexibility. Rather than always picking the highest and lowest candidates (which would produce extreme quality gaps), the paper introduces two offset parameters $o_r, o_c \in \mathbb{N}$ that control how far from the base translation the rejected and chosen options are selected:

$y_i^r = y_i^{\max(1, \, b_i - o_r)} \quad \text{(the candidate $o_r$ positions below the base translation)}$ $y_i^c = y_i^{\min(K, \, b_i + o_c)} \quad \text{(the candidate $o_c$ positions above the base translation)}$

What this produces: explicit control over the quality gap between chosen and rejected translations. A small offset $o_r$ means the rejected translation is only slightly worse than the base model's output; a large offset means it is much worse. Similarly, a small $o_c$ means the chosen is only slightly better; a large $o_c$ means it is substantially better. The constraint that $y_i^r \preceq y_i^{\text{Base}} \preceq y_i^c$ ensures the base model's own greedy output serves as a natural "anchor" — the model always sees a rejected option that is worse than what it currently produces and a chosen option that is better.

Multi-system replica dataset. To enable a fair comparison between the multi-system and mono-system approaches, the paper first constructs a mono-system dataset designed to approximately replicate the quality properties of the multi-system preference data. Specifically, the offset parameters are chosen so that the average xCOMET-QE score of rejected translations in the mono-system dataset is roughly 87.8 (matching the multi-system's 87.9) and the average chosen score is roughly 97.3 (matching the multi-system's 97.2). These statistics are reported in Appendix B.1, Table 11. This replica dataset is used for the head-to-head comparison in Table 4.

Quality grid datasets for optimization analysis. For the experiment in Section 5.2, the paper constructs nine additional mono-system preference datasets by crossing three levels of chosen quality (Low, Mid, High) with three levels of rejected quality (Low, Mid, High). The levels are defined as:

Rejected Low: average xCOMET-QE ≈ 75.4 (substantially below base model quality)
Rejected Mid: average xCOMET-QE ≈ 84.5 (moderately below base model quality)
Rejected High: average xCOMET-QE ≈ 92.2 (slightly below base model quality)
Chosen Low: average xCOMET-QE ≈ 93.6 (slightly above base model quality)
Chosen Mid: average xCOMET-QE ≈ 95.8 (moderately above base model quality)
Chosen High: average xCOMET-QE ≈ 98.2 (substantially above base model quality)

This grid covers nine combinations (e.g., Chosen=High / Rejected=Mid, Chosen=Low / Rejected=High, etc.) and enables the paper to map out how downstream performance depends on both chosen and rejected quality independently — something that is impossible in the multi-system setting where quality levels are determined by the available external systems.

Why the mono-system approach matters: generating candidates solely from the base model eliminates the need for external systems (GPT-4, human references) while providing fine-grained control over the quality properties of preference pairs. This control is the critical advantage — the paper can systematically vary the quality gap and measure the effect, deriving actionable recommendations (Section 5.2: rejected quality should be moderate, not extreme; chosen quality should be as high as possible; the gap should not be maximally wide). Moreover, because all candidates come from the same model distribution, the mono-system approach limits the adverse cross-metric effects observed in the multi-system setting (Section 5.1, Table 4), which the paper hypothesizes is because candidates from the same distribution share similar structural properties, reducing the likelihood that alignment on one metric (e.g., neural adequacy) will catastrophically disrupt another (e.g., lexical overlap).

Alignment Metrics

The paper uses three distinct metrics to induce preferences in the training data. The choice of alignment metric $m$ determines what quality signal the model learns to optimize — and, critically, whether that signal aligns with or conflicts with other evaluation criteria.

xCOMET-QE (Guerreiro et al., 2023): A reference-free neural metric that scores a translation given only the source sentence and the generated translation. It is based on the XLM-RoBERTa architecture, fine-tuned on human judgment data with additional training on fine-grained error span annotations. The "QE" suffix stands for Quality Estimation, indicating it does not require a reference translation. The model produces a continuous score typically in the range 0–100 (though not strictly bounded), where higher scores indicate better translations judged along dimensions of adequacy, fluency, and meaning preservation. xCOMET-QE is the primary alignment metric in most experiments because it represents the state-of-the-art in reference-free evaluation and — crucially for practical deployment — can be used at inference time without access to gold references.

CometKiwi (Rei et al., 2022b, 2023): Another reference-free neural metric, specifically the WMT'23 version (CometKiwi-XXL). Like xCOMET-QE, it requires only source and translation. It is trained on quality estimation data from WMT shared tasks and uses a similar XLM-RoBERTa backbone. While both xCOMET-QE and CometKiwi are reference-free neural metrics, they differ in their training data, architectures, and the specific human judgments they were calibrated against. The paper uses CometKiwi as an alternative alignment metric to test whether findings generalize across neural metrics or are specific to xCOMET-QE.

chrF (Popović, 2015): A reference-based lexical metric that computes the character n-gram F-score between the generated translation and a gold reference. Specifically, it counts matching character n-grams (typically n=1 to 6) between the hypothesis and reference, computing precision (fraction of hypothesis n-grams that appear in the reference) and recall (fraction of reference n-grams that appear in the hypothesis), then combines them as an F-score. chrF correlates better with human judgments than word-level metrics like BLEU, especially for morphologically rich languages, because it captures sub-word similarities (shared stems, partial cognates) that word-level overlap misses. However, it fundamentally measures surface overlap, not semantic adequacy — a translation can have perfect chrF by exactly matching the reference, even if a different wording would be more natural or adequate.

Why these three metrics: the paper deliberately includes both neural and lexical alignment metrics to investigate cross-metric dynamics. Neural metrics (xCOMET-QE, CometKiwi) capture semantic quality — does the translation preserve meaning and read fluently? Lexical metrics (chrF) capture surface fidelity — does the translation use words and character sequences similar to the reference? These dimensions are correlated but not identical: a translation can be semantically excellent while using very different wording (low chrF), or can have high lexical overlap while being semantically awkward (high chrF but low neural scores). By aligning on one metric and evaluating on all three, the paper can detect when the alignment process causes the model to optimize for the alignment metric at the expense of other quality dimensions — a form of metric overfitting.

The choice of reference-free neural metrics for most experiments is deliberate: reference-free metrics are usable at deployment time, making the alignment pipeline practical. Reference-based metrics like chrF require gold references, which are unavailable for real-world translation requests — a model aligned on chrF would be learning to imitate the reference's surface form but could not be evaluated on chrF in production.

Training Configuration

All fine-tuning experiments are performed on ALMA-13B-LoRA, a 13-billion-parameter LLaMA-based model that has been fine-tuned for translation using LoRA (Low-Rank Adaptation) adapters of rank 16. The paper replicates the exact training hyperparameters from Xu et al. (2024a) to ensure comparability and to isolate the effect of experimental variables (preference data construction, alignment metric) from training dynamics.

Base model. ALMA-13B-LoRA is a decoder-only LLM based on the LLaMA architecture, fine-tuned for multilingual translation following the ALMA recipe (Xu et al., 2023, 2024a). It uses LoRA adapters — lightweight trainable matrices inserted into the model's attention and feedforward layers — rather than full model fine-tuning, which makes training computationally feasible for a 13B model on academic hardware. The LoRA rank is 16, meaning each adapter consists of two low-rank matrices $A \in \mathbb{R}^{d \times 16}$ and $B \in \mathbb{R}^{16 \times d}$ whose product $AB$ is added to the original weight matrix. Only the adapter parameters are updated during fine-tuning; the base model weights remain frozen. This dramatically reduces the number of trainable parameters and memory requirements compared to full fine-tuning.

Optimization hyperparameters. Training uses the AdamW optimizer with the following settings, quoted directly from Section 3.2: "LoRA fine-tuned with rank 16 for one epoch, starting with a learning rate of $10^{-4}$ , using inverse square root decay and a batch size of 128. The $\beta$ parameter of the CPO objective function is set equal to $0.1$ ."

Breaking these down:

One epoch: the model sees each preference pair exactly once during training. This is relatively short — the preference dataset contains over 20,000 samples, so the model takes roughly $\lceil 20000 / 128 \rceil \approx 157$ optimizer steps per epoch. Training for only one epoch reduces the risk of overfitting to the preference data distribution.
Learning rate $10^{-4}$ : the initial learning rate for the AdamW optimizer. This is a typical value for LoRA fine-tuning of large language models — high enough to make meaningful updates in one epoch, but not so high as to destabilize training.
Inverse square root decay: the learning rate decreases proportionally to $1 / \sqrt{t}$ where $t$ is the step number. This schedule reduces the learning rate gradually over training, with most of the learning happening early and fine-grained adjustments happening later. Specifically, the learning rate at step $t$ is $\eta_t = \eta_0 / \sqrt{t}$ for $t \geq 1$ .
Batch size 128: each optimizer step processes 128 preference pairs (each consisting of a source, chosen translation, and rejected translation). Given the average sequence length of translation data, this batch size fits within the memory constraints of typical GPU setups for 13B LoRA models.
$\beta = 0.1$ : the CPO-specific hyperparameter controlling the strength of the preference term relative to the SFT term, as explained in the CPO loss function subsection above.

Code implementation. All models are "trained using the code implementation provided by Xu et al. (2024a)" (Section 3.2), meaning the paper uses the same training scripts, data loading, and evaluation pipelines as the original CPO paper. This eliminates implementation-level confounds — any differences in results between this paper and Xu et al. (2024a) are due to experimental design choices (candidate systems, metrics, evaluation data), not software differences.

SFT baseline. For every CPO experiment, the paper trains a corresponding SFT model on the identical preference data, using only the chosen translations. Formally, the SFT model minimizes:

$L_{\text{SFT}} = -\frac{1}{N} \sum_{i=1}^{N} \left[ \log \pi_\theta (y_i^c \mid x_i) \right]$

This is the standard supervised fine-tuning objective: maximize the likelihood of the high-quality (chosen) translations. The SFT baseline is critical because it isolates the contribution of the preference (rejected) signal. If CPO outperforms SFT, the preference term is providing additional value beyond simply training on good translations. If CPO underperforms SFT, the rejected signal is actively harmful. If they perform similarly, the preference term adds no benefit — and the complexity of constructing preference pairs may not be justified.

Why this training setup: by replicating Xu et al. (2024a)'s exact hyperparameters and code, the paper ensures its experiments are directly comparable to the results that established CPO as an effective approach. This means any new findings about sensitivity, metric inconsistency, or mono-system effectiveness cannot be attributed to differences in learning rate, batch size, or training duration — they are genuine properties of how CPO interacts with preference data construction choices.

Evaluation Framework

Inference setup. All generations at inference time are produced using greedy decoding — at each step, the model selects the single token with the highest probability, with no sampling, no beam search, and no reranking. This is explicitly justified in Section 3.3: "following other works on LLM-based translation (Alves et al., 2024; Briakou et al., 2024), all generations at inference time are produced using greedy decoding, as it provides maximum computational efficiency while preserving high output quality."

The practical implication is that any improvements in translation quality come entirely from the fine-tuned model weights, not from test-time computation. This contrasts with decoding-based quality improvement methods (MBR, N-best reranking) that invest computation at inference time. Greedy decoding is deterministic — the same model produces the same output for the same input every time — which eliminates variance from sampling and makes the evaluation results directly attributable to the training procedure.

Inference is performed using the vLLM library (Kwon et al., 2023), a high-throughput inference engine that uses PagedAttention for efficient memory management. This is a practical implementation detail that does not affect output quality but enables fast evaluation across the thousands of test sentences.

Evaluation datasets. The paper evaluates on two standard WMT test sets:

WMT'22: 17,471 source-reference pairs spanning the same 10 language pairs as the preference data. Language statistics are provided in Appendix B.2, Figure 3: the distribution varies by language pair, with zh→en being the largest at 11% of the test set and is→en being the smallest at 6%.
WMT'23: a separate test set evaluated in Appendix A, providing an independent validation of the main paper's findings. The WMT'23 dataset has a different language distribution (detailed in Appendix B.2, Figure 3), with en→de and en→ru being particularly well-represented.

Using two test sets from different years tests whether findings generalize across data distributions and time periods. The main paper focuses on WMT'22, with WMT'23 results confirming that all key patterns replicate.

Evaluation metrics. Three primary evaluation metrics are reported for all experiments, identical to the available alignment metrics:

xCOMET-QE: reference-free neural metric (same as alignment option)
CometKiwi: reference-free neural metric (same as alignment option)
chrF: reference-based lexical metric (same as alignment option)

Additionally, Appendix A reports Metric-X (the reference-based version of Metric-X-Large, Juraska et al., 2023 — a neural metric trained on human judgments with access to references) and BLEU (Papineni et al., 2002 — the classic word n-gram overlap metric). These provide further dimensions for assessing cross-metric consistency.

Why multi-metric evaluation is central to the paper's contribution: the entire investigation of "adverse metric effects" and "metric consistency" depends on evaluating aligned models on metrics other than the one they were optimized for. If the paper only reported the alignment metric, it would conclude that CPO uniformly improves translation quality — missing the critical finding that neural-metric-aligned CPO can degrade lexical metrics while SFT does not. The diagnostic value comes from the combination of (a) aligning on one metric, (b) evaluating on that same metric (to confirm alignment worked), and (c) evaluating on other metrics (to detect side effects). This design is what enables the paper to claim that "preference-based alignment may lead to instability across downstream evaluation metrics" (Abstract) — a claim that would be invisible in a single-metric evaluation.

Statistical testing. The paper uses one-tailed paired Student's t-tests at the 5% significance level to assess whether differences between models (e.g., SFT vs. CPO, multi-system vs. mono-system) are statistically significant. The pairing is by test sentence: for each of the ~17,000 test sentences, both models produce a translation, both are scored, and the score difference is computed. The t-test assesses whether the mean difference is significantly different from zero. One-tailed tests are appropriate because the hypotheses are directional — e.g., "CPO improves xCOMET-QE over SFT" (not just "CPO differs from SFT"). Results reported in italic font in all tables denote statistically significant differences.

Metric aggregation. Results are reported separately for into-English (xx-en: cs→en, de→en, is→en, ru→en, zh→en) and out-of-English (en-xx: en→cs, en→de, en→is, en→ru, en→zh) translations. This split is important because translation difficulty and quality patterns often differ by direction — translating into English (a high-resource target language with abundant training data) is typically easier than translating out of English into lower-resource languages. The paper's key finding about adverse neural-lexical effects is particularly pronounced for out-of-English translations (e.g., en-xx chrF drops from 49.49 to 47.69 with CPO on xCOMET-QE, Table 1), suggesting that the alignment dynamics differ by translation direction.

Why this evaluation framework matters for the paper's claims: the combination of greedy decoding, multi-metric evaluation, and statistical testing creates a rigorous experimental environment where observed differences can be confidently attributed to the training procedure. Greedy decoding eliminates sampling variance; multi-metric evaluation reveals cross-metric dynamics; the large test sets (~17,000 sentences) provide sufficient statistical power to detect even modest effects; and the paired t-test accounts for per-sentence variability. This framework justifies the paper's main conclusions — that CPO exhibits sensitivity to candidate systems, that SFT is more robust, and that mono-system alignment is a viable alternative — with statistical rigor rather than anecdotal observation.

Summary of Design Choices and Their Justifications

CPO rather than DPO or other preference methods: CPO's combination of preference signal and SFT term, along with its removal of the reference policy, makes it more efficient than DPO and reportedly more effective (Xu et al., 2024a). Investigating CPO specifically allows the paper to contribute to understanding a method already demonstrated to achieve state-of-the-art translation performance.
Three candidate systems (Base, GPT-4, Reference) for multi-system experiments: replicates Xu et al. (2024a) exactly, enabling direct comparison and establishing a known starting point before systematic ablation.
Reference-free metrics (xCOMET-QE, CometKiwi) as primary alignment targets: these metrics represent the deployment-relevant scenario where gold references are unavailable, making the findings applicable to real-world systems.
Both neural and lexical alignment metrics: enables investigation of whether alignment behavior depends on the type of quality signal (semantic vs. surface-level), which turns out to be critically important — lexical alignment with CPO fails catastrophically (Table 1).
Mono-system quality grid with controlled offsets: the offset mechanism $o_r, o_c$ provides an experimental knob for independently varying chosen and rejected quality that is impossible in the multi-system setting, enabling the fine-grained analysis in Section 5.2.
Greedy decoding for all evaluation: isolates training effects from inference-time computation, ensuring that observed improvements come from model weights rather than test-time search or reranking.
Multi-metric evaluation on large test sets with statistical testing: provides the diagnostic power to detect cross-metric dynamics that would be invisible in single-metric evaluations, and ensures findings are statistically reliable.
One epoch of training with fixed hyperparameters from Xu et al. (2024a): controls for training dynamics as a confound, ensuring that differences between experimental conditions are attributable to preference data construction rather than optimization details.

4. Key Insights and Innovations

Innovation 1: Preference-Based Alignment, Unlike SFT, Introduces Metric Instability — and the Instability Is Not Inherent to the Method, but to the Candidate Systems

The paper's most significant intellectual contribution is not a method but a diagnostic finding that fundamentally reframes how we should evaluate alignment techniques. Prior work on CPO (Xu et al., 2024a) and related preference methods reported aggregate metric improvements and treated them as unqualified progress. The implicit assumption was that if a model's score on the alignment metric improves, the model is "better." This paper challenges that assumption by demonstrating that the same alignment procedure that improves neural metrics can simultaneously degrade lexical metrics, and — critically — that this degradation is not an inevitable consequence of preference optimization but a function of which candidate systems are used to construct the preference data.

This is a conceptual shift from evaluating alignment methods on a single axis (does the target metric go up?) to evaluating them on metric consistency across neural and lexical evaluation dimensions. The evidence is stark and specific. In Table 1, aligning ALMA-13B-LoRA on xCOMET-QE using the standard multi-system setup (Base + GPT-4 + Reference) with CPO improves out-of-English xCOMET-QE by +0.84 points over the base model while simultaneously dropping out-of-English chrF by 1.80 points (from 49.49 to 47.69). SFT on the same data, by contrast, improves both metrics (+0.47 xCOMET-QE, +0.79 chrF). This is not a small tradeoff but a qualitatively different pattern: CPO with multi-system data creates a model that is better according to neural judges but worse according to lexical ones, while SFT creates a model that is better according to both.

What makes this a genuine innovation rather than an obvious observation is the second-order finding that the instability is traceable to specific candidate systems, not to CPO itself. Section 4.2 demonstrates that removing the gold reference from the candidate pool eliminates the en-xx chrF drop under CPO xCOMET-QE alignment (Table 2: "No Ref" CPO achieves 60.59 chrF for xx-en and 50.67 for en-xx, both improvements over the base model). Similarly, Section 5.1 shows that mono-system alignment (using only the base model's own translations) almost completely eliminates the neural-lexical divergence (Table 4: en-xx chrF drops only from 49.49 to 49.02 under CPO, compared to the 47.69 drop in the multi-system setting). The adverse metric effect is thus not a property of preference optimization — it is a property of mixing candidate translations from systems with different surface-level characteristics (specifically, including gold references, which by definition have perfect lexical overlap with themselves), which causes the model to learn spurious correlations between neural quality signals and lexical patterns.

This reframes the narrative around preference alignment from "DPO/CPO causes metric overfitting" — a pessimistic conclusion that would discourage adoption — to "preference alignment with heterogeneous candidate sources can cause metric divergence, and controlling candidate homogeneity mitigates it." This is an actionable diagnostic insight: practitioners can monitor cross-metric consistency as a signal of whether their preference data construction is introducing harmful distributional biases, and they can switch to mono-system data generation if they observe neural-lexical divergence.

The comparison to prior work is what makes this insight sharp. Xu et al. (2024a) reported CPO's success on the alignment metric but did not systematically analyze cross-metric effects. Kocmi et al. (2024) warned about the general danger of "using the same metric for evaluation that was used during training" but did not investigate whether this danger depends on the alignment method, the candidate systems, or the metric type. This paper provides the first controlled evidence that SFT is robust to this problem while CPO is sensitive, and that the sensitivity is controllable through candidate system choice — a nuanced picture that neither the CPO advocates nor the metric-gaming skeptics had articulated.

Innovation 2: The Mono-System Approach as a Controlled Alternative That Reveals What Actually Matters in Preference Data

The paper's second conceptual contribution is the mono-system experimental framework itself — not as a cheaper alternative to multi-system alignment (though it is that), but as a scientific instrument for studying what properties of preference data drive alignment success. Prior work on preference-based translation alignment treated the candidate systems as a fixed recipe: use the base model, a strong external model, and gold references (Xu et al., 2024a). This recipe works, but it conflates multiple variables — the absolute quality of the chosen translation, the absolute quality of the rejected translation, the gap between them, the diversity of system distributions, and the presence of gold references with their unique lexical properties. The mono-system approach disentangles these variables by using only the base model's own top-p-sampled outputs and explicitly controlling the quality offsets of chosen and rejected translations.

This is methodologically significant because it converts an observational setup into an interventional one. In the multi-system setup, the qualities of chosen and rejected translations are whatever the available external systems happen to produce — you observe correlations between candidate compositions and downstream performance, but you cannot independently vary chosen quality and rejected quality because they are jointly determined by which systems are in the pool. In the mono-system setup with explicit offset parameters $o_r$ and $o_c$ , you can set rejected quality to Low, Mid, or High independently of chosen quality, producing a 3×3 grid of preference datasets that covers the space of possible quality configurations. This is exactly what Section 5.2 does, and it produces findings that were inaccessible to prior work.

The grid experiment (Figure 1) reveals several non-obvious properties of effective preference data:

A maximum quality gap is not optimal. The naive assumption — if a small quality gap is good for learning, a larger gap should be better — is falsified. The optimal configuration (point H in Figure 1, achieving 91.1 xCOMET-QE on WMT'22) uses rejected translations averaging around 90% of the base model's quality and chosen translations around 105%. Pushing rejected quality much lower (point C, rejected at 75.4, chosen at 98.2) produces worse downstream performance (90.9) despite a much larger training gap. This suggests that extremely low-quality rejected examples provide a noisy or uninformative contrastive signal — the model learns to distinguish terrible translations from excellent ones, but this does not transfer to distinguishing good translations from slightly better ones, which is the actual deployment scenario.
Rejected quality matters independently of chosen quality. Moving from rejected=Low to rejected=Mid at any fixed chosen level consistently improves downstream performance (e.g., points A→D→G all improve as rejected quality increases from 75.4 to 84.5 to 92.2). This is not obvious a priori — one might expect that as long as the rejected option is worse than the base model's output, its absolute quality should not matter. The finding implies that the contrastive signal is most informative when the rejected translation is close enough to the chosen one that the model must learn subtle quality distinctions rather than coarse ones.
Chosen quality should be as high as possible, but with diminishing returns. At any fixed rejected level, increasing chosen quality from Low (93.6) to Mid (95.8) to High (98.2) monotonically improves performance, but the gap between Mid and High is smaller than between Low and Mid. This suggests a saturating benefit: once the chosen translation is sufficiently good, further improvements in the training signal provide marginal gains.

These findings constitute a practical theory of preference data construction for translation that was absent from prior work. Before this paper, the only guidance was "use multiple systems including GPT-4." After this paper, the guidance is more nuanced and more actionable: "ensure the rejected translation is moderately worse than your model's baseline output (~10% lower quality), make the chosen translation as good as possible, and do not artificially maximize the gap — a moderate gap with a high-quality chosen option is better than an extreme gap." This represents a genuine advance in understanding, not just an incremental empirical result.

The mono-system approach also functions as a robustness diagnostic. By comparing mono-system CPO to multi-system CPO under identical alignment metrics (Table 4), the paper isolates the effect of candidate system diversity. The fact that mono-system CPO achieves performance comparable to multi-system CPO on neural metrics while nearly eliminating lexical regression demonstrates that the benefits of preference optimization do not depend on access to external systems — a finding with substantial practical implications for resource-constrained settings.

Innovation 3: CPO Is Brittle to Preference Setting Design in Ways That SFT Is Not — A Cautionary Finding with Methodological Implications

The paper's third conceptual contribution is a negative result with substantial positive implications for how alignment research should be conducted. Across multiple experiments, the paper demonstrates that CPO exhibits a brittle sensitivity to the design of preference data that SFT does not share. This is not a minor implementation detail — it is a fundamental property of the CPO objective that has implications for reproducibility, robustness, and the interpretation of published benchmark results.

The evidence for this brittleness is specific and multi-faceted:

Fixing the chosen system destroys CPO's advantage over SFT. Section 4.3 (Table 3) shows that when the chosen translation is always the base model's output, CPO fails to outperform the trivial baseline of SFT on base model translations — a setting where SFT should do nothing (since it is fine-tuning on the model's own greedy output). Specifically, CPO with Chosen=Base achieves 87.94 xx-en xCOMET-QE versus SFT's 88.17 — a small degradation rather than an improvement. When the chosen system is always the reference, CPO performance collapses across all metrics (xx-en xCOMET-QE drops to 81.95 vs. the base model's 87.80, and en-xx chrF plummets to 39.81 vs. 49.49). SFT on the same fixed-chosen data does not exhibit these catastrophic failures. This demonstrates that CPO's effectiveness depends on having a diverse pool of candidate systems where the chosen option varies in its source — a requirement that is not obvious from the CPO paper and that practitioners following the recipe without understanding this sensitivity could easily violate.
Lexical alignment with CPO fails catastrophically. Table 1 shows that aligning on chrF — where the chosen translation is almost always the gold reference (since references have perfect chrF by definition) — causes CPO performance to degrade across the board. xx-en xCOMET-QE drops from 87.80 to 78.51, CometKiwi drops from 80.86 to 75.62, and chrF itself drops from 58.53 to 45.32. The alignment metric that CPO is explicitly trained to optimize decreases — a complete failure of the optimization objective. SFT on the same chrF-based chosen translations shows modest stagnation or slight decline but nothing approaching this collapse. This reveals that CPO's preference term requires the alignment metric to produce a meaningful quality ranking that correlates with actual translation quality across the candidate distribution. When the alignment metric is chrF, the ranking is dominated by surface overlap with the reference, and maximizing the probability gap between high-chrF and low-chrF translations does not correspond to improving general translation quality — it corresponds to overfitting to reference-matching surface patterns.
Removing specific candidate systems changes CPO behavior dramatically, but SFT is relatively insensitive. Section 4.2 (Table 2) demonstrates that removing GPT-4 from the candidate pool significantly degrades CPO's alignment metric performance (en-xx xCOMET-QE drops from 92.75 to 92.22) but barely affects SFT (92.38 to 92.33). Removing the reference eliminates the neural-lexical divergence for CPO but has minimal effect on SFT's already-stable cross-metric behavior. The systematic pattern across all these ablations is that SFT is robust to preference data design choices; CPO is not.

Why is this conceptually significant? It challenges the implicit assumption in the preference optimization literature that the objective function is the primary determinant of alignment success, with the data playing a supporting role. The paper demonstrates that CPO's effectiveness is jointly determined by the objective and the data construction in ways that are not separable — a finding with substantial implications for how alignment research should be conducted. Specifically:

Reproducibility concerns: if CPO's performance is highly sensitive to which systems are used for candidate generation, then reported benchmark results may not replicate when practitioners use different external systems or when external system quality changes over time (e.g., GPT-4 is continuously updated). SFT's robustness makes it a more reproducible baseline.
Interpretation of published gains: the strong CPO results reported by Xu et al. (2024a) depend not just on the CPO objective being effective, but on the specific combination of ALMA-13B-LoRA + GPT-4 + gold references as candidate systems. The paper's ablation experiments suggest that removing GPT-4 would reduce those gains, and removing references would change the cross-metric profile. This means CPO's state-of-the-art status is contingent on a specific experimental configuration that may not generalize.
Guidance for practitioners: the paper effectively provides a decision tree for whether to use CPO or SFT. If you have access to diverse, high-quality external systems and care primarily about the alignment metric, CPO can provide additional gains. If you need robustness across metrics, if external systems are unavailable, or if the alignment metric has properties that make preference ranking unreliable (as with chrF), SFT is the safer choice. This is a practical insight that directly contradicts the implicit message of the CPO paper (which presented CPO as a drop-in improvement over SFT in all cases).

This finding is not a criticism of CPO per se — it is a characterization of its operating conditions. The paper does not argue that CPO is a bad method; it argues that CPO's benefits are conditional on experimental design choices that were not previously recognized as critical. This is a fundamentally constructive contribution that helps the field use preference alignment more effectively by understanding when and why it works, rather than blindly applying it as a universal recipe.

Innovation 4: A Principled Decomposition of Alignment Sensitivity into Candidate System Effects vs. Objective Function Effects

The paper's fourth conceptual contribution is the explicit experimental decomposition of alignment behavior into two independent axes: the learning objective (SFT vs. CPO) and the preference data construction (multi-system vs. mono-system, choice of candidate systems, quality properties of pairs). While this may sound like standard ablation methodology, it is actually a novel analytical framework for the alignment literature, which has historically confounded these axes by (a) comparing different alignment methods on different datasets, and (b) treating the preference data as a fixed input rather than a design space to be explored.

The decomposition is not merely organizational — it generates non-trivial insights that neither axis alone could produce:

The interaction between objective and data is where the interesting dynamics live. CPO with multi-system data produces neural-lexical divergence; CPO with mono-system data does not. SFT with either data type does not produce divergence. This is a three-way interaction — the metric instability is a product of the CPO objective AND the presence of heterogeneous candidate systems AND the specific metric being aligned on. None of these factors alone explains the phenomenon. Prior work that studied preference optimization without systematically varying data construction would have observed the divergence (or not) as a property of their specific setup and drawn incomplete conclusions.
The mono-system grid experiment (Section 5.2) converts the data axis from a categorical variable (which systems?) to a continuous one (what quality levels?), enabling dose-response analysis. This is conceptually important because it moves the investigation from "does candidate system X help?" to "how does alignment performance scale with chosen and rejected quality?" The latter question is more general and more scientifically informative. The answer — that there is an optimal region in the (chosen quality, rejected quality, gap) space, and that both too little and too much gap can be harmful — would be impossible to discover in a multi-system setup where quality levels are fixed by the available systems.
The decomposition reveals that SFT and CPO occupy different positions in a robustness-performance tradeoff space. SFT is a low-variance, moderate-bias estimator of the quality signal — it consistently improves both neural and lexical metrics, but the improvements on the alignment metric are smaller than CPO's. CPO is a higher-variance, lower-bias estimator — it can extract more alignment-metric gain from the preference data, but at the risk of catastrophic failure when the data is poorly constructed (fixed chosen system, chrF alignment) and moderate adverse effects (neural-lexical divergence) even with well-constructed data. This tradeoff characterization is actionable: practitioners can choose their method based on their risk tolerance and evaluation requirements, rather than universally preferring one over the other.

What makes this a genuine innovation rather than routine ablation is that the decomposition is the paper's primary contribution, not a sidebar to a method proposal. Most alignment papers introduce a new method and include a few ablations to justify design choices. This paper does the inverse: it takes an existing method and makes the systematic exploration of its design space the entire scientific contribution. The result is a much richer understanding of how preference-based alignment works than would be possible from a single-method paper with a fixed experimental configuration.

This decomposition also provides a template for future alignment research. The paper implicitly argues — through its structure and findings — that alignment methods should be evaluated not just on a single benchmark with a single data construction strategy, but across a matrix of (objective × data configuration × evaluation metric) to characterize their robustness and operating conditions. If this methodological norm were adopted, the field would be less vulnerable to the kind of brittle, configuration-dependent results that the paper exposes in CPO. The paper thus contributes not just findings about CPO specifically, but a research methodology for alignment diagnostics that is its most generalizable contribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation dataset is the WMT'22 test set, consisting of 17,471 source-reference pairs spanning 10 language directions: five into-English (cs→en, de→en, is→en, ru→en, zh→en) and five out-of-English (en→cs, en→de, en→is, en→ru, en→zh). Results on WMT'23 test data are provided in Appendix A for independent validation. All preference data is built on the FLORES-200-based dataset from Xu et al. (2024a), containing over 20,000 translation pairs across the same six languages and ten directions.
Base model. All experiments use ALMA-13B-LoRA (Xu et al., 2023), a 13-billion-parameter decoder-only LLM based on the LLaMA architecture, fine-tuned for multilingual translation using LoRA adapters of rank 16. The paper states this model is chosen to replicate the exact experimental setup of Xu et al. (2024a), ensuring direct comparability of findings about CPO behavior.
Metrics. Three primary evaluation metrics are reported: xCOMET-QE (Guerreiro et al., 2023) — a reference-free neural metric scoring semantic adequacy and fluency; CometKiwi (Rei et al., 2023) — another reference-free neural metric based on quality estimation training; and chrF (Popović, 2015) — a reference-based lexical metric computing character n-gram F-score overlap. Appendix A additionally reports Metric-X (Juraska et al., 2023), a reference-based neural metric, and BLEU (Papineni et al., 2002). Results are aggregated into xx-en (into-English) and en-xx (out-of-English) categories, with per-language-pair breakdowns available in Appendix Tables 9 and 10.
Baselines. The base model (ALMA-13B-LoRA without further fine-tuning) serves as the primary reference point. For every CPO experiment, a corresponding SFT model trained on the identical chosen translations (without the preference term) is the direct comparison baseline — this isolates the contribution of the rejected translation signal. The SFT objective is standard negative log-likelihood on chosen translations only.
Generation budget / compute accounting. All inference-time generation uses greedy decoding via the vLLM library (Kwon et al., 2023), producing exactly one output per source sentence with no sampling, beam search, or reranking. There is no test-time compute budget to account for — the experimental design deliberately shifts all quality optimization to training time through model weight updates, with inference cost held constant across all models. Training budget is one epoch for all models, with batch size 128.
Cross-validation / statistical protocol. Statistical significance is assessed using one-tailed paired Student's t-tests at the 5% significance level, pairing model outputs per test sentence. Results in italic font in all tables indicate statistically significant differences between the compared methods. The paper does not use cross-validation for model selection — each experimental configuration is trained once and evaluated on the fixed WMT'22 and WMT'23 test sets.

Main Quantitative Results

Multi-System Preference Fine-Tuning: CPO vs. SFT Across Alignment Metrics

Table 1 provides the foundational comparison between SFT on preferred translations and CPO in the multi-system setting (Base + GPT-4 + Reference candidates), using three different alignment metrics and evaluating on all three metrics. The results reveal distinctly different dynamics for neural versus lexical alignment:

Neural metric alignment (xCOMET-QE or CometKiwi) improves downstream performance on the alignment metric for both SFT and CPO, but CPO provides larger gains. When aligning on xCOMET-QE, CPO achieves 89.95 xx-en xCOMET-QE (up from the base model's 87.80) compared to SFT's 89.13 — a statistically significant advantage of +0.82 points over SFT. For en-xx, CPO reaches 92.75 (up from 91.91) versus SFT's 92.38. The same pattern holds when aligning on CometKiwi: CPO reaches 89.82 xx-en versus SFT's 89.26, and 92.19 en-xx versus 92.44 (a non-significant difference in the latter case). Cross-metric improvements are also observed: aligning on xCOMET-QE improves CometKiwi scores (81.89 for CPO vs. 81.49 for SFT on xx-en), and vice versa (82.04 for CPO vs. 81.70 for SFT on xx-en).

However, neural-metric-aligned CPO introduces adverse lexical effects that SFT avoids. For out-of-English translations, CPO alignment on xCOMET-QE drops chrF from the base model's 49.49 to 47.69 — a statistically significant degradation of 1.80 points. SFT on the same data improves chrF to 50.28. The same pattern appears with CometKiwi alignment: CPO drops en-xx chrF to 48.11 versus SFT's 50.49. For into-English translations, lexical effects are neutral or slightly positive but much smaller in magnitude (e.g., xx-en chrF: CPO 59.83 vs. SFT 59.82 with xCOMET-QE alignment).

Lexical metric alignment with CPO fails catastrophically. When aligning on chrF, CPO performance degrades across all metrics: xx-en xCOMET-QE plummets from 87.80 (base) to 78.51, CometKiwi drops from 80.86 to 75.62, and chrF itself falls from 58.53 to 45.32. This is a complete failure — the alignment metric the model was explicitly trained to optimize decreases substantially. SFT on the same chrF-based chosen translations shows more modest effects: xx-en chrF drops to 56.97, xCOMET-QE drops to 87.61, and CometKiwi remains roughly stable at 80.82. For en-xx, the CPO degradation is similarly severe (chrF drops from 49.49 to 42.50), while SFT remains stable (50.30).

The interactive dynamics are critical: for neural alignment, CPO outperforms SFT on neural metrics but underperforms on lexical metrics; for lexical alignment, CPO underperforms SFT across the board. This asymmetry demonstrates that CPO's effectiveness relative to SFT depends fundamentally on the type of quality signal being optimized.

Impact of Candidate Systems on Alignment Behavior

Table 2 systematically ablates the candidate system pool by removing one system at a time — No Base (only GPT-4 + Reference), No Ref (only Base + GPT-4), No GPT-4 (only Base + Reference) — for both neural (xCOMET-QE) and lexical (chrF) alignment, under both SFT and CPO optimization.

Removing systems significantly affects alignment metric performance, with CPO showing greater sensitivity than SFT. For CPO with xCOMET-QE alignment, removing GPT-4 causes the largest drop in alignment metric performance: en-xx xCOMET-QE falls from 92.75 (All systems) to 92.22 (No GPT-4), and xx-en drops from 89.95 to 88.81. Removing the base model has a smaller but statistically significant effect (en-xx: 92.74, xx-en: 89.59). Removing the reference has minimal impact on neural metrics (en-xx: 92.44, xx-en: 89.91). For SFT with xCOMET-QE alignment, the effects are more muted — the largest drop from removing GPT-4 takes en-xx from 92.38 to 92.33 and xx-en from 89.13 to 88.44, differences that are not statistically significant at the 5% level.

The reference is the source of neural-lexical divergence under CPO. When aligning on xCOMET-QE with CPO, the en-xx chrF drop from 49.49 (base) to 47.69 (All systems) is eliminated when references are removed from the candidate pool: No Ref CPO achieves 50.67 en-xx chrF, a statistically significant improvement. The No Base condition (50.48 vs. 48.54 chrF) also shows mitigation but less complete recovery. No GPT-4 still exhibits the chrF drop (46.82). This pinpoints the gold reference — with its perfect lexical overlap — as the specific candidate causing the adverse lexical effect when combined with CPO's preference term.

Lexical alignment failure is resolved by removing references. When aligning on chrF, the catastrophic CPO degradation documented in Table 1 (en-xx chrF falling to 42.50) is completely reversed by removing references from the candidate pool: No Ref CPO achieves 51.11 en-xx chrF, exceeding the base model's 49.49 and SFT's 50.30. xx-en chrF similarly recovers from 45.32 (All systems) to 60.63 (No Ref). This confirms that the chrF alignment failure is specifically caused by including references in the candidate pool — where the chosen translation is almost always the reference by definition — creating a pathological preference signal that CPO's objective cannot handle.

Removing systems can improve performance over the full candidate pool. Notably, for SFT with chrF alignment, No Ref achieves 60.17 xx-en chrF versus 56.97 with All systems — removing references actually improves performance on the alignment metric itself. Similarly, No Ref CPO with chrF alignment achieves 60.63 xx-en chrF, dramatically better than the 45.32 with All systems. This demonstrates that including references in the candidate pool is not just unnecessary but actively harmful for lexical alignment, even for SFT.

Fixed-Chosen System Experiments: Testing CPO Robustness

Table 3 reports results when the chosen translation is fixed to a single system — either always Base, always Ref, or always GPT-4 — with the rejected translation selected from the remaining systems where a lower-scoring candidate exists. Alignment is on xCOMET-QE.

CPO fails to beat SFT when the chosen system is fixed, regardless of which system is chosen. When Chosen = Base, CPO achieves 87.94 xx-en xCOMET-QE versus SFT's 88.17 — CPO actually performs slightly worse, though the difference is not statistically significant. For en-xx, CPO reaches 91.75 versus SFT's 91.94. When Chosen = GPT-4, CPO achieves 89.69 xx-en versus SFT's 89.81, and 90.50 en-xx versus 91.96. In both cases, CPO's ostensible advantage over SFT (observed in Table 1) disappears.

Fixing the reference as chosen causes catastrophic CPO degradation. When Chosen = Ref, CPO performance collapses: xx-en xCOMET-QE drops from the base model's 87.80 to 81.95 (versus SFT's 88.04), en-xx xCOMET-QE drops from 91.91 to 86.97 (versus SFT's 92.35), and en-xx chrF plummets to 39.81 (versus SFT's 50.12 and the base model's 49.49). This is worse than the base model across all metrics — including the alignment metric itself. SFT with Chosen = Ref does not exhibit this collapse, maintaining stable performance across metrics (xx-en xCOMET-QE: 88.04, en-xx chrF: 50.12).

CPO's brittleness is specific to fixed-chosen settings, not to the presence of references per se. The contrast between "Chosen = Ref" (catastrophic failure) and "All systems with references included" (moderate chrF drop, neural improvement) in Table 1 shows that CPO can handle references when they appear among naturally varying chosen options, but cannot handle them when they are the only positive signal. This implicates the diversity of chosen sources — not just their identity — as critical for CPO's effectiveness.

Mono-System vs. Multi-System Alignment

Table 4 presents the head-to-head comparison between multi-system (Base + GPT-4 + Reference) and mono-system (only base model candidates) preference data, with alignment on xCOMET-QE. The mono-system "replica" dataset is constructed to match the average chosen/rejected quality properties of the multi-system data (chosen average xCOMET-QE: 97.3 vs. 97.2; rejected: 87.8 vs. 87.9, per Appendix Table 11). An additional "optimized" mono-system variant uses the best quality configuration identified in Section 5.2.

Mono-system alignment improves downstream neural metrics over the base model, confirming that external systems are not necessary for effective alignment. Mono-system SFT achieves 88.51 xx-en xCOMET-QE (vs. base 87.80) and 92.17 en-xx (vs. base 91.91). Mono-system CPO achieves 89.35 xx-en and 92.69 en-xx. These are statistically significant improvements over the base model, though smaller than the multi-system gains.

CPO consistently outperforms SFT on neural metrics in the mono-system setting. Mono-system CPO achieves 89.35 xx-en xCOMET-QE versus SFT's 88.51, and 92.69 en-xx versus 92.17 — both statistically significant differences. This replicates the CPO > SFT pattern from the multi-system setting (Table 1) and confirms that the CPO objective's advantage over SFT is not dependent on multi-system data.

Multi-system alignment modestly outperforms mono-system on neural metrics, but at the cost of lexical consistency. Multi-system CPO achieves 89.95 xx-en xCOMET-QE versus mono-system's 89.35 (a statistically significant difference of 0.60 points) and 92.75 en-xx versus 92.69 (not significant). However, the en-xx chrF picture is starkly different: multi-system CPO drops chrF to 47.69 (from base model's 49.49), while mono-system CPO achieves 49.02 — essentially maintaining the base model's lexical performance while improving neural metrics. The difference in en-xx chrF between multi-system and mono-system CPO (47.69 vs. 49.02) is statistically significant.

Optimized mono-system data narrows the neural metric gap while preserving lexical consistency. Mono-system (opt.) CPO — using the best configuration from the quality grid experiment — achieves 89.58 xx-en xCOMET-QE (vs. multi-system's 89.95), 92.87 en-xx (vs. 92.75), and 49.11 en-xx chrF (vs. multi-system's 47.69). The neural metric gap shrinks to 0.37 xx-en and actually reverses for en-xx (+0.12 advantage to mono-system), while the lexical metric remains stable near the base model level. This demonstrates that careful tuning of preference data quality properties can match or approach multi-system performance without sacrificing metric consistency.

Optimizing Preference Data Quality in the Mono-System Setting

Figure 1 and the corresponding analysis in Section 5.2 map how downstream performance (WMT'22 average xCOMET-QE across all language pairs) varies with the average qualities of chosen and rejected translations in the mono-system preference data. Nine datasets are constructed crossing three rejected quality levels (Low: ~75.4, Mid: ~84.5, High: ~92.2) with three chosen quality levels (Low: ~93.6, Mid: ~95.8, High: ~98.2), with the base model's average quality serving as reference (93.09, per Appendix Table 11).

The quality gap between chosen and rejected is not monotonically beneficial. The naive expectation — larger gap, stronger training signal, better performance — is falsified. Point C (Rejected=Low/Chosen=High, the largest gap) achieves 90.9 xCOMET-QE, while point H (Rejected=Mid/Chosen=High) achieves 91.1 — the latter has a smaller gap but better downstream performance. This demonstrates that extremely low-quality rejected examples provide a less informative contrastive signal than moderately low-quality ones.

Rejected quality has an independent and substantial effect on downstream performance. At any fixed chosen quality level, moving rejected quality from Low to Mid consistently improves performance: at Chosen=Mid, Low→Mid increases from 90.8 (point D) to 91.0 (point E); at Chosen=High, Low→Mid increases from 90.9 (point C) to 91.1 (point H). Moving from Mid to High rejected quality shows diminishing or negative returns: at Chosen=High, Mid→High drops from 91.1 (point H) to 90.8 (point I). The optimal rejected quality appears to be around 90% of the base model's quality — high enough to provide a meaningful but not trivial contrast, low enough to clearly distinguish from the chosen option.

Chosen quality should be as high as possible, with diminishing returns at the top. At any fixed rejected level, increasing chosen quality from Low to Mid to High monotonically improves performance, but the gain from Mid to High is smaller than from Low to Mid. For example, at Rejected=Mid: Low (90.4, point G) → Mid (91.0, point E) → High (91.1, point H) — the first step gains 0.6 points, the second only 0.1.

The optimal configuration uses moderate rejected quality and maximally high chosen quality. Point H (Rejected=Mid, Chosen=High, achieving 91.1 average xCOMET-QE) represents the best-performing configuration. This translates to a practical recommendation: rejected translations should average approximately 10% lower quality than the base model's output (rejected ~90% of base quality), while chosen translations should be as high-quality as the base model can produce (~105% of base quality). The optimized mono-system model in Table 4 uses this configuration and achieves performance comparable to the multi-system setting (89.58 vs. 89.95 xx-en xCOMET-QE) while maintaining lexical consistency.

Ablation Studies and Robustness Checks

Alternative alignment metric (CometKiwi) replicates all key patterns: Table 1 shows that when CometKiwi replaces xCOMET-QE as the alignment metric, CPO still outperforms SFT on neural metrics (xx-en: 89.82 vs. 89.26; en-xx: 92.19 vs. 92.44 — the latter a non-significant difference), and the adverse lexical effect persists for en-xx (chrF: 48.11 for CPO vs. 50.49 for SFT). The cross-metric benefit is symmetric: aligning on CometKiwi improves xCOMET-QE scores, and vice versa. This generalization across two independently trained neural metrics strengthens the claim that the observed dynamics are properties of the preference-based alignment framework, not artifacts of a specific metric.

chrF as alignment metric reveals fundamental incompatibility with CPO: Table 1 demonstrates that lexical alignment causes CPO to degrade on its own optimization target (xx-en chrF: 45.32 vs. base 58.53; en-xx chrF: 42.50 vs. base 49.49) and on all other metrics. The No Ref ablation in Table 2 rescues lexical alignment performance (xx-en chrF: 60.63 for CPO, exceeding base 58.53), confirming that the failure mode is specifically tied to reference inclusion, not to lexical metrics per se. This is a critical negative result: it identifies a boundary condition where CPO should categorically not be used.

Candidate system ablation with CometKiwi evaluation: Appendix A, Table 5 replicates the Table 1 patterns on WMT'23 test data, confirming that neural alignment with CPO improves the alignment metric while introducing en-xx chrF drops (44.29 for CPO vs. base 47.31). Table 6 replicates the candidate system ablation findings on WMT'23: removing references eliminates the CPO chrF drop, and removing GPT-4 reduces alignment metric gains. The cross-dataset validation demonstrates that the findings are not specific to the WMT'22 test distribution.

Fixed-chosen experiments replicated on WMT'23: Appendix Table 7 confirms that Chosen = Ref causes catastrophic CPO degradation on WMT'23 (xx-en xCOMET-QE: 84.06 vs. base 88.00; en-xx chrF: 34.64 vs. base 47.31), while SFT remains stable. Chosen = Base and Chosen = GPT-4 show CPO failing to beat SFT, replicating the Table 3 findings.

Mono-system vs. multi-system on WMT'23: Appendix Table 8 replicates Table 4's key finding: mono-system CPO achieves comparable neural metric performance to multi-system CPO (89.33 vs. 89.77 xx-en) while substantially mitigating the en-xx chrF drop (46.65 vs. 44.29). The optimized mono-system variant narrows the neural gap further (89.36 vs. 89.77 xx-en) while maintaining lexical stability (46.48 en-xx chrF).

Additional evaluation metrics (Metric-X, BLEU) confirm the neural-lexical divergence pattern: Appendix Tables 9 and 10 provide comprehensive per-language-pair breakdowns with five evaluation metrics. The BLEU results closely track chrF patterns: multi-system CPO with xCOMET-QE alignment drops en-xx BLEU from 28.28 (base) to 25.63, while mono-system CPO achieves 27.74 — partial but not complete mitigation. Metric-X, a reference-based neural metric, shows smaller but directionally consistent effects: multi-system CPO drops en-xx Metric-X from 1.83 (base) to 1.64, while mono-system CPO achieves 1.69. The convergence of chrF, BLEU, and Metric-X on the same direction of effect — with Mono mitigating but not fully eliminating the degradation — strengthens confidence that the phenomenon is real and not an artifact of a single metric's scoring behavior.

Language-pair variation is substantial but does not change qualitative conclusions: Per-language-pair results in Tables 9 and 10 reveal that the adverse lexical effects under multi-system CPO are most pronounced for certain directions (e.g., en-de BLEU drops from 27.65 to 26.20 with CPO; en-is BLEU drops from 23.49 to 21.29) but appear consistently across all out-of-English pairs. The mono-system mitigation is also consistent across pairs. This breadth of evidence rules out the possibility that the aggregate results are driven by a single outlier language pair.

Critical Assessment

Do the Experiments Support the Claim That "Preference-Based Alignment Globally Outperforms SFT on High-Quality Data in Terms of Maximizing the Alignment Metric"?

The evidence is strong but conditional. For neural alignment metrics (xCOMET-QE, CometKiwi), CPO consistently outperforms SFT on the alignment metric in the multi-system setting — Table 1 shows statistically significant advantages for xx-en xCOMET-QE (+0.82 over SFT) and CometKiwi (+0.56) when each is the alignment target. This pattern replicates on WMT'23 (Table 5: +0.81 xCOMET-QE, +0.55 CometKiwi). However, "globally" overstates the case: CPO fails to outperform SFT on the alignment metric when (a) the chosen system is fixed to a single source (Table 3: Chosen=Base yields CPO 87.94 vs. SFT 88.17; Chosen=GPT-4 yields 89.69 vs. 89.81), (b) the alignment metric is lexical (Table 1: chrF alignment yields CPO 45.32 vs. SFT 56.97 xx-en chrF — CPO dramatically underperforms), or (c) references are included in chrF alignment (Table 2: No Ref rescues CPO to 60.63, but the "All systems" baseline including references fails). The claim holds specifically for neural metrics in settings with diverse, naturally varying chosen systems — a narrower condition than "globally outperforms SFT."

A genuine weakness: the paper does not sweep the β parameter of CPO. All experiments use β = 0.1, inherited from Rafailov et al. (2024). It is possible that different β values would change CPO's relative performance versus SFT, particularly in the fixed-chosen or lexical-alignment settings where CPO currently fails. The paper cannot rule out that CPO's brittleness is a consequence of suboptimal β rather than an inherent property of the preference term.

Do the Experiments Support the Claim That "Preference-Based Alignment Is Highly Sensitive to the Choice of Candidate Systems"?

Unequivocally yes. The systematic ablation in Table 2 demonstrates sensitivity along multiple dimensions: removing GPT-4 significantly degrades CPO's alignment metric performance (en-xx xCOMET-QE: 92.75 → 92.22); removing references eliminates the neural-lexical chrF divergence (en-xx chrF: 47.69 → 50.67); and removing the base model has intermediate effects. The fixed-chosen experiments in Table 3 push this further, showing that CPO's advantage over SFT disappears entirely when the chosen system is fixed to any single source. The sensitivity is not subtle — in some configurations (Chosen=Ref, chrF alignment with references), CPO produces models worse than the unaligned base model across all metrics, a catastrophic failure mode that SFT never exhibits. The mono-system results in Table 4 demonstrate that this sensitivity is controllable — using only base model candidates produces stable, if slightly lower, neural metric gains — but the sensitivity itself is robustly documented across datasets (WMT'22 and WMT'23), metrics, and ablation conditions.

A limitation: the paper only tests three candidate systems (Base, GPT-4, Reference) in the multi-system setting. This is inherited from Xu et al. (2024a) and is a reasonable starting point, but it leaves open questions. Would the sensitivity pattern change with four systems? With a different strong external system (e.g., Claude, Gemini)? With multiple base model variants? The qualitative finding — that CPO is more sensitive than SFT to candidate system composition — likely generalizes, but the specific thresholds (e.g., "removing GPT-4 hurts more than removing the reference") may be specific to the relative quality ordering of these three systems (GPT-4 > Base > Reference on average neural metrics, per Appendix Table 11).

Do the Experiments Support the Claim That "Aligning a Model Using Its Own Translations Achieves Performance Comparable to Employing Multiple External Systems"?

This claim is supported with a small but real performance gap. Table 4 shows that mono-system CPO achieves 89.35 xx-en xCOMET-QE versus multi-system's 89.95 — a 0.60-point gap that is statistically significant. For en-xx, the gap is 92.69 vs. 92.75, which is not statistically significant. The optimized mono-system narrows the xx-en gap to 0.37 (89.58 vs. 89.95) and actually edges ahead on en-xx (92.87 vs. 92.75). "Comparable" is therefore accurate in practical terms — the differences are within 0.5-0.6 xCOMET-QE points on a 100-point scale — but a rigorous reading must note that multi-system retains a small but detectable advantage on into-English translations.

The stronger part of this claim is the tradeoff characterization: mono-system achieves comparable neural metric performance while substantially improving metric consistency. The en-xx chrF comparison (mono-system: 49.02; multi-system: 47.69; base: 49.49) shows mono-system nearly preserving base-level lexical performance while multi-system degrades it. If a practitioner values cross-metric stability, mono-system is arguably preferable despite the small neural metric gap.

An important nuance: the mono-system approach requires generating K=50 candidates per source sentence using top-p sampling, scoring all 50 with xCOMET-QE, and constructing the preference dataset offline. This is computationally comparable to or cheaper than querying GPT-4 for 20,000+ sentences (since the base model is local and sampling is fast), but the computational cost of the mono-system data construction is not quantified or compared to multi-system costs anywhere in the paper. A full cost-benefit analysis would need to account for this.

What Would Strengthen the Paper?

Missing β sweep: The brittleness findings raise the question of whether CPO's sensitivity is inherent to the preference term or an artifact of β = 0.1. A sweep across β ∈ {0.01, 0.05, 0.1, 0.5, 1.0} in the fixed-chosen and chrF-alignment settings would reveal whether a different preference strength mitigates or exacerbates the observed failures. If a higher β (stronger preference signal) makes Chosen=Ref fail even more dramatically, that would strengthen the claim of inherent brittleness. If a lower β (closer to pure SFT) eliminates the failure, that would suggest the problem is β misspecification rather than a fundamental limitation.

Evaluation with human judgments: All claims about "improvement" and "degradation" are mediated entirely through automatic metrics. The paper acknowledges this limitation explicitly ("our evaluation relied on automatic metrics... with the latter closely approximating human judgments but still being unable to fully replace them"). The neural-lexical divergence finding is particularly vulnerable: it is possible that the "adverse lexical effect" (chrF drops) represents genuine quality improvement — the model producing more natural, less reference-bound translations that happen to have lower n-gram overlap. Human evaluation of multi-system CPO vs. mono-system CPO vs. base model outputs would be the gold standard for adjudicating whether the chrF drop is harmful (degraded translation quality) or benign (improved naturalness at the cost of reference alignment).

Expanded model family testing: All experiments use ALMA-13B-LoRA, a single model from a single architecture family (LLaMA-based). The paper's claims about CPO sensitivity could be specific to this model's training data, multilingual capabilities, or LoRA fine-tuning dynamics. Testing with at least one model from a different family (e.g., Mistral-based, Falcon-based, or a encoder-decoder architecture) and at a different scale (e.g., 7B) would establish whether the brittleness findings generalize. The paper acknowledges this in the limitations section.

Adaptive or multi-round alignment: The mono-system grid experiment (Section 5.2) identifies optimal quality configurations post-hoc. A natural extension — not attempted — would be iterative alignment: align the model, use the aligned model to generate new candidates (which will be higher quality), construct new preference data from these improved candidates, and align again. This could potentially close the remaining gap with multi-system alignment while maintaining mono-system's metric consistency benefits.

Inference-time decoding comparison: The paper compares quality-informed fine-tuning (CPO, SFT) against a base model with greedy decoding, but never compares against quality-informed decoding (MBR, N-best reranking) using the base model with equivalent total computation. A head-to-head between CPO with greedy decoding versus base model with MBR decoding at various candidate counts would directly address the question motivating the paper: does preference-based fine-tuning actually achieve "similar objectives to those of quality-aware decoding approaches" (Section 4.1) in terms of final output quality, and at what relative computational cost?

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Amortized

The assumption or constraint. The entire compute-optimal framework assumes the system can estimate each prompt's difficulty before allocating the inference budget, enabling per-prompt strategy selection. The paper's method for doing so — generating 2,048 samples per question and computing the pass@1 rate (oracle) or averaging PRM final-answer scores (predicted) — is extraordinarily expensive. The authors acknowledge this explicitly (Section 3.2): "our experiments do not account for this cost largely for simplicity." At 2,048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations).

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the estimation cost. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. For a system processing thousands of prompts, this overhead could erase or even reverse the claimed efficiency advantages. The severity depends on how often difficulty needs to be re-estimated — if prompts are drawn from a stationary distribution, amortization over many queries might make this viable; if each prompt is novel, the overhead destroys the gains.

What evidence exists in the paper. The 4× efficiency claim is derived from Figures 4 and 8, which plot accuracy against the strategy execution budget alone. The difficulty estimation cost — 2,048 samples per prompt — is stated in Section 3.2 but never appears on any budget axis. The paper treats this as a separate "exploration" phase and explicitly flags it as "a key avenue for future work" (Section 3.2), but no attempt is made to characterize the amortization break-even point (how many queries must share a difficulty estimate before the 4× savings outweigh the upfront cost).

Mitigation status. The paper does not address this beyond acknowledging it and suggesting future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No lightweight difficulty classifier is developed or evaluated. The predicted difficulty bins (using PRM scores instead of ground-truth correctness) reduce the oracle dependency but do not reduce the sample cost — they still require 2,048 generations per prompt. This limitation remains entirely unresolved and is arguably the largest barrier between the paper's findings and practical deployment.

The Scope Is Bounded by a Single Verifier Quality Level

The assumption or constraint. All findings about compute-optimal strategy selection, verifier over-optimization thresholds, and the pretraining-vs-inference tradeoff are conditioned on the specific PRM trained via Monte Carlo rollouts from PaLM 2-S* as described in Section 5.1 and Appendix D. The PRM's quality — its calibration, its vulnerability to exploitation, its difficulty-discrimination ability — determines where beam search over-optimizes, where revisions help vs. hurt, and which difficulty bins benefit from which strategies. The paper does not experiment with PRMs of different quality levels (e.g., trained on more data, with different architectures, or with different label sources).

The consequence. The central finding — that beam search over-optimizes on easy problems and degrades performance at high budgets (Figure 3, right) — may be specific to this PRM's quality. A better-calibrated PRM might push the over-optimization threshold higher, extending the regime where beam search is beneficial. Conversely, a worse PRM might cause over-optimization even at moderate budgets for medium-difficulty problems, collapsing the compute-optimal strategy to pure best-of-N. The difficulty thresholds that define the compute-optimal policy (easy → best-of-N, medium → beam search, hard → neither) are functions of PRM quality, not universal properties of test-time compute scaling.

What evidence exists in the paper. Figure 3 (right) shows clearly that verifier over-optimization is the primary bottleneck — beam search accuracy actually decreases with budget on easy problems. Appendix M provides qualitative examples of degenerate solutions that score highly under the PRM. However, the paper never varies PRM quality to test whether these phenomena are robust or PRM-specific. The PRM-vs-ORM comparison (Appendix F, Figure 14) shows that PRM outperforms ORM, but both are trained on the same base model and same data; neither establishes how results would change with a fundamentally different verifier quality level.

Mitigation status. Not addressed. The paper identifies verifier over-optimization as a central limiting factor (Section 5.3, Section 8) and frames it as a research direction ("improving verifier robustness is the key bottleneck"), but makes no attempt to characterize how verifier quality interacts with the compute-optimal policy. The authors do not sweep PRM training data size, architecture variants, or ensembling approaches to show whether the difficulty-dependent strategy selection is stable across verifier quality levels.

Generalization Is Limited to a Single Model Family and Single Reasoning Benchmark

The assumption or constraint. All experiments use PaLM 2-S* (Codey) on the MATH benchmark (500 test questions). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is asserted rather than tested. MATH consists exclusively of competition-level mathematics problems requiring multi-step symbolic reasoning with unambiguous ground-truth answers — a domain with properties that may not transfer: clear correctness signals (answers are right or wrong), structured solution steps, and a reliance on deductive reasoning rather than factual recall or creative generation.

The consequence. Several aspects of the findings could be model-specific or domain-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns (e.g., more likely to produce near-miss answers versus nonsensical ones), or different step-by-step reasoning style could exhibit different difficulty-dependent scaling curves. The finding that beam search over-optimizes on easy problems might not hold for a model whose output distribution is more concentrated around correct solutions.
The revision model's effectiveness depends on the base model's in-context learning capabilities — specifically, its ability to use previous incorrect answers as informative context for producing improved ones. This ability varies substantially across model families, and a model with weaker in-context revision capability would show different sequential-vs-parallel tradeoffs.
The MATH benchmark's structure (symbolic reasoning, step-by-step solutions, exact-answer verification) is ideal for both PRM training (Monte Carlo rollouts require unambiguous correctness checks) and difficulty estimation (pass@1 has a clear definition). Tasks without clean correctness signals — code generation where correctness is partial, open-ended QA, summarization, dialogue — would require fundamentally different verifier training and difficulty estimation approaches that the paper does not explore.

What evidence exists in the paper. None — the paper provides no replication on other model families, other benchmarks, or non-math reasoning domains. The test set is 500 questions, and the difficulty bins split this into ~100 questions per quintile, with two-fold cross-validation further halving the per-bin sample for strategy selection to ~50 questions. This is a small sample, and the paper does not report confidence intervals on the compute-optimal scaling curves. The authors acknowledge the domain limitation implicitly by focusing their claims on "reasoning problems" and suggesting future work on other domains (Section 8), but no robustness checks across domains are provided.

Mitigation status. The paper acknowledges this as a limitation only in passing (Section 8 suggests "extension to other domains"), without characterizing the specific dimensions along which generalization is uncertain. No experiments on other benchmarks, other model families, or other reasoning types are included. The findings should be understood as specific to PaLM 2-S* on MATH until replication demonstrates otherwise.

Hard Problems Remain Fundamentally Unsolved, Establishing a Hard Capability Ceiling

The assumption or constraint. The paper's approach — whether using search, revisions, or compute-optimal combinations — can only amplify existing model capability. It cannot create capability that was not present in the base model's output distribution. This is not an assumption the paper makes but rather a boundary condition it empirically discovers: for problems where the base model's pass@1 is near zero, no amount of test-time compute helps.

The consequence. Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and the ~14× larger pretrained model maintains a clear advantage. This means that for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution, pretraining remains the only viable path — test-time compute offers essentially zero return on investment.

This is a fundamental limitation for applications where the problem distribution includes a non-trivial fraction of genuinely hard problems. The paper's own difficulty estimation shows that difficulty bin 5 exists (roughly 20% of the MATH test set), and the compute-optimal policy for these problems is essentially "give up and use the base model" — no strategy improves over baseline. For a deployment where 20% of queries are in this regime, the headline 4× efficiency gains on easier problems must be weighed against zero gain on the hardest quintile.

What evidence exists in the paper. The flat bin 5 curves in Figures 3 and 7 are the clearest evidence. Figure 9 shows that even with unlimited test-time compute (scaling to the rightmost x-axis positions), bin 5 accuracy for the smaller model never approaches the larger pretrained model's performance. The FLOPs-matched comparison quantifies this: on hard problems at R ≫ 1, test-time compute shows a −52.9% relative disadvantage compared to pretraining (Figure 1, bottom-right bar chart).

Mitigation status. The paper is transparent about this limitation. Section 7 explicitly states that "test-time compute is powerful when problems are within the base model's reach... but it cannot compensate for fundamental capability gaps that larger pretraining would address." This is a fair characterization, but the paper does not explore whether there are methods to expand the base model's capability frontier at test time — for instance, by retrieving relevant knowledge, using tools, or decomposing problems in ways that make them tractable for the base model even if the original formulation is too hard. The limitation is acknowledged as a fact about the current approach, not proposed as a problem to solve.

Revisions and Search Are Never Combined, Leaving Potential Gains Unexplored

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided search (which improves candidate selection through verifier optimization) and iterative revisions (which improves the proposal distribution by conditioning on previous attempts) — but treats them as independent experimental pipelines. Section 8 explicitly acknowledges this gap: "we did not experiment with PRM tree-search techniques in combination with revisions."

The consequence. The paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths revealed by the difficulty-bin analysis: revisions work best on easy problems where the model's initial output is roughly correct and needs refinement (a local search in answer space), while search works best on medium problems where the model needs to explore qualitatively different solution strategies (a global search). Combining them — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue — could yield gains beyond either method alone, particularly on medium-difficulty problems where both mechanisms show complementary strengths. The compute-optimal policy as characterized in the paper is therefore optimizing over an incomplete strategy space — the true optimal policy might include combined strategies that were never tested.

The parallel-to-sequential ratio experiments (Section 6) and the search algorithm comparisons (Section 5) are presented as independent analyses, and the compute-optimal strategy selects between them per difficulty bin. But a strategy that uses beam search on top of revision model outputs — where the revision model generates high-quality candidates and beam search selects among them — might outperform either pure search or pure revisions at any budget level. The paper cannot rule this out because it was never tested.

What evidence exists in the paper. The difficulty-bin analysis provides indirect evidence that combining approaches could help. Figure 3 (right) shows beam search outperforming best-of-N on medium problems (bins 3–4). Figure 7 (right) shows balanced sequential-parallel ratios outperforming pure sequential or pure parallel on medium problems. If these effects are additive or synergistic, a combined approach might show even larger gains. But no direct experimental evidence exists — no model was trained or evaluated that uses both CPO and quality-informed decoding, or both revisions and PRM-guided search.

Mitigation status. The paper explicitly flags this as future work (Section 8) but makes no attempt to explore even a simple combination (e.g., using revision model outputs as candidates for best-of-N weighted selection, or applying the PRM to score revision chains). The omission is understandable given the paper's scope — it is already a dense empirical study covering search and revisions independently — but it means the reported efficiency gains should be understood as achievable with either search or revisions, not both simultaneously. A fully optimized system combining both might achieve substantially better results, or might reveal interference effects (e.g., revision model outputs might be harder for the PRM to score accurately due to distribution shift — a problem the paper already documents for base-model PRM applied to revision outputs in Appendix J). The interaction between these mechanisms is an open question.

The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate with No Principled Fix

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This creates an asymmetry: the model learns to revise incorrect answers into correct ones, but never learns to recognize when the current answer is already correct and should be preserved. At inference time, when the model's own revision chain produces a correct answer at step t, the model has no training signal for what to do at step t+1 — and often incorrectly "revises" the correct answer into an incorrect one.

The consequence. The paper reports that approximately 38% of correct answers produced during a revision chain get converted back to incorrect ones in the subsequent step (Section 6.1). This is a fundamental inefficiency: the model is simultaneously climbing toward correct answers and then undoing its own progress. The paper mitigates this with answer selection across the chain (majority voting or verifier-based selection, picking the best answer from any step rather than always taking the last revision), but this is an inference-time patch that does not address the underlying problem. It also means that increasing the revision chain length does not monotonically improve output quality — at some point, the probability of reversion balances or exceeds the probability of improvement, creating a performance ceiling.

The reversion problem also complicates the sequential-to-parallel ratio optimization. If the model tends to revert correct answers, long sequential chains waste computation by oscillating between correct and incorrect states. The optimal ratio found in Section 6 (Figure 7) may therefore reflect the reversion rate rather than a fundamental property of how sequential computation helps — a model with a lower reversion rate might benefit from longer pure-sequential chains.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The flattening of pass@1 improvement in the revision trajectory (Figure 6, left) is consistent with reversion limiting gains — accuracy plateaus around 24–25% after step 15–20 despite continuing to 64 steps. The paper also notes that the revision model's training data construction (pairing independently sampled correct and incorrect solutions post-hoc using edit distance, rather than on-policy multi-turn rollouts) "approximated the multi-turn structure" and may contribute to this issue.

Mitigation status. The paper partially mitigates the reversion problem with chain-wide answer selection (majority voting or verifier-based selection), but this treats the symptom rather than the cause. The authors acknowledge the issue but do not explore solutions such as training the model with correct-to-correct examples (teaching it to recognize when no revision is needed), adding an explicit "no change needed" token, or using the PRM to dynamically decide when to stop revising. The ReST $^{EM}$ experiment (Appendix K, Figure 16) further highlights the fragility of revision training — attempting to optimize the revision model with RL-style training caused performance to degrade substantially, suggesting the reversion problem may be difficult to fix with straightforward approaches.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around preference-based alignment for machine translation from a method-centric perspective (does alignment method X beat method Y on benchmark Z?) to a diagnostic perspective (what happens to a model across all evaluation dimensions when you optimize it against a single quality signal, and which experimental design choices control the outcome?). This is a reframing rather than a paradigm shift — the underlying CPO method is unchanged — but the reframing has substantial practical and methodological consequences for how alignment research should be conducted and evaluated.

The primary conceptual shift: metric consistency as a first-class evaluation criterion. Before this work, preference optimization methods like CPO and DPO were evaluated almost exclusively on the alignment metric itself — did COMET or xCOMET-QE go up? The paper demonstrates that this single-metric evaluation conceals critical dynamics: the same CPO procedure that improves neural metrics by ~2 points can simultaneously degrade lexical metrics by ~1.8 points (Table 1, en-xx chrF from 49.49 to 47.69), while SFT on identical data improves both. This finding elevates cross-metric consistency from a secondary concern to a primary diagnostic that any alignment paper should report. If the community adopts this norm — and the paper's explicit connection to Kocmi et al. (2024)'s warnings about metric gaming provides strong motivation — future alignment evaluations will be more informative and less vulnerable to metric-specific overfitting.

Reconciliation of conflicting intuitions about preference optimization. The paper resolves a latent tension in the alignment literature. On one side, CPO advocates (Xu et al., 2024a) showed strong aggregate improvements and presented CPO as a robust drop-in improvement over SFT. On the other side, metric evaluation skeptics (Kocmi et al., 2024) warned that optimizing against the same metric used for evaluation can produce misleading gains. This paper shows that both perspectives are partially correct, and the resolution depends on the candidate systems and alignment metric. CPO genuinely outperforms SFT on neural metrics when using diverse, naturally varying candidate systems (Table 1: +0.82 xCOMET-QE over SFT in multi-system xx-en). But it also genuinely introduces metric divergence when those candidate systems include gold references (Table 2: removing references eliminates the chrF drop). And it genuinely fails when the alignment metric is lexical (Table 1: chrF-aligned CPO degrades on chrF itself). The contradictory prior findings are reconciled by recognizing that CPO's behavior is not a fixed property of the method but an interaction between the objective and the preference data construction — a nuance that neither camp had articulated.

Research directions that become more attractive. The paper implicitly argues that understanding alignment dynamics is as important as developing new alignment algorithms. This pushes the field toward:

Systematic diagnostic frameworks: rather than proposing new loss functions, investigate the operating conditions of existing ones. The paper's decomposition of alignment behavior into (objective × candidate systems × metric type) provides a template.
Data-centric alignment research: the quality grid experiment (Section 5.2, Figure 1) demonstrates that preference data properties — chosen quality, rejected quality, the gap between them — have substantial and non-obvious effects on downstream performance. This opens a research avenue focused on preference data construction as a design space, not a fixed recipe.
Robustness as a primary metric: the finding that SFT is consistently more robust than CPO to preference data design choices (Tables 2, 3, 4) suggests that future alignment methods should be evaluated not just on peak performance but on performance variance across data construction strategies. A method that achieves +3 points on one configuration but crashes on another is less practically useful than one that achieves +1.5 points reliably across all configurations.

Research directions that become less attractive. The paper's negative results — particularly the catastrophic CPO failure under chrF alignment (Table 1) and fixed-chosen settings (Table 3) — suggest that blindly applying preference optimization to any quality metric is unlikely to work. Lexical metrics, which measure surface overlap rather than semantic quality, produce preference rankings that do not correlate with general translation quality, and CPO's preference term amplifies this mismatch rather than correcting it. This implies that alignment research should focus on metrics with demonstrated correlation to human judgments (neural metrics, human preference data) rather than expecting preference optimization to salvage weak quality signals. The ReST $^{EM}$ revision model degradation (a separate finding from the main paper but referenced in the broader literature) similarly suggests that iterative self-improvement with preference signals can backfire in ways that are not yet well-understood.

The mono-system finding as a practical democratization. The demonstration that mono-system alignment achieves performance comparable to multi-system alignment (Table 4: 89.35 vs. 89.95 xx-en xCOMET-QE) while eliminating the need for external systems like GPT-4 lowers the barrier to entry for preference-based fine-tuning. Research groups without API access to proprietary models or without budgets for large-scale human reference creation can now use preference alignment effectively using only their own model's sampled outputs. This is not a conceptual breakthrough but a practical one with significant implications for who can participate in alignment research and deployment.

Follow-Up Research This Work Enables

1. Characterizing the interaction between CPO's β parameter and preference data quality. The paper uses a fixed β = 0.1 throughout, inheriting this value from Rafailov et al. (2024). The brittleness findings — CPO degrading under chrF alignment, failing with fixed chosen systems, and over-optimizing when rejected quality is too low — all raise the question of whether β misspecification is contributing to these failures. A β sweep across the problematic configurations would be highly informative: does a lower β (closer to pure SFT) rescue chrF-aligned CPO from catastrophic failure? Does a higher β in the fixed-chosen setting cause even worse degradation, confirming that CPO requires naturally varying chosen sources? The quality grid experiment (Figure 1) could be repeated at multiple β values to map out how the optimal (rejected, chosen) quality configuration shifts with preference strength. A concrete experimental design: take the nine mono-system quality grid datasets from Section 5.2, train CPO models at β ∈ {0.01, 0.05, 0.1, 0.5, 1.0, 2.0} on each, and plot downstream xCOMET-QE as a function of (chosen quality, rejected quality, β). This would reveal whether the optimal rejected quality depends on β — for instance, whether stronger preference signals require larger quality gaps to avoid overfitting, or whether weaker signals can work with tighter gaps. The negative result that β tuning cannot rescue chrF alignment would strengthen the paper's claim that lexical metrics are fundamentally incompatible with CPO; the positive result that chrF alignment works at very low β would soften this claim and suggest that the failure is a matter of calibration rather than incompatibility.

2. Human evaluation of the neural-lexical divergence to determine whether it represents quality degradation or benign style shift. The paper's most striking finding — that multi-system CPO aligned on xCOMET-QE drops en-xx chrF from 49.49 to 47.69 while SFT raises it to 50.28 (Table 1) — is entirely mediated through automatic metrics. The interpretation as "adverse metric effects" assumes that higher chrF is better, but this assumption is contestable. Neural metrics like xCOMET-QE are trained to predict human judgments and correlate more strongly with them than chrF does; it is possible that the CPO-aligned model is producing translations that are genuinely better (more fluent, more adequate) but happen to use different wording than the reference, causing lower lexical overlap. In this scenario, the chrF drop would be a benign consequence of producing more natural, less reference-bound translations — not a quality regression but a style shift away from the reference's surface form. A controlled human evaluation on a stratified sample of test sentences — specifically selecting sentences where CPO and SFT disagree substantially in their chrF scores but CPO scores higher on xCOMET-QE — would distinguish between these interpretations. Human raters would evaluate both CPO and SFT outputs for adequacy, fluency, and overall preference without access to the reference. If raters prefer CPO outputs despite lower chrF, the "adverse effect" is benign and the paper's concern about metric consistency is overblown for this metric pair. If raters prefer SFT outputs or find CPO outputs to contain fluency issues or meaning errors, the chrF drop reflects genuine quality degradation and the paper's warnings are validated. This experiment would be labor-intensive but would substantially increase the paper's practical impact by grounding its central claim in human judgments rather than metric-internal dynamics.

3. Iterative mono-system alignment to close the remaining gap with multi-system performance. The paper shows that mono-system CPO with optimized preference data achieves 89.58 xx-en xCOMET-QE versus multi-system's 89.95 (Table 4) — a small but statistically significant gap. A natural extension is iterative alignment: after the first round of mono-system CPO, the aligned model produces higher-quality greedy outputs. These improved outputs can be used as the base translations for a new round of candidate generation and preference data construction, with the expectation that the maximum achievable chosen quality will be higher (since the model itself has improved). Concretely: (1) align the base model using the optimized mono-system configuration from Figure 1 (point H: Rejected=Mid, Chosen=High), (2) use the aligned model to generate new candidate translations via top-p sampling, (3) construct a new preference dataset using the same offset-based selection relative to the new (improved) greedy baseline, (4) re-align from the original base model using this second-round preference data. The hypothesis is that the second-round chosen translations will have higher absolute quality than the first-round ones, providing a stronger training signal that narrows or eliminates the multi-system gap. A negative result — iterative alignment plateaus or degrades — would be equally informative, suggesting that mono-system alignment hits a fundamental ceiling determined by the base model's output distribution that no amount of self-generated data can surpass. This experiment would also characterize the saturation behavior of preference-based alignment: how many rounds of self-improvement are viable before the model's output distribution collapses or overfits?

4. Cross-model-family replication to test whether CPO brittleness is specific to ALMA-13B-LoRA or a general property of preference optimization. All experiments use a single model from the LLaMA family with LoRA adapters. The paper's claims about CPO's sensitivity to candidate systems, fixed-chosen settings, and lexical alignment could be influenced by properties specific to this model: its multilingual training data, its LoRA fine-tuning dynamics, its base translation quality distribution, or its in-context learning behavior. A minimal replication would test CPO on at least one model from a different family (e.g., Mistral-7B, Falcon-7B, or an encoder-decoder model like mT5) and at a different scale (7B vs. 13B). The key experiments to replicate: (a) the SFT-vs-CPO comparison on xCOMET-QE multi-system data (Table 1 pattern), (b) the fixed-chosen experiments (Table 3 pattern — does Chosen=Ref cause catastrophic CPO failure across model families?), and (c) the mono-system-vs-multi-system comparison (Table 4 pattern). If the brittleness patterns replicate across model families, they are likely fundamental to the CPO objective's interaction with heterogeneous candidate distributions — a finding that would carry substantial weight for the field. If they are specific to ALMA-13B-LoRA, the paper's warnings about CPO sensitivity need to be qualified as model-specific. A particularly valuable negative result would be finding a model family where CPO is robust to fixed-chosen settings and chrF alignment — this would suggest that the brittleness is trainable or architecture-dependent, opening a path toward more robust preference optimization methods.

5. Preference data construction from on-policy vs. offline samples in the mono-system setting. The paper's mono-system preference data is constructed entirely offline: all 50 candidates per source sentence are generated from the base model before any alignment occurs, and preference pairs are selected from this static pool. This is computationally efficient but means the candidates are off-policy relative to the aligned model — the model sees preference pairs drawn from its pre-alignment distribution, not from the distribution it is converging toward during training. An on-policy variant would periodically regenerate candidates from the model being trained (e.g., after every 10% of an epoch) and update the preference pairs accordingly, similar to how RLHF pipelines use the current policy for exploration. The comparison between offline and on-policy mono-system CPO would reveal whether the preference signal degrades as the model diverges from its initial distribution. A concrete experiment: train three models — offline mono-system (paper's approach), on-policy with candidate regeneration every 20 optimizer steps, and a hybrid that regenerates once halfway through training. Compare downstream xCOMET-QE and cross-metric consistency. If on-policy training significantly improves alignment metric performance, it suggests that offline preference data becomes stale during training and that the paper's mono-system gains could be further improved. If on-policy training degrades performance (as ReST $^{EM}$ did for revisions), it suggests that the preference signal from the model's own evolving distribution has pathological properties that reward hacking or distribution collapse. Either outcome advances our understanding of how preference optimization interacts with the data distribution.

6. Generalization of compute-optimal preference data construction to other alignment methods and tasks. The quality grid experiment in Section 5.2 — systematically varying chosen and rejected quality to find the optimal configuration — is the paper's most methodologically novel contribution. It is specific to CPO and xCOMET-QE alignment for MT in this paper, but the methodology generalizes immediately to other preference optimization methods (DPO, SLiC, KTO) and other tasks (summarization, dialogue, instruction following). A natural follow-up would apply the same grid methodology to DPO-based MT alignment: construct the same nine quality-grid datasets, train DPO models on each (with and without reference policy), and map the downstream performance surface. The comparison would reveal whether CPO and DPO have different optimal quality configurations — for instance, does DPO's reference policy term make it more or less sensitive to rejected quality than CPO's SFT term? Similarly, applying the grid methodology to summarization (e.g., on the TL;DR or CNN/DailyMail datasets with a neural metric like SummaC or UniEval as the alignment target) would test whether the optimal quality configuration is task-dependent. The paper's finding that moderate rejected quality (around 90% of base model quality) is optimal might be specific to translation, where the base model's outputs are already reasonably good; for summarization, where base model outputs might be more variable, a different configuration might be optimal. This line of work would transform the quality grid from a one-off analysis into a general methodology for characterizing preference alignment dynamics across methods, tasks, and metrics.

Practical Applications and Downstream Use Cases

1. Deployment of translation systems in resource-constrained environments without access to proprietary models. The mono-system finding directly benefits organizations that cannot query GPT-4 (or similar commercial APIs) for preference data generation — whether due to cost, data privacy restrictions, or API unavailability in their region. The paper demonstrates that mono-system CPO achieves 92.87 en-xx xCOMET-QE on WMT'22 versus 92.75 for multi-system CPO (Table 4, optimized variant) while requiring only the base model's own sampled outputs for data construction. For a team deploying a translation system for a low-resource language pair where commercial API coverage is limited or nonexistent, the mono-system approach provides a self-contained alignment pipeline: (1) sample K=50 candidates per training sentence from the base model, (2) score with a reference-free neural metric like xCOMET-QE or CometKiwi, (3) construct preference pairs with moderate rejected quality (~90% of base model quality) and maximally high chosen quality, (4) apply CPO fine-tuning, (5) deploy with greedy decoding. The paper's quality grid analysis (Figure 1) provides concrete guidance for step 3 that would otherwise require extensive trial-and-error. The practical benefit is that a team with a single GPU and an open-source model can achieve alignment gains previously thought to require multi-system candidate pools.

2. Quality-informed fine-tuning with metric consistency guarantees for production MT systems. Production translation systems are evaluated on multiple dimensions — adequacy, fluency, and surface fidelity are all monitored, and regressions on any axis can block deployment. The paper's finding that multi-system CPO creates neural-lexical divergence (Table 1: en-xx chrF dropping from 49.49 to 47.69) while mono-system CPO nearly preserves lexical performance (49.02 in Table 4) is directly actionable for production teams. If a team is currently using SFT on high-quality data (as in Alves et al., 2024) and wants to upgrade to preference-based alignment for the additional neural metric gains, the mono-system approach is the lower-risk path — it captures most of CPO's neural metric benefit (~90% of the multi-system gain on xx-en xCOMET-QE: 89.58 vs. 89.95) without triggering the lexical regression that would require additional monitoring, rollback procedures, or multi-metric optimization. For a production system serving millions of translations per day, a 1.8-point chrF drop could manifest as user-visible degradation on dimensions like terminology consistency or named entity preservation that lexical metrics partially capture. Avoiding this through mono-system alignment is a concrete risk reduction with minimal performance sacrifice.

3. Preference data construction for languages with limited reference translation availability. The paper's use of reference-free neural metrics (xCOMET-QE, CometKiwi) for both alignment and evaluation is practically significant for low-resource language pairs where human-written gold references are scarce or nonexistent. Traditional lexical metric alignment (chrF, BLEU) requires references — a chicken-and-egg problem for languages where the goal is to build translation capability from limited parallel data. The paper's demonstration that reference-free neural alignment works effectively (Table 1: xCOMET-QE alignment improves xCOMET-QE, CometKiwi, and even chrF under SFT) and that mono-system data generation removes the need for external systems means the entire alignment pipeline — candidate generation, scoring, pair selection, and evaluation — can operate without any reference translations. This is a non-trivial practical capability: a team working on translation for a newly documented language can fine-tune a base multilingual model, generate candidates via sampling, score them with a reference-free quality estimation model (which only needs source and hypothesis, no reference), and align the model using CPO — all without creating a reference set. The paper's chrF alignment failure (Table 1) actually reinforces the value of reference-free neural metrics for this use case, since it demonstrates that lexical metrics are unreliable alignment targets even when references are available.

4. Cost-efficient alignment for organizations iterating on translation quality. The paper's comparison between SFT and CPO provides a concrete economic tradeoff: CPO requires constructing preference pairs (chosen + rejected translations), which is roughly twice the data labeling cost of SFT (which only needs chosen translations), for a gain of ~0.6-0.8 xCOMET-QE points on xx-en (Table 1) and ~0.4-0.5 points on en-xx. For an organization deciding whether the additional data construction effort is worthwhile, the paper's ablation experiments provide decision-making guidance. If the organization has access to diverse, high-quality candidate systems (particularly a strong system like GPT-4), CPO's additional cost is justified by measurable metric gains. If the organization is limited to its own model's outputs, CPO's gain over SFT is smaller but still statistically significant (Table 4: 89.35 vs. 88.51 xx-en xCOMET-QE, a +0.84 point advantage). However, if the organization's evaluation pipeline includes lexical metrics that must not regress, SFT is the safer choice regardless, since it achieves neural improvements without lexical side effects (Table 1: SFT on xCOMET-QE improves en-xx chrF to 50.28 vs. base 49.49, while CPO drops it to 47.69). This decision framework — balancing metric gains against construction costs and cross-metric risk — is directly extractable from the paper's results and requires no additional experimentation to apply.