DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains

🎯 Pitch

Even the most advanced LLMs still fall short of professional human translators on long-form, domain-intensive Chinese-English documents. A new benchmark of 200 texts averaging over 1,700 tokens exposes a persistent performance gap, especially in literary and non-academic content where cultural nuance and stylistic fidelity matter most.

1. Executive Summary

This paper introduces DiscoX, a new benchmark for evaluating discourse-level and expert-level Chinese-English translation, alongside Metric-S, a reference-free automatic evaluation system. DiscoX comprises 200 professionally-curated texts spanning 7 domains—averaging over 1,700 tokens—and Metric-S employs a multi-agent LLM workflow that assesses translations across three dimensions: accuracy, fluency, and appropriateness, with a hierarchical error de-duplication mechanism that traces derivative issues back to root causes (e.g., attributing a disfluent phrasing to an underlying lexical mistranslation rather than penalizing both). The strongest tested model, GPT-5-high, achieves an overall score of 76.66, still trailing the human expert baseline of 80.16, while Metric-S attains 70.3% pairwise consistency with human judgments—more than doubling XCOMET-QE's 34.7% on the same benchmark. The paper establishes that even state-of-the-art LLMs remain below professional human translation quality on long-form, domain-intensive texts, with the performance gap widening substantially on literary and non-academic content where cultural nuance and stylistic fidelity are essential.

2. Context and Motivation

The Core Problem: Translation Benchmarks Assess Sentence-Level Competence, Not Real-World Professional Translation

The fundamental gap this paper addresses is deceptively simple: existing translation benchmarks evaluate fragments, but professional translation demands coherence across entire documents. Current evaluation practice treats translation as a sentence-level task—feed a sentence in, get a sentence out, compare against a reference. DiscoX argues that this paradigm fundamentally misses what makes professional translation difficult and valuable.

Consider what a professional translator actually does when handling a scientific article, legal contract, or technical manual. They don't translate sentences in isolation; they maintain terminological consistency across thousands of words (e.g., ensuring "consideration" is rendered consistently throughout a 50-page contract), preserve logical flow across section boundaries (e.g., ensuring the argument in the Discussion section properly references findings from Results), and adapt stylistic register to match the conventions of the target domain (e.g., using the formal conventions of Chinese academic prose rather than colloquial equivalents). These discourse-level demands—coherence, consistency, and stylistic continuity across an entire text—are simply invisible to sentence-level evaluation.

The paper visualizes this gap through concrete examples in Figure 2 and throughout the rubric design. In scientific articles, "failing to sustain coherence across sections can distort the logical flow of arguments"; in legal contracts, "inconsistent translation of specialized terminology may weaken their binding force"; in technical manuals, "imprecise or stylistically inappropriate renderings may lead to misunderstandings that jeopardize operational safety." These are not marginal concerns—they represent the core value proposition of professional translation, and they are precisely what existing benchmarks do not measure.

Why This Gap Matters Now

The timing of this work is not accidental. Three converging trends make discourse-level evaluation urgent:

1. LLMs are approaching human parity on sentence-level metrics, creating a misleading impression of progress. As the paper notes in Section 1, "state-of-the-art systems [are] approaching human performance" on segment-level tasks. WMT 2024 findings explicitly state that "the LLM era is here but MT is not solved yet" (Kocmi et al., 2024, cited as [17]). This creates a dangerous illusion: if our benchmarks say LLMs are near-human, but those benchmarks don't capture the dimensions that matter for real-world use, we risk deploying systems that appear competent but fail in practice. DiscoX is designed to expose precisely this gap—and the results in Table 3 confirm it, with the best LLM (GPT-5-high at 76.66) still significantly below human experts (80.16) on dimensions that sentence-level metrics would miss entirely.

2. The shift toward LLM-based translation systems changes the evaluation landscape. Traditional neural machine translation (NMT) systems were fundamentally sentence-level processors—they translated one segment at a time, so evaluating them one segment at a time was reasonable. But LLMs can process entire documents in context, and their translation behavior changes when they do: they can maintain consistency across sections, adapt style to document-level conventions, and make translation decisions informed by surrounding context. However, they also exhibit new failure modes at this scale—summarization instead of translation, omission of large sections, degradation of coherence over long outputs (Section 3.1 notes that "LLMs exhibit a tendency to deviate from translation instructions in discourse-level tasks, often defaulting to text continuation or summarization"). Evaluating these capabilities requires benchmarks designed for discourse-length texts, not sentence-length ones.

3. Expert-domain translation is economically and socially critical, yet under-evaluated. Knowledge dissemination across languages—scientific papers, legal agreements, technical documentation, literary works—depends on high-quality translation that preserves not just literal meaning but domain-specific conventions, terminological precision, and stylistic appropriateness. The paper's construction effort itself testifies to this importance: 133 professionals (115 vertical domain experts and 18 linguistic specialists) contributed 1,330 person-hours to curate just 200 test cases (Section 2.1). This is not a toy problem; it represents the kind of translation work that costs organizations millions of dollars annually and where errors carry real consequences.

Where Existing Approaches Fall Short

The paper identifies failures across three fronts: benchmarks, metrics, and evaluation methodologies.

Benchmarks: Short Texts, General Domains, No Discourse Demands

Table 1 (a) makes the gap quantitative and stark. FLORES (Goyal et al., 2022) provides professionally translated sentences across 101 languages—but its average text length is 48.88 tokens. WMT 2024 spans news, speech, social media, and literary content—with an average of 45.84 tokens. Redtrans Bench (Guo et al., 2025) targets social network content with culturally-loaded and humorous material—at 59.46 tokens average. These are all effectively sentence- or short-paragraph-level benchmarks. DiscoX, by contrast, provides an average of 1,712.17 tokens per text—roughly 35 times longer than WMT and FLORES, and nearly 29 times longer than Redtrans Bench.

This length difference is not cosmetic; it fundamentally changes what evaluation means. At 50 tokens, a text might be a single complex sentence. At 1,700 tokens, a text is a full academic abstract with introduction, methods, results, and discussion—or several paragraphs of a legal contract with cross-referenced clauses—or a complete short essay with argumentative structure. Translation quality at this scale involves dimensions that are undefined at the sentence level: inter-sentential coherence (does paragraph 3 follow logically from paragraph 2?), terminological consistency (is the same technical term translated identically across all occurrences?), register maintenance (does the translation sustain formal academic register throughout, or does it drift into colloquialism?), and structural fidelity (does the translation preserve the original's section breaks, argument flow, and rhetorical moves?).

The WMT 2023 shared task on discourse-level literary translation (Wang et al., 2023, cited as [28]) highlighted this exact issue, as the paper notes: "the low accuracy of traditional metrics led to results with questionable credibility." This is a candid admission from the field itself—when we tried to evaluate discourse-level translation with our existing tools, we got unreliable results.

Metrics: Reference-Dependence, Limited Dimensionality, Weak Discourse Sensitivity

The paper categorizes existing metrics into three generations, each with limitations that compound at discourse scale:

N-gram metrics (e.g., ChrF, BLEU): These compare n-gram overlap between the translation and one or more reference translations. They fail on discourse-level texts for several reasons. First, they require reference translations, but at 1,700 tokens, the space of valid translations is so large that no single reference can represent it—two professional translators will produce substantially different but equally correct documents (as the paper notes in Section 6.2, "no single reference can capture the full range of valid outputs"). Second, n-gram metrics are fundamentally local: they compare tokens within a small window, making them blind to long-range dependencies like terminological consistency or argument structure. Third, they provide no diagnostic information—a low ChrF score tells you the translation is bad, but not why it's bad (accuracy? fluency? style?).

Neural metrics (e.g., COMET, MetricX, XCOMET): These use learned representations to score translations. The paper specifically evaluates XCOMET-QE (Guerreiro et al., 2024, cited as [10]) as a state-of-the-art reference-free metric. Table 4 delivers the decisive result: XCOMET-QE achieves only 34.7% pairwise consistency with human judgments on DiscoX, compared to Metric-S at 70.3%—a gap of over 35 percentage points. The divergence is catastrophic at the system level for Chinese-to-English translation: XCOMET-QE scores just 10.0% consistency (essentially random), while Metric-S maintains 80.0%. This is not a small performance difference; it suggests that XCOMET-QE is fundamentally unable to distinguish better from worse translations on discourse-level texts. The paper attributes this to the fact that these metrics, while more sophisticated than n-gram approaches, still operate primarily on local adequacy and fluency signals and "their evaluation accuracy declines when dealing with discourse-level texts, where evaluation requires modeling discourse-level coherence, consistency, and stylistic continuity" (Section 6.2).

LLM-as-a-judge (single model): The most recent paradigm uses an LLM to directly evaluate translation quality by prompting it with evaluation criteria. This approach offers flexibility and explainability without requiring reference translations. However, the paper's ablation experiments (Appendix A.2.1, Table 7) reveal that a single LLM with a detailed prompt achieves only 55.9% average consistency with human judgments on DiscoX, while a single LLM with a simple prompt drops to 24.9%—worse than XCOMET-QE. The paper identifies two failure modes: first, single-judge LLMs are vulnerable to known biases including position bias, verbosity bias, and self-preference bias (Wang et al., 2023, cited as [31]), which are amplified on long texts where evaluation requires sustained attention; second, a single LLM struggles to simultaneously assess accuracy, fluency, and appropriateness without conflating or missing errors—the ablation where all three dimensions are merged into a single prompt drops to 55.9%, compared to 70.3% for the multi-agent workflow (Table 7).

The self-preference bias finding in Appendix A.3 (Table 9) is particularly instructive: when o3-high is used as the judge, it ranks its own outputs first even though human experts rank it third. Gemini-2.5-Pro, by contrast, shows no self-preference bias, which the paper identifies as the rationale for selecting it as the default judge model.

Evaluation Methodology: Single-Dimension Scoring Misses the Multi-Faceted Nature of Translation Quality

Beyond specific benchmarks and metrics, the paper identifies a methodological gap: existing evaluation frameworks typically produce a single score, collapsing accuracy, fluency, and stylistic appropriateness into one number. This is adequate for sentence-level tasks where these dimensions are largely correlated—a sentence that is accurate is usually also fluent and stylistically appropriate. But at discourse scale, these dimensions can diverge dramatically.

The results in Table 3 make this concrete. Claude-4 achieves an Accuracy score of 39.38 (second only to GPT-5-high at 48.65) but a Fluency score of just 5.98—meaning it produces translations that are factually accurate but barely readable. Kimi-K2 shows the inverse pattern: it leads in Fluency at 16.44 but scores only 27.63 on Accuracy. A single-score metric would place these two models somewhere in the middle of the ranking and obscure their radically different failure profiles. Metric-S's multi-dimensional output (Accuracy / Fluency / Appropriateness, each with specific error taxonomies) makes these profiles visible and actionable—the paper explicitly frames this as enabling targeted model improvement rather than just ranking.

How This Paper Positions Itself

DiscoX positions itself not as an incremental improvement over existing benchmarks, but as a category-defining contribution—the first benchmark specifically designed for discourse-level, expert-domain translation evaluation. This is a strong claim, and the paper supports it through several design choices that distinguish it from prior work:

Benchmark design philosophy: real-world professional scenarios, not curated sentence pairs. The three-stage curation pipeline (Figure 3) is central to this positioning. Rather than extracting sentence pairs from existing parallel corpora, DiscoX sources complete texts from practicing professionals (115 vertical domain experts) and subjects them to a rigorous filtering process where texts are retained only if both of two state-of-the-art LLMs fail on at least eight predefined rubrics (Section 2.1). This "difficulty threshold" design means DiscoX is intentionally constructed to be hard in ways that matter—the rubrics encode domain-specific challenges that LLMs demonstrably struggle with, ensuring the benchmark discriminates effectively rather than saturating quickly. The paper notes a selection rate of approximately 30% from the initial 665 candidate texts, with the explicit goal of maximizing "precision and evaluative utility" (Section 2.1).

Metric design philosophy: structured multi-agent evaluation, not single-model scoring. Metric-S is positioned as a workflow rather than a single evaluation call. It decomposes evaluation into four stages—instruction-following check, quality estimation (three specialist judges), error de-duplication and attribution, and severity-weighted scoring—each handled by a separate LLM agent with a specialized prompt (Section 3, Figure 4). The ablation results in Table 7 validate this decomposition: removing de-duplication drops consistency from 70.3% to 66.0%, switching to equal weighting across dimensions drops it to 48.2%, and collapsing to a single judge with a simple prompt drops it to 24.9%. The paper's argument is that discourse-level evaluation is too complex for a single evaluation call—it requires structured reasoning about distinct dimensions, root-cause analysis of errors, and weighted aggregation, which the multi-agent workflow provides.

Pairing benchmark and metric as an integrated system. A distinctive aspect of DiscoX's positioning is that the benchmark and evaluation metric are designed together, not independently. The rubrics authored during data annotation (Section 2.1) feed directly into Metric-S's Accuracy judge (Section 3.2), which checks whether key terms are handled correctly according to expert specifications. The three-dimensional quality decomposition (accuracy, fluency, appropriateness) structures both the human evaluation framework and the automated metric, enabling the consistency validation in Table 4. This co-design contrasts with the typical workflow where benchmarks are built first and metrics are selected or developed post-hoc, often leading to poor alignment between what the benchmark tests and what the metric measures (as WMT 2023 experienced with discourse-level literary translation).

Scope: Chinese-English, both directions, academic and non-academic domains. The paper positions DiscoX within a specific and important language pair (Chinese-English, the most widely translated language pair globally by volume) and across seven secondary domains spanning academic (Social Sciences, Natural Sciences, Humanities, Applied Disciplines) and non-academic (News and Information, Domain-Specific Scenarios, Literature and Arts) contexts. This breadth is deliberate: it enables the paper to analyze domain-dependent and direction-dependent performance patterns (Section 5.1, 5.2) rather than simply reporting aggregate scores. The finding that models are systematically stronger on zh→en than en→zh—with DeepSeek-V3 showing a 34.8-point gap—and stronger on academic than literary content are empirical results that a narrower benchmark could not produce.

The Paper's Central Thesis

Implicit in DiscoX's design is a thesis about what professional translation evaluation requires: discourse-level coherence, domain-specific precision, and multi-dimensional quality assessment are not optional luxuries but core requirements, and any evaluation framework that omits them will systematically overestimate translation quality while providing no diagnostic signal for improvement. The paper validates this thesis negatively (existing metrics fail dramatically on DiscoX, as shown in Tables 4 and 7) and positively (Metric-S achieves strong human alignment, and its multi-dimensional output reveals complementary strengths and weaknesses across models that single-score metrics would obscure). The human expert still outperforms all LLMs (Table 3), establishing DiscoX as a benchmark with headroom—a critical property for driving future progress rather than declaring premature victory.

3. Technical Approach

3.1 Reader Orientation

This paper develops two interconnected systems: a benchmark called DiscoX for testing discourse-level, expert-domain translation, and an automatic evaluation metric called Metric-S for scoring translations on that benchmark. The core problem is that existing translation benchmarks use short, sentence-length texts and produce single-number scores, neither of which captures the challenges of professional translation—maintaining consistency, style, and terminological precision across thousands of words of specialized content. The solution takes the shape of a co-designed benchmark-and-metric pair: DiscoX provides 200 rigorously-curated, long-form texts with expert-authored rubrics specifying critical translation checkpoints, and Metric-S uses a multi-agent LLM workflow that decomposes evaluation into three independent dimensions, traces errors back to their root causes, and aggregates them into a severity-weighted final score.

3.2 Big-Picture Architecture (Diagram in Words)

The DiscoX system has two major subsystems that operate in sequence:

Subsystem 1: DiscoX Benchmark Construction (offline, completed once). This is a three-stage curation pipeline—Data Annotation, Quality Control & Filtering, and Reviewing & Selection—that transforms 665 candidate texts from 115 vertical domain experts into 200 validated evaluation tasks, each paired with expert-authored rubrics specifying domain-critical translation requirements. The pipeline's job is to produce a test set that is simultaneously long, expert-level, and difficult enough that current state-of-the-art models fail nontrivially.

Subsystem 2: Metric-S Evaluation (applied to each translation output). This is a four-stage processing pipeline—Instruction Following Check, Quality Estimation (three specialist LLM judges), Error De-duplication and Attribution, and Severity-Weighted Score Calculation—that takes a source text and a model's translation output, verifies the output is a valid translation, identifies errors across accuracy, fluency, and appropriateness dimensions, resolves duplicate error counts by tracing derivative errors to root causes, and outputs a final score in the range 0–100 with per-dimension breakdowns.

Information flows as follows: a source text from DiscoX is provided to a translation model → the model produces an output → Metric-S first checks if the output is a valid translation (instruction-following gate) → if valid, three specialist judges independently evaluate the output for accuracy, fluency, and appropriateness errors → a de-duplication judge resolves overlapping error claims → the final score calculator applies severity-weighted deductions and outputs Accuracy, Fluency, Appropriateness, and Overall scores.

3.3 Roadmap for the Deep Dive

First, the DiscoX benchmark construction pipeline—how texts are sourced, filtered for difficulty, and paired with rubrics—because the benchmark's design directly determines what Metric-S measures and why existing metrics fail on it.
Second, the instruction-following check, since it is the gating mechanism that prevents Metric-S from evaluating non-translation outputs (summaries, continuations) as if they were translations.
Third, the three quality estimation judges (Accuracy, Fluency, Appropriateness) and their specialized prompts, because they form the core evaluation engine and embody the paper's decomposition of translation quality into independent dimensions.
Fourth, the error de-duplication and attribution mechanism, since it is the most novel component of the Metric-S workflow and directly addresses the double-counting problem that arises from multi-dimensional evaluation.
Fifth, the DiscoX score calculation formula, including the severity weighting scheme and dimension weights, which translates identified errors into a final numeric score.
Sixth, the human judgment alignment methodology, because it provides the validation framework that justifies Metric-S's design choices and enables comparison against baseline metrics.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark-and-metric design paper whose core idea is that discourse-level, expert-domain translation evaluation requires (a) test cases that are long, specialized, and deliberately difficult, and (b) an evaluation workflow that decomposes quality into independent dimensions, traces errors to root causes, and weights errors by severity.

DiscoX Benchmark Construction: Three-Stage Expert Curation Pipeline

The DiscoX benchmark is not assembled from existing parallel corpora or machine-extracted sentence pairs. Instead, it is constructed through a human-intensive, three-stage pipeline (Figure 3) designed to produce 200 test cases that satisfy three simultaneous requirements: (1) discourse-level length (average exceeding 1,700 tokens), (2) expert-domain specialization across seven secondary domains, and (3) sufficient difficulty that current state-of-the-art LLMs fail on multiple predefined criteria.

Stage 1: Data Annotation. The pipeline begins with 115 Vertical Domain Experts—professionals with 4–10 years of experience in practical fields, or Master's/PhD holders from top-tier Chinese universities for academic disciplines (Appendix B, Table 12). Each expert collects source texts from their respective fields. The paper imposes three requirements on every candidate text:

The text must "reflect authentic professional scenarios"—meaning it should be the kind of text that would actually be translated in professional practice, not a constructed or simplified example.
The text must "exceed a minimum length of 1,500 characters (Chinese) or words (English)"—this length threshold is what pushes DiscoX into the discourse-level regime, since 1,500 words is approximately 3–5 pages of academic prose, far beyond the 45–60 token averages of existing benchmarks.
The text must be "specific, self-contained, and amenable to the creation of unambiguous rubrics"—meaning it must contain identifiable domain-critical elements (technical terms, culture-loaded expressions, stylistically marked passages) that can be turned into verifiable evaluation criteria.

Each source text is then paired with "a comprehensive set of expert-authored rubrics that delineate specific, verifiable evaluation criteria" across four categories: Grammar, Topic Terms, Terminology, and Culture-loaded Words. A rubric is a concrete checkpoint specifying exactly what the translation must get right. The paper provides examples in Section 2.1 and Appendix C.3. For instance, a rubric for a literary text reads: "Checkpoint 1: The term ('yuanzi') in context must be translated as 'Ditan Park' or 'the park', not 'garden'." For an academic text, a rubric might read: "'组织学分型' must be translated as 'Histologic type' or 'Histologic sub-type' and must be consistent throughout." For a legal text: "'hold harmless and defend' must be translated as '使……免受损害，并为其进行抗辩'; omitting 'defend' is an error."

The critical design choice here is that rubrics are pre-authored by domain experts, not discovered post-hoc or generated automatically. This ensures that the evaluation criteria are grounded in genuine domain requirements rather than whatever a language model happens to notice. It also enables a key feature of Metric-S: the Accuracy judge receives these rubrics as explicit checkpoints to verify (Section 3.2), making the evaluation reproducible and criterion-referenced rather than purely subjective.

This initial phase yields 665 candidate tasks with an average of 9.38 rubrics each.

Stage 2: Quality Control and Filtering. The initial pool of 665 tasks undergoes a two-part quality control process designed to ensure both professionalism and difficulty.

First, linguistic specialists (the 18 Linguistic Experts described in Appendix B—certified MTI graduates with 5–16 years of experience and high-level language certifications such as TEM-8 or IELTS 8.0) conduct peer review to "ensure textual professionalism." This review checks that source texts are well-formed, rubrics are concrete and verifiable, and the overall task reflects authentic professional demands.

Second, and more distinctively, each task is subjected to an LLM difficulty verification test. The paper tests each candidate text against "two SOTA LLMs" (the exact models are not named in the main text, but Appendix C.1 references the system prompt used). A task advances to the next stage only if both models fail on a minimum of eight predefined rubrics. In operational terms: for each of the 665 candidate texts, the two LLMs produce translations, each translation is checked against all ~9.38 rubrics for that text, and if either model correctly handles eight or more of the rubric checkpoints, the task is rejected as too easy.

This is a deliberately stringent threshold. The paper describes it as "a stringent difficulty threshold" and the filtering stage as designed "to establish a high and standardized difficulty" (Section 2.1). The explicit rationale is that a benchmark that current models can already ace provides no signal for improvement—by filtering for tasks where both tested models fail on at least eight domain-critical terms, DiscoX ensures it measures capabilities that remain challenging.

The filtering stage also includes a qualitative component: tasks flagged as "too easy" are explicitly marked in the pipeline diagram (Figure 3) with the notation "ANYONE passes (TOO EASY !!!)" and are removed. This dual-filtering—linguistic peer review for quality, LLM difficulty testing for challenge level—is what gives DiscoX its distinctive character as a stress test rather than a typical evaluation set.

Stage 3: Reviewing and Selection. The tasks that survive Stage 2 enter a final expert review stage. From this pool, domain experts "select the final 200 tasks, representing a selection rate of approximately 30%" (Section 2.1). This selection is not random; it is designed to ensure "a balanced and diverse distribution across domains" (Section 2.2, Table 2). The experts then perform "a final refinement, correcting any remaining flaws in the source texts and honing the rubrics based on the error patterns that are observed in the LLM outputs from the filtering stage."

This last refinement step is noteworthy: the rubrics are not static but are iteratively improved based on actual LLM failure patterns. If the filtering stage reveals that models consistently fail on a particular type of terminological distinction, the experts sharpen the corresponding rubric to make the evaluation criterion more precise. This creates a feedback loop between empirical model weakness and evaluation design, ensuring that the final benchmark targets precisely the capabilities that are most lacking.

Domain composition and scale. The resulting 200 tasks are distributed across two primary domains (Academic: 121 tasks; Non-Academic: 79 tasks) and seven secondary domains as detailed in Table 2. The Academic domain comprises Social Sciences (38 tasks), Natural Sciences (35), Humanities (28), and Applied Disciplines (20), with an average token length of 1,875.58. The Non-Academic domain comprises News and Information (37), Domain-Specific Scenarios (28), and Literature and Arts (14), with an average token length of 1,450.49. The overall average is 1,712.17 tokens, which the paper notes is "substantially longer than those in typical segment-level translation benchmarks" (Section 2.2)—by a factor of approximately 35× compared to FLORES and WMT.

The paper reports a total curation cost of 1,330 person-hours across 133 professionals (Section 1, Abstract). This scale is important not just as a badge of effort but as evidence that constructing discourse-level, expert-domain benchmarks is fundamentally more expensive than constructing sentence-level ones—you need domain experts to author rubrics, linguistic experts to verify quality, and iterative testing to establish difficulty thresholds. This cost structure partially explains why such benchmarks haven't existed before.

Why this pipeline design over alternatives? The paper makes implicit design choices that distinguish DiscoX from prior benchmarks:

Expert-authored rubrics over post-hoc evaluation criteria: Rather than defining evaluation dimensions abstractly and asking judges to apply them, DiscoX pre-specifies concrete checkpoints that encode domain knowledge. This makes evaluation more reproducible and reduces the burden on the evaluator (human or automated) to independently possess domain expertise.
Difficulty filtering over representative sampling: Most benchmarks aim to be representative samples of a domain. DiscoX explicitly rejects representativeness in favor of difficulty—texts are retained only if they are demonstrably hard for current models. This makes DiscoX a "stress test" benchmark, analogous to adversarial test sets in NLP, designed to expose capability gaps rather than measure average performance.
Iterative rubric refinement based on model errors: The rubrics are not fixed at annotation time but are refined in Stage 3 based on observed LLM failures. This creates a benchmark that is actively tuned to discriminate among models, rather than one that might accidentally include tasks where all models perform identically.

Metric-S: Instruction Following Check

Before any quality evaluation occurs, Metric-S applies a binary gate: the Instruction Following Check (Section 3.1). The problem this addresses is specific to discourse-level translation with LLMs: "LLMs exhibit a tendency to deviate from translation instructions in discourse-level tasks, often defaulting to text continuation or summarization" (Section 3.1). When given a long source text, a model might produce a summary of the text, a commentary on it, or a continuation in the same language—none of which constitute a valid translation output, but all of which might superficially resemble one (being in the target language and topically related to the source).

The Instruction Following Judge (detailed prompt in Appendix C.2.1) checks for specific violations:

Empty output: The model produces no translation at all.
Language mismatch: The output is in the same language as the source, indicating the model did not perform translation but instead continued, summarized, or analyzed the text.
Severe omission: The output omits more than 50% of the source content ("输出译文大篇幅漏译，经比较漏译大于 50%" in the Chinese prompt).

The judge also specifies what counts as acceptable: if the model first summarizes the source text and then provides a full translation, this is considered valid because the translation component is present. Similarly, if the output contains code-switching (mixed Chinese and English) but overall constitutes a translation attempt with more than 50% in the target language, it passes.

Any output that fails the Instruction Following Check is "immediately assigned a zero score and excluded from further evaluation" (Section 3.1). This is a hard filter: if the output is not a valid translation, there is no point in evaluating its accuracy, fluency, or appropriateness.

Why this gate exists: For sentence-level translation, instruction following is rarely an issue—models reliably translate when given short inputs. For discourse-level translation, the failure mode is common enough that evaluating non-translations as if they were translations would inflate error counts (every omitted section would count as a mistranslation) and produce uninterpretable scores. The gate ensures that Metric-S only evaluates genuine translation attempts.

Metric-S: Quality Estimation — The Three Specialist Judges

For outputs that pass the instruction-following check, Metric-S deploys three independent LLM judges, each with a specialized prompt and a distinct evaluation mandate (Section 3.2, Figure 4). The full prompts for each judge are provided in Appendix C.2.2. The core design principle is specialization: rather than asking one judge to evaluate everything simultaneously, Metric-S gives each judge a narrow, well-defined responsibility with explicit instructions to ignore dimensions handled by other judges.

Accuracy Judge

The Accuracy Judge evaluates "how faithfully the translation preserves the source text's meaning, factual information, and emotional tone" (Section 3.2). Its mandate covers five specific error types, each defined in the judge prompt (Appendix C.2.2):

Mistranslation (错译): The translation's meaning differs from the source. This includes cases where individual words are translated correctly but the overall semantic meaning is wrong, and cases where the translation is lexically accurate but contextually inappropriate.
Omission (漏译): The translation lacks factual information or emotional expression present in the source. The judge prompt specifically distinguishes between minor omissions (affecting a word or phrase) and severe omissions (summarizing content, causing semantic drift).
Untranslated (未译): The translation retains source-language text without translating it. The judge prompt provides explicit examples: if a translation outputs "The 天气 is pretty good" (retaining Chinese "天气" for "weather" without translation), this is flagged as Untranslated and assigned the highest severity level (Extremely Critical). However, if the translation follows standard conventions—e.g., rendering "雨果 (Victor Hugo)" where the Chinese translation precedes the parenthetical original—this is considered correct because the target-language text is present.
Over-translation (增译): The translation adds factual information or emotional content not present in the source.
Other accuracy issues: A catch-all for accuracy problems not covered by the above categories.

Integration with rubrics. A distinctive feature of the Accuracy Judge is that it receives the expert-authored rubrics (from Stage 1 of benchmark construction) as explicit checkpoints to verify. The prompt structure in Appendix C.2.2 (labeled "PE of Checkpoints") instructs the judge to:

Check each rubric checkpoint in order against the translation
For each checkpoint, determine whether the translation correctly handles the specified term or expression
Output the result of each checkpoint check, regardless of whether it passes or fails

This rubric integration is what makes Metric-S's accuracy evaluation criterion-referenced rather than purely open-ended. The rubrics encode domain-specific requirements (e.g., "ETV6-NTRK3 gene rearrangement must be translated as 'ETV6-NTRK3 基因重排'") that a general-purpose LLM judge would not independently know to check. By providing these as explicit evaluation targets, Metric-S ensures that domain-critical errors are systematically detected.

Severity levels for accuracy errors. The Accuracy Judge classifies each detected error into one of three severity levels:

Major (普通): Word-level mistranslations or omissions that affect understanding of a phrase but not the paragraph. The prompt's description: "词级别的错译，漏译，影响语句理解，不影响段落理解" (word-level mistranslation or omission, affects sentence understanding but not paragraph understanding).
Critical (严重): Sentence-level mistranslations or omissions, or added content that distorts the meaning of a substantial passage. Also includes errors in key structural elements like section headings. The prompt warns judges to use this label judiciously: "在进行问题严重程度分类时，请审慎打出【严重 (Critical)】标签" (when classifying severity, please apply the Critical label cautiously).
Extremely Critical (非常严重): Reserved for two specific cases: (a) retaining source-language text without translation, and (b) very large-scale omissions or summary-style output. The prompt explicitly states: "【非常严重 (Extremely Critical)】标签只适用于我提到的特殊保留未译，以及极其严重的漏译情况" (the Extremely Critical label applies only to the specific untranslated cases I mentioned, and extremely severe omission cases).

Fluency Judge

The Fluency Judge evaluates "the quality of the translation from the perspective of the target language" (Section 3.2), assessing whether the translated text reads as if produced by a native speaker. Its mandate explicitly excludes accuracy concerns—the prompt instructs: "你无需关注翻译错误、中英夹杂等其他问题" (you need not focus on translation errors, code-switching, or other such issues). The Fluency Judge covers four error types:

Language fluency (语言流畅度): The translation must be smooth and natural, avoiding word-for-word translation that produces stilted or awkward output. The prompt specifically calls out "翻译腔" (translationese)—the unnatural phrasing that results from adhering too closely to source-language structures. Long and complex sentences receive particular scrutiny.
Grammatical correctness (语法正确性): The translation must be free of morphological and syntactic errors. The prompt states: "译文的词法、句法不能存在问题，不能出现病句" (the translation's morphology and syntax must not have problems; ungrammatical sentences are not permitted).
Logical coherence (逻辑连贯性): The translation must preserve the source text's logical structure. The prompt provides specific distinctions: if the source uses parallel structure, the translation must not convert it to a progressive structure; if the source uses causal reasoning, the translation must not weaken or omit the causal relationship. The prompt also notes a cross-linguistic difference: "中文句子间不太关注逻辑衔接，对只需意会的表达容忍度较高；而英文表达要求用词准确、逻辑清晰、表意明确" (Chinese sentences are less concerned with explicit logical connectors and tolerate implied meaning; English expression requires precise wording, clear logic, and explicit meaning).
Lexical consistency (词汇一致性): Key terms—especially proper names, place names, technical terms, and culturally specific vocabulary—must be translated consistently throughout the text. The prompt gives the example of "Syracuse," which can be translated as either "锡拉库萨" or "叙拉古"; using both in the same document could lead a reader to think they refer to different places.

Severity levels for fluency errors. The Fluency Judge uses a simpler severity scheme than the Accuracy Judge: each error is simply classified as "有问题" (has a problem). This is because fluency issues, while they degrade reading quality, do not have the same potential for catastrophic meaning distortion as accuracy errors, and the paper weights them correspondingly lower in the final score (20 points maximum for Fluency vs. 60 for Accuracy).

Appropriateness Judge

The Appropriateness Judge evaluates "a higher-level expectation of translation quality" (Section 3.2) that goes beyond basic usability to assess whether the translation preserves stylistic, emotional, and cultural dimensions of the source. The judge prompt frames this as the role of "a strict literary critic" (一位严格的文学评论家) who is "extremely sharp, skilled at detecting subtle differences in style, emotional coloring, literary merit, and cultural dimensions" (你极度敏锐，擅长洞察文本在风格、情感色彩、文采和文化层面的细微差异). The Appropriateness Judge covers four dimensions:

Style (风格): Whether the translation matches the source's register and genre conventions. The prompt example contrasts academic prose ("The prevailing neuroscientific paradigm posits...") with an inappropriately colloquial translation ("现在主流的看法是..."—using casual phrasing like "看法" instead of formal academic terminology).
Emotional tone (情感色彩): Whether the translation preserves the source's affective content. If the source expresses joy, the translation must also express joy at a comparable intensity.
Literary merit (文采): Applicable when the source is a literary text. The translation must achieve "comparable vividness, imagery, rhythm, and artistic beauty" (在语言运用上达到了与原文同等的生动性、形象性、节奏感和艺术美感) to the original.
Cultural adaptation (文化适应): Whether culturally loaded expressions—metaphors, allusions, slang, puns—are handled so that target-language readers can understand the intended meaning without losing the cultural specificity.

Severity levels for appropriateness errors. Like the Fluency Judge, the Appropriateness Judge uses a simple binary: "有问题" (has a problem). The prompt describes this as: "译文虽然可用，但在风格上未能 100% 复现原文的精髓" (the translation is usable, but fails to 100% reproduce the essence of the original in style). This framing is important: appropriateness errors are about failing to achieve excellence, not about producing unusable output—which is why they are weighted at only 20 points in the final score.

Why three judges instead of one? The paper's ablation study (Appendix A.2.1, Table 7) provides direct evidence for this design choice. A single LLM judge with a merged prompt covering all three dimensions achieves only 55.9% average pairwise consistency with human judgments, compared to 70.3% for the three-judge workflow. The paper attributes this to the difficulty of simultaneously attending to accuracy, fluency, and appropriateness in a single evaluation pass, especially for long texts where the cognitive load of evaluation is already high. By separating these dimensions, each judge receives a focused prompt that specifies exactly what to look for and—crucially—what to ignore, reducing the risk of dimension conflation and error double-counting.

Error De-duplication and Attribution

The most architecturally novel component of Metric-S is the error de-duplication and attribution stage (Section 3.3). The problem it solves is structural to multi-dimensional evaluation: "a single root error can propagate into multiple derivative issues" (Section 3.3). If a model mistranslates a key term, the Accuracy Judge might flag it as a mistranslation, the Fluency Judge might flag the resulting awkward phrasing as a fluency problem, and the Appropriateness Judge might flag the loss of the original's stylistic register—all of which trace back to the same root cause. Without de-duplication, this single error would be penalized three times, producing an unfairly low score.

The De-duplication Judge (prompt in Appendix C.2.3) operates according to a hierarchical priority system:

Priority 1: Extremely Critical Accuracy errors. The prompt states: "accuracy 的【非常严重 (Extremely Critical)】标签是最高优先级，无论是什么内容，和谁重复，都保留【非常严重 (Extremely Critical)】" (the Accuracy Extremely Critical label is the highest priority; regardless of content or what it overlaps with, the Extremely Critical label is retained). This means that if an untranslated segment is flagged as both an Accuracy error (Extremely Critical) and a Fluency error, only the Accuracy error is kept.

Priority 2: Rubric-defined violations. The prompt states: "无论哪个维度和 checkpoints 重复，均归因为 checkpoints，这是第二优先级" (regardless of which dimension overlaps with checkpoints, the error is attributed to checkpoints; this is the second priority). This ensures that rubric violations—which encode domain-specific requirements—are systematically attributed to the Accuracy dimension (since rubrics are fed to the Accuracy Judge), even if other judges independently notice the same issue.

Priority 3: Causal analysis for remaining overlaps. For errors that don't fall under Priority 1 or 2, the De-duplication Judge performs causal analysis: "causal analysis determines which error is primary" (Section 3.3). The paper provides a worked example in Appendix C.4. In the example, a sentence from the source text ("Does Machiavelli hold up Borgia as the model prince?") is omitted from the translation. The Accuracy Judge flags this as an Omission error (type: 漏译). The Fluency Judge independently flags it as a Logical Coherence error (type: 逻辑连贯性), noting that omitting the rhetorical question disrupts the argument flow. The Appropriateness Judge also flags it as a Style error (type: 风格), noting that converting a rhetorical question to a declarative statement changes the text's genre register. The De-duplication Judge identifies: "三者均指出了译文遗漏了原文关键设问句... 根本原因为内容遗漏" (all three identify the omission of the key rhetorical question... the root cause is content omission). It attributes the error to Accuracy and instructs the deletion of the overlapping Fluency and Appropriateness errors. The final output shows only the Accuracy Omission error retained, while the Fluency and Appropriateness judges' outputs are trimmed to include only their non-overlapping errors.

What the De-duplication Judge does NOT do: The prompt explicitly excludes two cases from de-duplication review:

Errors where different dimensions identify genuinely different problems are not deduplicated. For example, if the Accuracy Judge flags a Style error as "this belongs in Accuracy dimension," the De-duplication Judge does not intervene—these are dimension classification disagreements, not duplicate penalties.
Cases where different judges reach different conclusions about the same text segment are not deduplicated. If the Fluency Judge says a passage has a problem but the rubric check says it's correct, the De-duplication Judge does not reconcile these—they reflect genuine disagreement between evaluation perspectives, not redundant error counting.

Why this mechanism matters: The ablation in Table 7 shows that removing the de-duplication step drops Metric-S's average pairwise consistency with human judgments from 70.3% to 66.0%. This is a meaningful but not catastrophic drop (4.3 percentage points), which suggests that while de-duplication provides a measurable improvement, the larger gains in Metric-S come from the multi-dimensional evaluation structure itself. However, de-duplication likely matters more for score fairness than for ranking consistency—a model that makes one catastrophic error should not be penalized three times for it, even if the relative ranking of models doesn't change much.

The de-duplication mechanism also serves an attribution function beyond score correction. By identifying root causes, it produces a cleaner error taxonomy where each error is classified under its primary dimension. This matters for the paper's stated goal of providing "clear, actionable feedback" (Appendix A.4): a model developer who sees "3 Accuracy errors, 2 Fluency errors, 1 Appropriateness error" receives a more interpretable diagnostic than one who sees "6 errors" with no root-cause analysis.

DiscoX Score Calculation

The final score is computed from the de-duplicated error lists using a weighted, severity-sensitive formula (Section 3.4). The overall score is defined as:

$\text{Score} = S_{\text{Acc}} + S_{\text{Flu}} + S_{\text{App}}$

where for each dimension $x \in \{\text{Acc}, \text{Flu}, \text{App}\}$ , the subscore is:

$S_x = \text{MAX}_x - \sum_{i=1}^{N_x} w_i^x e_i^x$

where $\text{MAX}_x$ is the maximum possible score for dimension $x$ , $N_x$ is the number of errors detected in that dimension, $w_i^x$ is the point deduction for error $i$ in dimension $x$ (determined by its severity level), and $e_i^x$ is an indicator that the error $i$ exists (always 1 for each identified error).

The dimension weights (the $\text{MAX}_x$ values) are:

Accuracy: 60 points (maximum)
Fluency: 20 points (maximum)
Appropriateness: 20 points (maximum)

The total maximum score of 100 decomposes into 60% accuracy, 20% fluency, and 20% appropriateness. This weighting reflects the paper's implicit prioritization: getting the factual content right is paramount, while stylistic and linguistic quality are secondary but non-trivial considerations.

Severity-based deduction schedule. The per-error deductions $w_i^x$ depend on the error's severity level, with different schedules for different dimensions:

For Accuracy errors, the deductions are:

Minor: 2 points per error
Major: 5 points per error
Critical: 10 points per error
Extremely Critical: 50 points per error

The 50-point deduction for Extremely Critical errors is calibrated to be catastrophic: since the maximum Accuracy score is 60, a single Extremely Critical error (typically an untranslated segment) drops the Accuracy score to at most 10, effectively failing the translation on that dimension regardless of other performance. This aligns with the paper's qualitative judgment that retaining source-language text without translation is a fundamental failure, not a minor issue.

For Fluency errors, all errors are treated at a uniform severity (the Fluency Judge only produces a binary "有问题" label), with a deduction of 2 points per error. The prompt does not distinguish among minor, major, and critical fluency errors—the rationale appears to be that fluency issues, while they degrade reading quality, do not vary dramatically in severity in ways that would justify differentiated penalties.

For Appropriateness errors, all errors are treated at a uniform severity (the Appropriateness Judge also produces a binary "有问题" label), with a deduction of 5 points per error. The higher per-error deduction for appropriateness (5 vs. 2 for fluency) likely reflects the paper's judgment that stylistic and cultural failures, while less common than fluency issues, represent more significant deviations from professional translation standards.

What the formula computes in operational terms: Given a source text, a translation output, and the de-duplicated error lists from the three specialist judges, the score calculator:

Starts with 60 potential points for Accuracy and subtracts the severity-weighted sum of accuracy errors.
Starts with 20 potential points for Fluency and subtracts 2 points for each fluency error.
Starts with 20 potential points for Appropriateness and subtracts 5 points for each appropriateness error.
Sums the three subscores. If any subscore would go negative, it is clamped to zero (the paper does not explicitly state this, but it is implied by the subtraction structure).

The resulting score is in the range [0, 100], with 100 representing a perfect translation and 0 representing catastrophic failure (e.g., empty output, or output that fails the instruction-following check).

Why this weighting and severity structure? The paper provides part of the rationale implicitly through the ablation study (Appendix A.2.1, Table 7). When all three dimensions are given equal weight (rather than the 60/20/20 split), average pairwise consistency with human judgments drops from 70.3% to 48.2%—a 22.1 percentage point decline. This strongly suggests that human experts weight accuracy much more heavily than fluency or appropriateness when making holistic judgments of translation quality, and the 60/20/20 split captures this weighting. The severity deduction schedule similarly reflects the empirical finding that certain errors (untranslated segments, large omissions) are viewed as categorically worse than others by human evaluators, and the linear point deductions with differentiated magnitudes provide a simple parametric form for this nonlinear severity judgment.

The paper additionally notes in Appendix E that "the details of severity levels in different domains" vary, though the specific domain-dependent adjustments are not elaborated in the main text. This suggests that the severity schedule may be tuned per-domain based on expert judgment about what constitutes a critical error in each field—for instance, a terminological inconsistency in a legal contract might be weighted more severely than in a news article.

Human Judgment Alignment Methodology

To validate that Metric-S's scores reflect genuine translation quality rather than idiosyncratic LLM preferences, the paper measures pairwise consistency between Metric-S and professional human judgments (Section 4.2, Appendix G). This methodology is adopted from the WMT 2024 metrics shared task framework (Freitag et al., 2024, cited as [6]).

Evaluation setup. The evaluation is conducted on two test sets: the DiscoX benchmark (50 randomly sampled cases) and the WMT 2024 general translation task (for comparison with reference-based metrics). For each dataset, multiple translation systems produce outputs, and both human linguistic experts and automated metrics assign scores.

Pairwise consistency framework. The core question the framework answers is: "how often does the metric agree with human experts on which of two system outputs is superior?" (Section 4.2). This is operationalized at two levels:

System-level consistency: For each pair of translation systems (A, B), the metric's ranking of A vs. B is compared against the human experts' ranking. The paper uses Soft Pairwise Accuracy (SPA), which "provide[s] a more nuanced comparison by accounting for statistical uncertainty in human rankings" (Section 4.2). SPA accounts for the fact that when two systems have very similar human scores, the human ranking between them is uncertain, and a metric should not be penalized for reversing them.

Segment-level consistency: For each pair of individual translation outputs (for the same source text), the metric's score difference is compared against the human judgment of which is better. To handle ties, "metric scores [are treated] as consistent if their difference is less than 0.05" (Section 4.2). This threshold prevents the metric from being penalized for distinguishing between translations that human evaluators consider essentially equivalent.

Score normalization. Both human and metric scores are "normalized to a [0, 1] range for fair comparison" (Section 4.2). This ensures that the consistency measurement is about relative ranking rather than absolute score calibration—a metric that systematically scores higher or lower than humans can still achieve perfect pairwise consistency if it correctly orders all pairs.

Baseline metrics. Two baseline metrics are compared, selected to represent different generations of translation evaluation:

XCOMET-QE (Guerreiro et al., 2024, cited as [10]): A state-of-the-art neural metric that operates reference-free (the "QE" suffix stands for Quality Estimation). XCOMET is trained to detect fine-grained translation errors and produces a scalar quality score.
ChrF (Popović, 2015, cited as [23]): A traditional n-gram overlap metric that requires reference translations. ChrF is evaluated only on WMT 2024 (where references exist), not on DiscoX, because DiscoX does not provide reference translations—a design choice consistent with the paper's position that discourse-level translation has no single correct output.

Results interpretation. The consistency scores reported in Table 4 (70.3% for Metric-S vs. 34.7% for XCOMET-QE on DiscoX) represent the fraction of pairwise comparisons where the metric's ranking agrees with human judgment. At the system level, 80.0% (zh→en) and 90.0% (en→zh) mean that when comparing two translation systems, Metric-S correctly identifies the better system in 8 out of 10 and 9 out of 10 cases respectively, accounting for statistical uncertainty via SPA. At the segment level, 54.8% (zh→en) and 56.4% (en→zh) mean that for individual translation pairs, Metric-S agrees with human preference in just over half of cases—substantially above chance (50% for binary choice) but far from perfect, indicating that segment-level evaluation remains a harder problem than system-level ranking.

Why this evaluation framework? The pairwise consistency approach has two key properties that make it appropriate for validating Metric-S:

It doesn't require absolute score agreement. Human evaluators and automated metrics operate on fundamentally different scales—humans make holistic judgments, while Metric-S aggregates component subscores. Requiring the metric to reproduce human absolute scores would penalize systematic scale differences that don't affect practical utility (a metric that always scores 10 points lower than humans but correctly ranks all systems is perfectly useful). Pairwise consistency measures only whether the metric's relative judgments match humans'.
It handles the no-reference setting. Since DiscoX does not provide reference translations (consistent with the paper's argument that discourse-level translation has no single reference that captures all valid outputs), reference-based metrics like ChrF and BLEU cannot be applied. The pairwise consistency framework works with any metric that produces scalar scores, reference-based or not, enabling apples-to-apples comparison between Metric-S and XCOMET-QE.

Ablation-based validation. The paper does not rely solely on the pairwise consistency numbers to validate Metric-S's design. Appendix A.2 provides a series of ablations (Table 7) that demonstrate each component's contribution:

Removing de-duplication: drops from 70.3% to 66.0%
Using only the Accuracy Judge (no Fluency or Appropriateness): drops to 62.5%
Merging all dimensions into a single judge with detailed prompt: drops to 55.9%
Equal weighting of all dimensions: drops to 48.2%
Single judge with simple prompt: drops to 24.9%

The monotonic degradation as components are removed—and particularly the catastrophic drop from 70.3% to 24.9% when the entire multi-agent workflow is collapsed to a single simple-prompt judge—provides strong evidence that Metric-S's performance is genuinely due to its architectural design rather than just the underlying capability of the judge LLM (Gemini-2.5-Pro).

Judge model robustness. Appendix A.2.2 (Table 8) further shows that Metric-S's advantage is not tied to a specific judge model. When Gemini-2.5-Pro is replaced with DeepSeek-R1 or o3-high as the judge, Metric-S's average consistency drops to 57.8% and 58.2% respectively—lower, but still dramatically higher than XCOMET-QE's 34.7%. This suggests that the structured multi-agent workflow provides benefits independent of the specific LLM used as the judge, though a stronger judge model (Gemini-2.5-Pro) amplifies those benefits.

Self-preference bias check. Appendix A.3 (Table 9) addresses a potential confound: if the judge model prefers its own translations, the consistency numbers might reflect self-bias rather than genuine quality assessment. The paper tests Gemini-2.5-Pro, DeepSeek-R1, and o3-high as judges on a set of translations including each judge model's own output. Gemini-2.5-Pro shows no self-preference (its ranking correlates with human judgments, not with which model produced the output). In contrast, o3-high ranks its own outputs first even though humans rank it third. This finding is cited as the rationale for selecting Gemini-2.5-Pro as the default judge model—it provides the strongest absolute performance and the least self-preference bias.

Fine-grained accuracy validation. Beyond pairwise consistency at the score level, Appendix A.5 (Table 11) evaluates whether the LLM judges' individual error identifications are correct. For 250 cases (50 texts × 5 judge models), linguistic experts manually verify each error type and severity judgment. The Accuracy Judge produces correct assessments in 98.40% of cases, the Fluency Judge in 95.20%, and the Appropriateness Judge in 96.40%. These numbers indicate that when Metric-S flags an error, it is almost certainly a genuine error—the metric's imperfection lies more in missing errors (false negatives) than in inventing them (false positives), which is the safer direction for an evaluation tool meant to provide actionable feedback.

4. Key Insights and Innovations

Innovation 1: Difficulty-Filtered Benchmark Construction as a Diagnostic Methodology

The dominant paradigm in translation benchmark construction is representative sampling: collect a diverse set of texts from target domains, ensure they're professionally translated, and evaluate models against this sample. FLORES, WMT, and Redtrans Bench all follow this pattern—they aim to measure how well models perform on average across a distribution. Performance numbers from these benchmarks answer the question: "If you deploy this model on typical texts from this domain, how often will it produce acceptable translations?"

DiscoX makes a fundamentally different methodological move: it constructs a benchmark by deliberately selecting for difficulty. The three-stage pipeline doesn't just filter for quality—it applies a stringent difficulty threshold where candidate texts are retained only if both of two state-of-the-art LLMs fail on at least eight predefined rubrics (Section 2.1). This is an inversion of standard benchmarking logic. Rather than asking "how good is the average translation?", DiscoX asks "where do models still systematically fail?"

This is not a minor tweak to existing methodology—it's a different epistemological stance about what benchmarks are for. Representative benchmarks measure current capability; difficulty-filtered benchmarks measure capability ceilings. The 30% selection rate (200 texts retained from 665 candidates) makes this explicit: DiscoX is discarding the tasks where models already succeed in order to concentrate evaluation resources on the frontier of failure.

The significance of this choice extends beyond the specific benchmark. It introduces a generalizable design principle: for capabilities that are economically critical but where existing benchmarks are approaching saturation (as the paper argues sentence-level translation benchmarks are), deliberately difficulty-filtered test sets provide more diagnostic signal per evaluation dollar than representative samples. This principle—benchmarks as stress tests rather than surveys—has been applied in other NLP subfields (adversarial datasets, challenge sets), but DiscoX represents its first systematic application to professional translation evaluation, where the stakes of underestimating capability gaps are particularly high.

The evidence that this methodological choice matters is in the results it enables. Table 3 shows that even GPT-5-high, the strongest tested model, scores 76.66 against a human baseline of 80.16—a meaningful gap that would be invisible on a benchmark where models already approach human parity. More importantly, the difficulty filtering means that DiscoX's failure patterns are informative: they reveal specific capability gaps (Section 5.1–5.3) rather than just reporting that models are "not quite as good as humans." The finding that thinking models systematically underperform non-thinking counterparts (Table 5), for instance, emerges because DiscoX contains tasks where the thinking process leads to over-summarization—a failure mode that representative benchmarks, with their easier texts, would not expose.

The cost side of this methodology is significant: 1,330 person-hours from 133 professionals to produce 200 test cases. This is roughly an order of magnitude more expensive per test case than representative sampling approaches. But the paper implicitly argues—convincingly, given the massive economic stakes of professional translation quality—that this cost is justified when the alternative is benchmarks that declare victory prematurely and allow systematic failures to go undetected.

Innovation 2: Error De-Duplication as a Necessary Component of Multi-Dimensional Evaluation

Multi-dimensional evaluation is not new in machine translation. The Multidimensional Quality Metrics (MQM) framework, COMET's error-span annotations, and various taxonomies have long recognized that translation quality has distinct facets. What is new in Metric-S is the recognition that multi-dimensional evaluation creates a specific technical problem— error double-counting from dimension interactions—and that addressing this problem requires a dedicated architectural component rather than ad-hoc post-processing.

The insight is structural: when you decompose evaluation into independent dimensions (Accuracy, Fluency, Appropriateness), you create a situation where a single translator error can manifest as symptoms in multiple dimensions simultaneously. A mistranslated term (Accuracy) produces awkward phrasing (Fluency) and disrupts stylistic register (Appropriateness). If each dimension's judge operates independently—as they must, to preserve specialization—the same root error gets counted multiple times unless there's an explicit mechanism to trace derivative errors back to their causes.

The paper's key conceptual move is recognizing that this isn't a bug to be tolerated but a first-class design problem that the evaluation architecture must solve. The field's prior approaches to multi-dimensional evaluation either (a) had a single judge assess all dimensions simultaneously (which the paper shows produces poor results: 55.9% consistency for a merged-prompt single judge vs. 70.3% for the full workflow, Table 7), (b) accepted double-counting as an acceptable imprecision (implicitly treating dimension scores as independent when they aren't), or (c) relied on human evaluators who naturally perform root-cause attribution as part of holistic judgment.

Metric-S's De-duplication Judge (Section 3.3, Appendix C.2.3) operationalizes this insight through a hierarchical priority system: Extremely Critical Accuracy errors override all overlaps, rubric violations are attributed to Accuracy as the second priority, and remaining overlaps are resolved through causal analysis. The worked example in Appendix C.4 makes the logic concrete: when three judges independently flag the same omitted sentence as an Accuracy omission, a Fluency coherence problem, and an Appropriateness style violation, the De-duplication Judge identifies content omission as the root cause and retains only the Accuracy error.

The ablation evidence (Table 7) confirms that de-duplication matters but also reveals something subtler: removing it drops consistency from 70.3% to 66.0%, a 4.3 percentage point decline. This is meaningful but not catastrophic, which suggests that the primary driver of Metric-S's performance is the three-judge decomposition itself, with de-duplication providing an important but secondary correction. This is consistent with the paper's framing: de-duplication solves a real problem introduced by multi-dimensional evaluation, but the decomposition creates more value than the double-counting problem subtracts.

The broader significance is that Metric-S establishes de-duplication as a design requirement for any multi-agent evaluation system. As LLM-as-a-judge approaches become more common and more complex (multiple judges, multiple dimensions, multiple rounds of refinement), the interaction between evaluators will increasingly produce correlated errors that need systematic resolution. Metric-S provides a template for how to architect this resolution: a dedicated attribution agent with explicit priority rules and causal reasoning.

Innovation 3: The Complementary-Strengths Diagnosis as a Benchmarking Output

Standard translation benchmarks produce a single ranking—Model A is better than Model B. Even when they report dimension-level scores (as WMT does with its MQM annotations), these are typically presented as diagnostic supplements to the primary ranking. The conceptual contribution of DiscoX's results analysis is that the pattern of dimension-level performance across models is itself a primary finding, not a secondary detail.

This emerges most clearly from Table 3, but its significance only becomes apparent when you read the dimension columns across rows rather than within them. GPT-5-high leads in Accuracy (48.65) and Appropriateness (12.80). Kimi-K2 leads in Fluency (16.44) but scores only 27.63 on Accuracy. Claude-4 achieves an Accuracy of 39.38—second only to GPT-5-high—but a Fluency of just 5.98, producing translations that are factually accurate but barely readable. DeepSeek-V3 leads in Fluency (16.20) but scores a mediocre 22.80 on Accuracy. No model achieves balanced excellence across all three dimensions.

The insight is that these aren't just "different models have different strengths"—that's true of nearly every benchmark. What's distinctive here is that the strengths are genuinely complementary rather than correlated. Fluency and Accuracy are sufficiently independent in discourse-level translation that a model can be state-of-the-art on one dimension while being substantially below average on another. For sentence-level translation, accuracy and fluency are more tightly coupled (a mistranslation usually produces an ungrammatical sentence, and a grammatical sentence is usually accurate), so dimension-level analysis reveals less. At discourse scale, the coupling breaks: a model can produce fluent, natural-sounding prose that systematically omits or fabricates domain-specific content, or accurate but translationese-ridden output that preserves terminology while violating target-language norms.

This finding has practical implications that a single-score ranking would obscure. A user choosing between GPT-5-high and Kimi-K2 faces a genuine tradeoff—accuracy vs. fluency—that depends on their use case. A scientific journal editing translations might prefer GPT-5-high's accuracy; a literary publisher might value Kimi-K2's fluency more. The dimension-level results make this tradeoff visible and quantifiable in a way that "GPT-5-high is 5 points better than Kimi-K2" does not.

The paper also uses this diagnostic lens to analyze domain and direction asymmetries (Section 5.1–5.2). The finding that all models are stronger on zh→en than en→zh—with DeepSeek-V3 showing a 34.8-point gap—is not just a performance observation but a diagnostic signal about underlying capability asymmetries: the models have systematically better English generation than Chinese generation, likely due to training data imbalances. The academic-vs-literary performance gap similarly diagnoses a specific capability limitation: models handle structured, logical text better than texts requiring cultural nuance and stylistic sensitivity.

This reframes what a benchmark's primary output should be. Rather than a leaderboard that collapses all dimensions into a single ranking, DiscoX + Metric-S argue for diagnostic profiling—a multi-dimensional score that reveals capability structure, not just capability level. The paper's leaderboard (Figure 1) partially reflects this by showing the stacked dimension scores, though the overall ranking is still dominated by the Accuracy-weighted total. A fully realized diagnostic benchmark would foreground the dimension profiles and treat the overall score as a convenient summary rather than the primary metric.

Innovation 4: The Thinking-Model Translation Deficit as a Cautionary Empirical Finding

Section 5.3 reports a result that is counterintuitive enough to qualify as a genuine empirical contribution: thinking-enhanced (chain-of-thought) models systematically underperform their non-thinking counterparts on discourse-level translation. Qwen-3-235B's thinking variant scores 49.97 vs. 59.66 for the non-thinking variant—a 9.69-point gap driven primarily by a 9.95-point Accuracy deficit. Claude-4 shows a smaller but consistent gap (53.53 thinking vs. 54.03 non-thinking).

This is a negative result with positive implications. The dominant narrative around reasoning models (o1, o3, DeepSeek-R1, Qwen-3-thinking) is that chain-of-thought reasoning improves performance on complex tasks requiring multi-step inference. Translation, particularly at discourse scale with domain-specific requirements, would seem to fit this profile: it requires reasoning about terminology choices, maintaining consistency across sections, and adapting register to context. The intuitive expectation would be that thinking models perform better on DiscoX than their non-thinking counterparts.

The paper's error analysis identifies why the opposite occurs: thinking models "either over-summarize the source text, leading to omissions (under-translation), or introduce extraneous structural content (over-translation), thereby compromising translation faithfulness" (Section 5.3). In operational terms, the chain-of-thought process—which is designed to help models plan complex outputs—appears to interfere with the specific requirement of translation, which is faithful reproduction rather than reasoned generation. The thinking process may cause the model to treat the source text as a prompt to be analyzed and re-expressed rather than a template to be faithfully rendered, leading it to summarize, reorganize, or elaborate rather than translate.

This finding is significant beyond translation specifically because it identifies a capability-interference effect in reasoning models: the same mechanism that improves performance on math, coding, and logical reasoning appears to degrade performance on faithful reproduction tasks. If this effect generalizes—if thinking models also underperform on other tasks requiring strict fidelity to source material (document summarization with low hallucination, faithful paraphrasing, instruction-following with exact constraints)—it suggests that reasoning capabilities are not uniformly beneficial and that model selection should be task-dependent in ways the current "use the thinking model for everything hard" heuristic misses.

The paper doesn't deeply investigate why this interference occurs, and the sample size (two model pairs) is small. But the empirical pattern is clear enough—and contrary enough to prevailing assumptions—that it constitutes a genuine finding rather than noise. It also suggests a concrete research direction: can thinking models be trained or prompted to suppress their tendency to reorganize content when the task demands faithful reproduction? Or is the interference structural to how chain-of-thought reasoning interacts with output generation?

Innovation 5: Rubric-Criterion-Referenced Evaluation as a Bridge Between Human Expertise and Automated Metrics

The deepest conceptual tension in machine translation evaluation is between reference-based metrics (which are reproducible but miss valid variation) and human judgment (which captures quality holistically but is expensive and inconsistent). Recent reference-free neural metrics like XCOMET-QE attempt to bridge this gap by learning quality predictors from human annotations, but they remain black boxes—they produce a score without explaining what errors caused it.

Metric-S introduces a third paradigm that hasn't been fully articulated in prior MT evaluation work: rubric-criterion-referenced evaluation. The expert-authored rubrics from DiscoX's construction pipeline (Section 2.1) serve as explicit, verifiable evaluation criteria that are fed directly to the Accuracy Judge (Section 3.2). A rubric like "'hold harmless and defend' must be translated as '使……免受损害，并为其进行抗辩'; omitting 'defend' is an error" is not a reference translation—it's a criterion that specifies a necessary condition for correctness on a specific domain-critical element. The judge doesn't compare the full translation against a reference; it checks whether the specific criterion is satisfied.

What makes this innovative is the division of evaluation labor between humans and machines. Humans (the 115 vertical domain experts) contribute what they're uniquely good at: identifying what matters in a translation and specifying criteria that capture domain-specific requirements. Machines (the LLM judge) contribute what they're good at: systematically checking large numbers of criteria against long texts without fatigue or inconsistency. This is fundamentally different from both (a) fully manual evaluation (expensive, inconsistent at scale) and (b) fully automated evaluation (misses domain-specific requirements that aren't learnable from surface patterns).

The evidence that this approach works is in the fine-grained accuracy numbers (Appendix A.5, Table 11): the Accuracy Judge correctly identifies rubric-related errors in 98.40% of cases. This near-perfect performance on criterion-referenced checks contrasts with the more modest segment-level pairwise consistency (54.8–56.4%), suggesting that rubric checks—where the judge has an explicit target to verify—are substantially more reliable than open-ended quality assessment.

The broader significance is that rubric-criterion-referenced evaluation provides a practical framework for domain experts to encode their knowledge into automated evaluation pipelines without needing to be ML engineers. The rubrics are written in natural language, authored by domain experts following their professional judgment, and consumed by LLMs that can interpret natural language criteria. This dramatically lowers the barrier to creating domain-specific evaluation benchmarks: a legal translation expert doesn't need to train a classifier; they just need to write down what terms and constructions must be handled correctly. The scaling implication is that this approach could be replicated across arbitrary domains (medical, financial, technical) at a cost proportional to expert time rather than ML engineering time.

This insight also clarifies why Metric-S dramatically outperforms XCOMET-QE on DiscoX (70.3% vs. 34.7%, Table 4): XCOMET-QE, like all trained metrics, learns to predict quality from patterns in its training data. If its training data doesn't include discourse-level texts with domain-specific rubric-style annotations, it has no way to learn that "ETV6-NTRK3 gene rearrangement must be translated with specific molecular biology terminology." Metric-S doesn't need to learn this—it reads it from the rubric. The rubric mechanism makes domain knowledge an input to evaluation rather than something that must be implicitly captured in training, which is the key architectural difference that enables Metric-S's strong performance on DiscoX's deliberately difficult, domain-intensive texts.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the DiscoX benchmark—200 professionally-curated Chinese-English translation tasks spanning 7 domains (4 Academic, 3 Non-Academic), with an average input length of 1,712.17 tokens (Section 2.2, Table 2). The benchmark is balanced across Academic (121 tasks) and Non-Academic (79 tasks) primary domains, with 100 tasks each for en→zh and zh→en translation directions. Each task includes expert-authored rubrics specifying between 1 and 20+ domain-critical checkpoints that translations must handle correctly.
Base model(s). The evaluation covers 20 translation systems: 7 open-source LLMs (Qwen-3-235B, Qwen-3-235B-T, Qwen-3-14B, Qwen-3-8B, Kimi-K2, DeepSeek-V3, DeepSeek-R1), 11 closed-source LLMs (GPT-5-high, GPT-4.1, GPT-4o, Gemini-2.5-Pro, Gemini-2.5-Flash-Lite, Claude-4, Claude-4-T, o3-high, o4-mini-high, Doubao-1.6-T, Grok-4), 1 domain-specific LLM (Youdao-14B), and 1 neural machine translation system (Google-NMT). The paper states these models were chosen to represent "a broad set of LLMs" spanning different architectures, scales, and training paradigms (Section 4.1), though specific model sizes (parameter counts) are not reported for most models. The human expert baseline is a translation produced by a professional human translator evaluated under the same Metric-S framework.
Metrics. The primary metric is the DiscoX Score (0–100), computed by Metric-S as the sum of Accuracy (max 60), Fluency (max 20), and Appropriateness (max 20) subscores, with severity-weighted error deductions. For metric validation, the paper uses pairwise consistency with human judgments (Section 4.2), operationalized as the fraction of pairwise system or segment comparisons where Metric-S agrees with professional human evaluators on which output is superior. System-level consistency uses Soft Pairwise Accuracy (SPA) to account for statistical uncertainty in human rankings; segment-level consistency treats metric scores as consistent if their difference is less than 0.05 (Appendix G). Other metrics used for comparison include XCOMET-QE (reference-free neural metric) and ChrF (n-gram overlap, reference-based, evaluated only on WMT 2024).
Baselines. Two metric baselines are evaluated: XCOMET-QE (Guerreiro et al., 2024, cited as [10]), a state-of-the-art reference-free neural metric for translation quality estimation, and ChrF (Popović, 2015, cited as [23]), a traditional character n-gram overlap metric that requires reference translations and is evaluated only on the WMT 2024 general translation task. For the main translation results (Table 3), the baselines are the 19 non-human systems compared against the Human Expert reference point (score of 80.16). No traditional MT system is presented as a primary baseline in the main results, though Google-NMT (37.10) and Youdao-14B (46.37) are included among the 20 evaluated systems.
Generation budget / compute accounting. The paper does not report generation budgets in the conventional sense (number of tokens, FLOPs, or inference time). Each model is prompted once per DiscoX task (200 tasks × 1 generation each = 200 translations per model). The system prompt is standardized across all models: "Translate from Source Language to Target Language. Do not output content beyond the translation" (Appendix C.1). No sampling with multiple seeds, beam search, or best-of-N is performed; results represent single-pass translation quality. The paper notes that models process the full discourse-level text in a single generation, with no segment-by-segment processing.
Cross-validation / statistical protocol. For the main translation results (Table 3), each model's score is the average across 200 tasks, with no cross-validation or statistical significance testing reported in the main text. For stability analysis, three independent runs were performed for each of six representative models on all 200 tasks (Appendix A.1, Table 6), with standard deviations ranging from 0.21 (GPT-5-high) to 1.36 (Qwen-3-235B), indicating high output stability. For metric validation (Table 4), the paper randomly samples 50 cases from DiscoX and 50 cases from WMT 2024, with specific adjustments for different evaluation levels as described above. For the pairwise consistency framework, the paper follows the WMT 2024 metrics shared task methodology (Freitag et al., 2024, cited as [6]). The fine-grained accuracy validation (Appendix A.5, Table 11) uses 250 cases (50 texts × 5 judge models), with linguistic experts manually verifying error type and severity labels.

Main Quantitative Results

Overall Model Rankings on DiscoX

The headline result is presented in Table 3: even the strongest LLM (GPT-5-high at 76.66) trails the human expert baseline of 80.16 by 3.50 points, while the second-ranked model (Gemini-2.5-Pro) falls to 71.25—an 8.91-point gap below human performance. The full ranking reveals a steep performance hierarchy:

GPT-5-high (76.66) leads across Accuracy (48.65) and Appropriateness (12.80), establishing it as the strongest overall model.
Gemini-2.5-Pro (71.25) ranks second with balanced performance (Accuracy: 46.68, Fluency: 13.14, Appropriateness: 11.43).
Qwen-3-235B (59.66) is the highest-ranking open-source model, substantially ahead of the next open-source system (Kimi-K2 at 55.80).
A large cluster of models scores in the 49–56 range: Kimi-K2 (55.80), o3-high (55.57), o4-mini-high (55.09), Claude-4 (54.04), Claude-4-T (53.53), Qwen-3-235B-T (49.97), GPT-4.1 (49.65), DeepSeek-V3 (49.60), Doubao-1.6-T (49.51).
Below this cluster, performance drops to the 37–46 range: DeepSeek-R1 (46.06), Youdao-14B (46.37), Gemini-2.5-Flash-Lite (44.01), Grok-4 (43.82), GPT-4o (39.93), Qwen-3-14B (39.36), Google-NMT (37.10).
Qwen-3-8B (28.37) is the weakest evaluated model, underscoring a substantial scale effect.

The domain-specific Youdao-14B (46.37) and the traditional NMT system Google-NMT (37.10) both trail all large general-purpose LLMs, confirming the paper's claim that "general-purpose LLMs significantly outperform traditional MT models and domain-specific LLMs in discourse-level translation tasks" (Section 5.4). The gap between the best open-source model (Qwen-3-235B at 59.66) and the best closed-source model (GPT-5-high at 76.66) is 17.00 points, suggesting that frontier closed-source models retain a substantial advantage on this task.

Multi-Dimensional Performance Profiles: Complementary Strengths, Not Universal Competence

The dimension-level breakdown in Table 3 reveals that no model achieves balanced excellence across all three dimensions. This is most striking when comparing the top-ranked models:

Accuracy (max 60): GPT-5-high leads at 48.65, followed by Gemini-2.5-Pro (46.68) and Claude-4 (39.38). The drop from second to third place (7.30 points) is nearly as large as the gap from third to the cluster of models around 28–33 (Qwen-3-235B at 33.15, Grok-4 at 31.38, o4-mini-high at 29.55, Doubao-1.6-T at 29.30, GPT-4.1 at 29.25). This steep Accuracy gradient means that most models—even strong ones—are losing 30–40 points (out of 60) on accuracy alone, primarily through domain-critical terminology errors, omissions, and rubric violations.

Fluency (max 20): The Fluency scores show an inverted pattern. Kimi-K2 leads at 16.44, followed by DeepSeek-V3 (16.20) and DeepSeek-R1 (16.11)—all models that rank substantially lower overall (6th, 12th, and 14th respectively). GPT-5-high scores 15.21 on Fluency (4th place), while Claude-4 scores just 5.98—the worst Fluency among all models except Qwen-3-8B (5.84). Claude-4's profile (Accuracy 39.38, Fluency 5.98) represents the most extreme divergence: its translations are factually accurate but barely readable, suggesting a model that prioritizes terminological precision over natural target-language expression.

Appropriateness (max 20): GPT-5-high leads at 12.80, with Kimi-K2 (11.73), Qwen-3-235B (11.55), and Gemini-2.5-Pro (11.43) close behind. No model exceeds 14.40 (the human score), and most models cluster in the 7–12 range, suggesting that stylistic, emotional, and cultural dimensions remain difficult for all current systems.

The practical implication is that model selection for professional translation should be use-case-dependent: a scientific publisher prioritizing factual accuracy might choose GPT-5-high; a literary publisher valuing fluency might prefer Kimi-K2; a legal firm requiring balanced performance might select Gemini-2.5-Pro. A single-score ranking (Overall column) obscures these tradeoffs.

Translation Direction Asymmetry: zh→en Substantially Outperforms en→zh

Figure 5 (subplot a) and Appendix H reveal a systematic performance gap between translation directions, with all models scoring higher on Chinese-to-English than on English-to-Chinese tasks. The paper reports specific numbers for individual models:

DeepSeek-V3 shows the most extreme asymmetry: a 34.8-point gap between zh→en and en→zh performance. This is by far the largest directional disparity, consistent with the paper's characterization of DeepSeek-V3 as an English-centric model.
Doubao-1.6-T is the most balanced model, with a gap of only 7.2 points.
Most models show gaps in the 15–25 point range, though the paper does not provide a full table of per-model directional differences.

The paper attributes this asymmetry to three factors (Section 5.1): (1) data imbalance favoring English corpora in pretraining, (2) English-centric model architectures requiring greater adaptation for Chinese generation, and (3) inherent linguistic complexities of Chinese (rich morphology, implicit logic, reliance on word order and context). The implication is that evaluating models only on zh→en (the easier direction) would systematically overestimate their translation capability—a finding that matters for production deployments where en→zh is equally important.

Domain-Based Performance Disparity: Academic > Non-Academic

Figure 5 (subplot b) and Appendix I show that models consistently perform better on academic texts than on non-academic texts, with the gap being most pronounced for literary and artistic content. Specific numbers:

GPT-5-high leads in both domains, demonstrating the strongest generalizability.
Kimi-K2 shows a particular strength in literary translation, consistent with its fluency-focused design (it leads in Fluency at 16.44 overall).
The academic advantage is attributed to structured, logical formats being "more amenable to accurate machine translation," while literary works demand "complex syntax, rhetorical devices, and cultural nuances" that current models struggle to reproduce (Section 5.2).

This domain effect means that aggregate DiscoX scores mask substantial performance variation: a model scoring 55 overall might be at 65 on academic papers and 40 on literature, depending on its architecture. For deployment in specific domains (e.g., a legal translation service vs. a literary publishing house), the domain-level breakdown is more informative than the aggregate score.

Thinking vs. Non-Thinking Models: A Systematic Performance Deficit

Table 5 presents one of the paper's most counterintuitive findings: thinking-enhanced (chain-of-thought) models underperform their non-thinking counterparts on discourse-level translation. The comparison is available for two model pairs:

Qwen-3-235B: The thinking variant (Qwen-3-235B-T) scores 49.97, while the non-thinking variant scores 59.66—a gap of 9.69 points driven almost entirely by a 9.95-point Accuracy deficit (23.20 vs. 33.15). Fluency is nearly identical (15.54 vs. 14.96), and Appropriateness is similar (11.23 vs. 11.55).
Claude-4: The thinking variant (Claude-4-T) scores 53.53, while the non-thinking variant scores 54.03—a smaller gap of 0.50 points. The difference is concentrated in Fluency (5.47 vs. 5.98) rather than Accuracy (38.98 vs. 39.38).

The paper's error analysis identifies the mechanism: thinking models "either over-summarize the source text, leading to omissions (under-translation), or introduce extraneous structural content (over-translation), thereby compromising translation faithfulness" (Section 5.3). The chain-of-thought reasoning process, designed for complex multi-step inference, appears to interfere with the specific requirement of faithful reproduction, causing models to treat the source as a prompt to be analyzed and re-expressed rather than a template to be rendered.

This finding has practical importance beyond translation: it suggests that reasoning capabilities are not uniformly beneficial and that thinking models may underperform on tasks requiring strict fidelity to source content. The paper does not investigate whether this deficit can be mitigated through prompting (e.g., instructing the thinking model to suppress summarization tendencies), which is a notable open question.

Metric-S Validation Against Human Judgments

Table 4 presents the core metric validation results on DiscoX, establishing that Metric-S achieves 70.3% overall average pairwise consistency with human judgments, more than doubling XCOMET-QE's 34.7%. The breakdown reveals:

System-level consistency:

zh→en: Metric-S achieves 80.0%, while XCOMET-QE scores just 10.0%—a 70-percentage-point gap indicating that XCOMET-QE is essentially random at distinguishing better from worse Chinese-to-English translation systems on discourse-level texts.
en→zh: Metric-S achieves 90.0%, XCOMET-QE achieves 70.0%—a more modest but still substantial 20-point gap.

Segment-level consistency:

zh→en: Metric-S achieves 54.8%, XCOMET-QE achieves 26.4%.
en→zh: Metric-S achieves 56.4%, XCOMET-QE achieves 32.4%.

The segment-level numbers are substantially lower than system-level numbers for both metrics, reflecting the intrinsic difficulty of judging individual translation pairs. However, Metric-S's segment-level consistency (54.8–56.4%) remains well above chance (50%), while XCOMET-QE's segment-level performance (26.4–32.4%) is below chance—meaning XCOMET-QE's segment-level judgments are systematically anti-correlated with human preferences on DiscoX.

The paper also validates Metric-S on the WMT 2024 en→zh general translation task (Appendix A.4, Table 10), where reference-based metrics can be compared. Metric-S achieves 72.30% overall average consistency (system-level 90.0%, segment-level 54.6%), comparable to the reference-based XCOMET at 68.80% (system-level 80.0%, segment-level 57.6%) and outperforming ChrF at 55.70% (system-level 60.0%, segment-level 51.4%). This result demonstrates that Metric-S, despite being reference-free, matches or exceeds reference-based metrics even on general-domain tasks—and dramatically outperforms them on discourse-level expert-domain tasks where reference-based approaches break down.

Ablation Studies and Robustness Checks

Multiple runs stability: The paper performs three independent runs for each of six representative models (GPT-5-high, Gemini-2.5-Pro, Qwen-3-235B, Qwen-3-235B-T, Claude-4, Claude-4-T) on all 200 DiscoX tasks (Appendix A.1, Table 6). Standard deviations range from 0.21 (GPT-5-high) to 1.36 (Qwen-3-235B), with mean scores differing from single-run scores by less than 2 points in all cases. This indicates that model outputs are stable across generations and that the reported scores are not artifacts of sampling noise.

Metric-S component ablation: The paper systematically removes components of the Metric-S workflow and measures the impact on pairwise consistency with human judgments (Appendix A.2.1, Table 7). All experiments use Gemini-2.5-Pro as the judge model:

Metric-S (Original): 70.3% average consistency (system-level 85.0%, segment-level 55.6%).
Removing error de-duplication: Drops to 66.0% (system-level 80.0%, segment-level 52.0%)—a 4.3-point decline. The de-duplication mechanism provides a measurable but secondary benefit; the primary performance driver is the three-judge decomposition.
Using only the Accuracy Judge (no Fluency or Appropriateness): Drops to 62.5% (system-level 70.0%, segment-level 55.0%). The segment-level consistency is nearly unchanged (55.0% vs. 55.6%), suggesting that the Fluency and Appropriateness judges primarily improve system-level discrimination rather than individual segment judgments.
Single LLM judge with detailed prompt (all dimensions merged): Drops to 55.9% (system-level 60.0%, segment-level 51.8%). The 14.4-point decline from the full workflow demonstrates that merging evaluation dimensions into a single prompt substantially degrades performance, even with a detailed prompt.
Equal weighting of all three dimensions (instead of 60/20/20): Drops to 48.2% (system-level 50.0%, segment-level 46.4%). This 22.1-point decline is the largest single ablation effect, confirming that the 60/20/20 accuracy-weighted split closely matches human evaluators' implicit prioritization of factual correctness over fluency and style.
Single LLM judge with a simple prompt: Drops to 24.9% (system-level 20.0%, segment-level 29.8%)—a catastrophic 45.4-point decline that is well below chance at the system level. This establishes a lower bound: a naive LLM-as-a-judge approach is actively worse than random on this task.

Judge model robustness: The paper replaces Gemini-2.5-Pro with alternative judge models and remeasures Metric-S consistency (Appendix A.2.2, Table 8):

Metric-S (Gemini-2.5-Pro): 70.3% average (system-level 85.0%, segment-level 56.0%).
Metric-S (DeepSeek-R1): 57.8% average (system-level 70.0%, segment-level 45.6%).
Metric-S (o3-high): 58.2% average (system-level 70.0%, segment-level 46.4%).
XCOMET-QE (baseline for comparison): 34.7% average (system-level 40.0%, segment-level 29.4%).

The key finding: even with substantially weaker judge models (DeepSeek-R1 or o3-high), Metric-S still dramatically outperforms XCOMET-QE (57.8–58.2% vs. 34.7%). The absolute performance drop is expected—a weaker judge produces less reliable evaluations—but the relative advantage of the workflow architecture persists across judge models. This is strong evidence that Metric-S's performance is driven by its multi-agent design, not merely by using a particularly capable LLM as the judge.

Self-preference bias check: The paper tests whether judge models exhibit bias toward their own translations (Appendix A.3, Table 9). For a set of translations including outputs from the judge models themselves:

Gemini-2.5-Pro: Shows no self-preference. Its ranking correlates with human judgments, and it does not rank its own outputs higher than human evaluators do.
o3-high: Shows pronounced self-preference bias. As a judge, o3-high ranks its own outputs first (score of 77.54), while human experts rank it third (score of 51.70). It also inflates scores for other models substantially (e.g., Gemini-2.5-Pro gets 74.96 from o3-high vs. 69.66 from Gemini-2.5-Pro as judge).
DeepSeek-R1: Shows moderate score inflation but less severe ranking distortion than o3-high.

These findings justify the choice of Gemini-2.5-Pro as the default judge model (it combines the highest absolute consistency with the lowest self-preference bias) and warn that using a model as both a translator and a judge can produce systematically misleading evaluations.

Fine-grained judge accuracy: Beyond pairwise consistency at the score level, the paper evaluates whether individual error identifications by the LLM judges are correct (Appendix A.5, Table 11). For 250 cases (50 texts × 5 judge models), linguistic experts manually verify error types and severity labels:

Accuracy Judge: 246 correct out of 250 (98.40%).
Fluency Judge: 238 correct out of 250 (95.20%).
Appropriateness Judge: 241 correct out of 250 (96.40%).

These near-perfect accuracy rates indicate that when Metric-S's judges flag an error, it is almost certainly a genuine error. The imperfect segment-level pairwise consistency (54.8–56.4%) despite near-perfect error detection suggests that Metric-S's main limitation is false negatives (missing errors that humans would catch) rather than false positives (flagging non-errors). This is the safer failure mode for an evaluation tool: it may be slightly lenient, but it rarely penalizes models for things that aren't actually wrong.

Critical Assessment

Claim 1: "Current LLMs lag behind professional human translators on discourse-level, expert-domain translation"

Supported, with a specific empirical basis. The claim that GPT-5-high scores 76.66 against a human baseline of 80.16 (Table 3) is the headline evidence. The 3.50-point gap is meaningful but modest—it suggests that the best LLMs are approaching but not yet matching professional human performance on this specific benchmark. Several qualifications apply:

First, the human baseline is a single expert translation, evaluated by Metric-S. The paper does not report inter-annotator agreement among multiple human translators, nor does it provide confidence intervals on the human score. It is possible that different professional translators would produce translations scoring anywhere from 75 to 85 on Metric-S, which would change the interpretation of the 3.50-point gap. The claim would be stronger if the paper reported human performance as a distribution (e.g., mean and standard deviation across multiple translators) rather than a point estimate.

Second, the gap is driven primarily by Appropriateness (Human: 14.40, GPT-5-high: 12.80, a 1.60-point gap) and Fluency (Human: 15.96, GPT-5-high: 15.21, a 0.75-point gap). Accuracy is nearly at human level (Human: 49.80, GPT-5-high: 48.65, a 1.15-point gap). This means the remaining capability gap is concentrated in the dimensions least emphasized by Metric-S's weighting scheme (Fluency and Appropriateness together account for only 40 of 100 points). A user who values accuracy above all else might reasonably conclude that GPT-5-high is already at professional quality; a literary publisher who values stylistic fidelity would reach the opposite conclusion. The "gap" is dimension-dependent.

Third, the claim generalizes to other models but the magnitude varies enormously. Gemini-2.5-Pro at 71.25 trails GPT-5-high by 5.41 points, and all other models trail by 20+ points. The statement "current LLMs lag behind professional human translators" is true for every model tested, but the practical significance ranges from "nearly there" (GPT-5-high) to "nowhere close" (Qwen-3-8B at 28.37, a 51.79-point gap). The paper could strengthen this claim by characterizing which models are approaching professional quality and on which dimensions, rather than reporting a single aggregate number.

Claim 2: "Metric-S demonstrates strong consistency with human judgments, significantly outperforming existing metrics"

Supported, with important qualifiers about what "strong consistency" means in practice. The 70.3% overall pairwise consistency on DiscoX (Table 4) is genuinely strong compared to XCOMET-QE's 34.7%, and the ablation evidence (Table 7) convincingly demonstrates that this advantage stems from Metric-S's multi-agent architecture rather than the underlying judge model. However:

System-level consistency is high; segment-level consistency is modest. Metric-S achieves 80–90% at correctly ranking systems but only 54.8–56.4% at correctly ranking individual translation pairs. The segment-level numbers are better than chance (50%) but far from ceiling, meaning that Metric-S is reliable for comparing systems but significantly less reliable for comparing individual translations. This matters for use cases like "which of these two translations of this specific document is better?"—Metric-S's answer is correct only slightly more often than a coin flip. The paper acknowledges this implicitly by reporting both levels, but the framing as "strong consistency" should be understood as primarily a system-level claim.

The validation sample is small and stratified. The metric validation uses 50 randomly sampled cases from DiscoX (25% of the benchmark). For segment-level evaluation, this means 50 pairwise comparisons per direction—a sample size where the standard error on a proportion near 55% is approximately 7 percentage points. The true segment-level consistency could plausibly be anywhere from 48% to 62% at a 95% confidence level. The paper does not report confidence intervals, which would make the precision of these estimates clearer.

The paper does not test whether Metric-S's advantage persists when evaluating non-DiscoX discourse-level texts. The WMT 2024 validation (Appendix A.4, Table 10) shows Metric-S achieving 72.30% consistency on general-domain sentence-level translation, which is promising but does not directly test whether the rubric mechanism—a key differentiator from XCOMET-QE—generalizes to expert-domain texts outside DiscoX where no pre-authored rubrics exist. A stronger validation would include human correlation on a held-out expert-domain test set not used in the 50-case validation sample.

Claim 3: "Performance is imbalanced across dimensions, revealing complementary strengths and weaknesses"

Strongly supported, and this is the paper's most robust contribution beyond the benchmark itself. The dimension-level profiles in Table 3 (Claude-4: Accuracy 39.38, Fluency 5.98; Kimi-K2: Accuracy 27.63, Fluency 16.44; DeepSeek-V3: Accuracy 22.80, Fluency 16.20) are not subtle effects—they are stark divergences that no single-score metric would reveal. The paper demonstrates that these profiles are not noise through the multiple-runs stability analysis (Table 6), which shows standard deviations of 0.21–1.36 points across three runs, far smaller than the 10+ point gaps between dimensions within individual models.

The main limitation is that the paper does not investigate why these profiles diverge so dramatically. Are Claude-4's fluency problems due to training data composition? Architecture? RLHF tuning that over-optimizes for factual accuracy? The profiles are diagnostic but not explanatory—they tell you that models have different strengths, not why. Follow-up work could investigate whether these dimension tradeoffs are inherent (models must sacrifice fluency for accuracy given fixed capacity) or contingent (different training recipes produce different tradeoffs, and training for balanced performance is possible).

Claim 4: "Even the most advanced LLMs still trail human experts on these tasks"

Supported, but the paper overstates the gap for the strongest models in the Abstract while providing appropriate nuance in Section 5. The Abstract states that "even the most advanced LLMs still trail human experts on these tasks" without qualification. This is true in aggregate (Table 3), but the gap between GPT-5-high and the human baseline (3.50 points) is small enough that different reasonable choices—a different human translator, a different set of 200 test cases, a different evaluation framework—could plausibly eliminate or reverse it. The more precise claim, supported by the data, is: "On this specific benchmark with this specific evaluation metric, GPT-5-high slightly underperforms a single professional human translation, while all other tested models trail by substantial margins." The Abstract's framing is defensible but less precise than the body of the paper.

Missing Experiments and Baselines

No comparison with reference-based metrics on DiscoX using synthetic references. The paper argues that discourse-level translation has no single correct reference, which is a valid theoretical position. However, a practical comparison would strengthen the case: generate multiple reference translations (e.g., by having 3–5 professional translators each produce a translation, or by using GPT-5-high's output as a synthetic reference) and test whether reference-based metrics with multiple references can approach Metric-S's performance. Without this comparison, the claim that reference-based metrics are "inadequate" (Section 1) rests entirely on XCOMET-QE's poor performance in the reference-free setting, not on a direct apples-to-apples comparison.

No latency or cost analysis for Metric-S. The four-stage multi-agent workflow (Instruction Check, 3 Quality Judges, De-duplication) requires at least 5 LLM calls per translation, each processing the full source text and translation (average 1,712 tokens input). For a 200-task benchmark, that's 1,000+ LLM calls. The paper does not report the computational cost, latency, or dollar cost of running Metric-S compared to XCOMET-QE (a single neural network forward pass). This matters for practical adoption—a metric that requires 5× the compute of alternatives needs to justify that cost with sufficient accuracy gains.

No sensitivity analysis of the severity deduction schedule. The paper assigns specific point deductions (2 for minor Accuracy, 5 for major, 10 for critical, 50 for extremely critical; 2 for Fluency; 5 for Appropriateness) without testing whether different deduction scales would change model rankings. Given that a single Extremely Critical error (50 points) effectively zeros out the Accuracy score, the relative ranking of models that produce different distributions of error severities could be sensitive to these weights. An ablation where deductions are varied (e.g., Extremely Critical at 30, 40, or 60 points) would establish whether the reported rankings are robust to the specific severity calibration.

No test of rubric contribution to Metric-S performance. The rubrics are central to DiscoX's design (Section 2.1) and are fed to Metric-S's Accuracy Judge (Section 3.2). However, the paper does not ablate the rubric component: what is Metric-S's consistency with human judgments when rubrics are not provided to the Accuracy Judge? This would quantify how much of Metric-S's advantage over XCOMET-QE comes from the multi-agent workflow vs. from the expert-authored criterion-referenced checkpoints. If removing rubrics drops consistency substantially, it strengthens the rubric mechanism's importance; if consistency remains high, it suggests the workflow alone is sufficient and rubrics are an optional enhancement.

No human evaluation of DiscoX's difficulty calibration. The paper filters texts based on "both of two SOTA LLMs fail on at least eight predefined rubrics" (Section 2.1), but does not report whether human translators also find these texts difficult. It is possible that DiscoX selects for texts where LLMs happen to fail (due to training data gaps or architectural biases) but that are actually easy for humans, which would inflate the human-LLM gap. Reporting human expert performance on the difficulty-filtered vs. unfiltered tasks would validate that the filtering selects for genuine translation difficulty rather than LLM-specific weaknesses.

Conditional Claims

The following claims hold conditionally based on the paper's experimental design:

"DiscoX is the first benchmark for discourse-level, expert-domain translation" holds if you accept the paper's operationalization of these terms: >1,500 characters/words, expert-authored rubrics, and difficulty-filtered curation. Benchmarks like the WMT discourse-level literary translation shared task (Wang et al., 2023, 2024) also evaluate discourse-level translation, but with different design choices (literary domain only, different difficulty calibration, different evaluation framework). DiscoX is more comprehensive in domain coverage and more systematic in difficulty filtering, but the "first" claim depends on how precisely "discourse-level, expert-level" is defined.
"Metric-S significantly outperforms existing metrics" holds on the DiscoX benchmark specifically and for reference-free comparison with XCOMET-QE. On WMT 2024 general-domain tasks (Appendix A.4, Table 10), Metric-S performs comparably to reference-based metrics (72.30% vs. XCOMET's 68.80%), with the advantage narrowing substantially. The "significant outperformance" claim is specific to discourse-level, expert-domain texts where reference-based metrics fail.
"Thinking models underperform non-thinking models" holds for the two model pairs tested (Qwen-3-235B and Claude-4), with the effect being large for Qwen (9.69 points) and negligible for Claude (0.50 points). Whether this generalizes to other thinking/non-thinking model pairs (e.g., DeepSeek-V3 vs. DeepSeek-R1, though DeepSeek-R1 at 46.06 does score lower than DeepSeek-V3 at 49.60, a 3.54-point gap shown in Table 3) is suggested but not systematically tested. The paper also does not test whether prompting the thinking models with translation-specific instructions (e.g., "do not summarize; translate faithfully") mitigates the deficit.

6. Limitations and Trade-offs

The Benchmark Validates Task Difficulty Against LLMs, But Does Not Calibrate Difficulty Against Human Translators

The assumption or constraint. The DiscoX curation pipeline applies a stringent difficulty filter to candidate texts: a task advances only if "both [tested SOTA] models fail on a minimum of eight predefined rubrics" (Section 2.1). This explicitly optimizes for LLM-difficulty—texts where current state-of-the-art models systematically fail on domain-critical checkpoints. The paper does not report how human translators perform on these same tasks relative to the unfiltered candidate pool, nor does it establish whether the 200 selected texts represent genuinely difficult translation problems or simply problems where LLMs have specific weaknesses.

The consequence. Without human difficulty calibration, it is possible that DiscoX selects for LLM-specific failure modes rather than intrinsically difficult translation challenges. For example, if the tested SOTA LLMs share a common architectural bias (e.g., poor handling of certain Chinese syntactic constructions due to training data imbalance), the difficulty filter would enrich for texts containing those constructions, producing a benchmark where the human-LLM gap is artificially inflated. A human translator might find these same texts straightforward—the gap would then measure LLM limitations rather than the inherent difficulty of professional discourse-level translation.

This has direct implications for interpreting the headline result in Table 3: the 3.50-point gap between GPT-5-high (76.66) and the human expert baseline (80.16) might overstate the true capability gap if DiscoX is systematically biased toward tasks where LLMs happen to struggle. Similarly, the finding that "even the most advanced LLMs still trail human experts" (Section 7) would be less meaningful if human translators would score near-perfect on the same tasks—the gap would reflect benchmark construction choices rather than a fundamental performance ceiling.

What evidence exists in the paper. The paper does not report human performance on the difficulty-filtered vs. unfiltered candidate texts, nor does it provide any analysis of whether the 200 selected tasks are also difficult for humans. The human expert baseline is a single translation evaluated only on the final DiscoX set (Table 3), with no comparison to human performance on the 465 rejected candidates. The paper's ablation on multiple runs (Appendix A.1, Table 6) validates that model outputs are stable, but this addresses measurement reliability, not benchmark validity—it shows that the scores are reproducible, not that they measure what they claim to measure.

The expert curation process (133 professionals, 1,330 person-hours) ensures domain authenticity, but domain authenticity is orthogonal to human difficulty. A text can be genuinely representative of professional translation work while being straightforward for a competent human translator—the two properties are independent. The paper conflates them by using "expert-level" to describe both the domain specialization of the texts and the challenge level of the benchmark, without establishing that these are empirically linked.

Mitigation status. The paper does not acknowledge this as a limitation, nor does it suggest future work to calibrate human difficulty. The filtering methodology is presented as a strength ("a stringent difficulty threshold," Section 2.1) without discussion of what the threshold is measuring. A straightforward validation would be to have 3–5 professional translators each translate a random sample of both accepted and rejected texts, then compare human error rates across the two sets. If humans also find the accepted texts substantially harder (making more rubric-level errors), the difficulty filter is validated. If humans perform similarly on both sets, the filter is selecting for LLM-specific weaknesses rather than genuine translation difficulty.

The Cost of Difficulty Estimation Is Structural to the Benchmark Design and Does Not Scale

The assumption or constraint. DiscoX's difficulty filtering requires that candidate texts be tested against two SOTA LLMs before they can be included in the benchmark. The paper reports an initial pool of 665 candidate texts, of which 200 were retained—a 30% selection rate (Section 2.1). Each candidate text required two full LLM translations and rubric-level evaluation (checking each of the ~9.38 average rubrics per text). The paper does not report the computational cost of this filtering step, but it is substantial: 665 texts × 2 LLMs × ~1,700 tokens per translation per model, plus the cost of rubric-checking (which at the filtering stage was presumably done by humans or LLMs—the paper does not specify). For the DiscoX construction, this cost was borne once. But any attempt to extend the benchmark—adding new domains, updating texts as LLMs improve, or creating analogous benchmarks for other language pairs—must repeat this filtering process. The difficulty threshold is relative to "two SOTA LLMs" at the time of construction; as models improve, tasks that were once hard become easy, and the benchmark must be periodically re-filtered or replaced to maintain its diagnostic value.

This is the benchmark-level analog of the difficulty estimation cost problem in the reference paper (the compute-optimal scaling paper's Section 3.2, where generating 2,048 samples to estimate question difficulty was flagged as prohibitively expensive). In both cases, the adaptive mechanism (difficulty filtering for DiscoX, difficulty-conditioned strategy selection for the compute-optimal paper) provides better diagnostic signal but at a computational cost that is not amortized in the headline evaluation numbers.

The consequence. Extending DiscoX to new domains or language pairs is prohibitively expensive for most research groups. The pipeline requires:

115+ vertical domain experts to source and rubricate candidate texts
18+ linguistic specialists for peer review
Access to two SOTA LLMs for difficulty testing
Manual review of LLM outputs against ~9+ rubrics per text to determine whether the "minimum 8 failures" threshold is met

The 1,330 person-hours figure reported in the paper covers only the construction of the initial 200-task benchmark, not any replication, extension, or maintenance. For a research group wanting to build a DiscoX-style benchmark for, say, Japanese-English medical translation, they would need to replicate this entire pipeline—not just translate the existing DiscoX texts, because the domain-specific rubrics and difficulty calibration are language-pair-specific.

Furthermore, the difficulty threshold is temporally fragile. As new LLMs surpass the two SOTA models used for filtering (which the paper does not name, but which were presumably frontier models at construction time), tasks that passed the filter (because both models failed on ≥8 rubrics) may become trivially easy. DiscoX would then lose its ability to discriminate among models, saturating like the sentence-level benchmarks it was designed to replace. The paper provides no mechanism for benchmark maintenance or versioning—no schedule for re-filtering, no protocol for adding harder tasks as capabilities improve.

What evidence exists in the paper. The paper explicitly reports the curation scale (133 professionals, 1,330 person-hours, 665 → 200 selection rate) in Section 1 and Section 2.1, but frames these as evidence of quality rather than as a cost barrier to extension. Table 2 shows the domain breakdown, and Table 3 shows that even GPT-5-high still underperforms humans, indicating the benchmark currently has headroom—but provides no guidance on when that headroom will be exhausted. The multiple-runs analysis (Appendix A.1, Table 6) validates output stability but does not address benchmark longevity.

Mitigation status. The paper does not acknowledge the extension cost as a limitation. The framing throughout is that DiscoX is a one-time contribution to the community ("We are releasing both DiscoX and Metric-S to the community to foster research," Section 7), with no discussion of how the benchmark should evolve as models improve. A partial mitigation would be to release the candidate pool (all 665 texts) along with the rubric annotations, so that future benchmark versions could draw from the same curated source material with updated difficulty filters. Alternatively, the paper could specify which two SOTA LLMs were used for filtering, so that the community can track when those models are surpassed and re-filter accordingly. Neither is mentioned.

Metric-S Requires 5+ LLM Calls per Evaluation, Making It Computationally Expensive for Large-Scale Use

The assumption or constraint. Metric-S is a multi-agent workflow that requires at minimum five separate LLM calls per translation evaluation: one Instruction Following Check, three Quality Estimation judges (Accuracy, Fluency, Appropriateness), and one De-duplication Judge (Section 3, Figure 4). Each call processes the full source text and translation (average 1,712 tokens input per DiscoX task), meaning the total input tokens processed across all judges is approximately 5 × 1,712 = 8,560 tokens per evaluation, plus output tokens for the judges' error reports. This is in addition to the translation model's own generation cost.

For the experiments reported in Table 3, with 20 systems × 200 tasks = 4,000 translations to evaluate, Metric-S requires approximately 20,000 LLM calls (4,000 × 5 judges). The paper uses Gemini-2.5-Pro as the judge model (Appendix A.2.2), a frontier closed-source model whose inference cost is non-trivial. At typical API pricing for models of this capability tier, evaluating all 20 systems on all 200 DiscoX tasks would cost thousands of dollars in inference alone—before accounting for the human expert time for validation (Table 11, 250 cases manually verified).

The consequence. Metric-S's computational cost creates a practical barrier to adoption in several common use cases:

Rapid model development cycles: A research team iterating on translation model fine-tuning might want to evaluate dozens of checkpoints per day. At 5+ LLM calls per evaluation and 200 test cases per checkpoint, this becomes economically prohibitive for all but the best-resourced labs.
Large-scale system comparison: A benchmark leaderboard evaluating hundreds of models (as WMT does annually) would incur inference costs proportional to the number of models × 200 tasks × 5 judges. This cost structure incentivizes evaluating fewer models or fewer tasks, reducing statistical power.
Production monitoring: Deploying Metric-S as a continuous quality monitor for a production translation system—evaluating a sample of outputs each day—would incur ongoing inference costs that may exceed the translation costs themselves for high-volume systems.

The paper does not provide any cost analysis, latency measurements, or comparison to cheaper alternatives. XCOMET-QE, by contrast, is a single neural network forward pass—it produces a score in milliseconds on commodity hardware with no API costs. The ablation in Table 7 shows that a single LLM judge with a detailed prompt achieves 55.9% consistency (vs. Metric-S's 70.3%), suggesting a possible cost-quality Pareto frontier where cheaper single-judge configurations might be preferable for budget-constrained applications. The paper does not explore this tradeoff.

What evidence exists in the paper. The paper provides no cost analysis whatsoever. The number of LLM calls per evaluation (5) can be inferred from the workflow description (Section 3, Figure 4), but the paper never explicitly counts them, reports total inference tokens, or estimates dollar costs. The ablation experiments in Table 7 compare consistency across configurations but do not include a "cost-normalized" comparison—e.g., how does Metric-S at 5 calls compare to running the single-judge configuration 5 times and ensembling, or to running a single judge 5× with different prompts? These comparisons would reveal whether Metric-S's advantage is architectural or simply reflects more total inference compute.

Table 4 shows Metric-S achieving 70.3% consistency vs. XCOMET-QE's 34.7%, but without a cost axis, it's impossible to judge whether this 2×+ accuracy improvement justifies the likely 100×+ computational cost increase (LLM inference vs. single neural forward pass). Table 8 further shows that Metric-S's performance degrades with weaker judge models (DeepSeek-R1: 57.8%, o3-high: 58.2%), but these models are themselves large LLMs whose inference cost, while lower than Gemini-2.5-Pro, is still orders of magnitude above XCOMET-QE.

Mitigation status. The paper does not address this limitation. There is no mention of cost in the main text (Sections 1–7), and Appendix A focuses on accuracy and consistency validation without reporting computational requirements. The paper frames Metric-S as a community resource ("We are releasing both DiscoX and Metric-S to the community," Section 7) without acknowledging that using the metric requires substantial API access to frontier LLMs. A partial mitigation—not mentioned in the paper—would be to evaluate whether a smaller, cheaper LLM can serve as the judge without catastrophic accuracy loss, potentially through distillation or fine-tuning on the human-validated error annotations. The fine-grained accuracy numbers in Table 11 (98.40%, 95.20%, 96.40% per-judge accuracy) suggest high-quality training data exists for such an effort, but the paper does not pursue it.

Single Benchmark, Single Language Pair, and Single Human Baseline Restrict the Generality of the Findings

The assumption or constraint. All experiments are conducted on a single benchmark—DiscoX—which covers one language pair (Chinese↔English) across seven domains, evaluated against a single human expert translation as the reference point (Table 3). The paper presents findings about model capabilities (e.g., "LLMs are better at zh→en than en→zh," Section 5.1; "thinking models underperform non-thinking counterparts," Section 5.3; "academic translation is easier than literary translation," Section 5.2) as general claims about LLM translation behavior, but the empirical basis for these claims is entirely within the Chinese-English context and against a single human reference.

The consequence. Several of the paper's headline findings may be language-pair-specific rather than universal properties of LLM translation:

The zh→en advantage (Section 5.1): The paper attributes this to "data imbalance, where high-quality English corpora are more abundant than Chinese ones" and "English-centric model architectures." These explanations would predict the same asymmetry for any language pair involving English and a lower-resource language (e.g., en→ar would be worse than ar→en)—but they would not predict the direction for non-English language pairs (e.g., zh→ja vs. ja→zh). The finding may not generalize.
The thinking-model deficit (Section 5.3): The paper observes this in two model pairs (Qwen-3-235B and Claude-4) for Chinese-English translation specifically. Whether thinking models over-summarize when translating between other language pairs—or whether the effect is specific to the interaction between chain-of-thought reasoning and Chinese-English cross-lingual transfer—is untested.
The academic > literary domain effect (Section 5.2): This is attributed to literary texts requiring "complex syntax, rhetorical devices, and cultural nuances," which is plausible across language pairs, but the specific magnitude of the gap almost certainly depends on the cultural distance between the source and target languages. The gap might be larger for Chinese↔English (languages with radically different literary traditions) than for, say, French↔English (shared European literary conventions).

The single human baseline is an even more significant constraint on the headline finding that "even the most advanced LLMs still trail human experts" (Abstract). Human translation quality varies substantially across translators—factors including experience, domain familiarity, time constraints, and stylistic preferences all produce variation. A different professional translator might produce a translation scoring anywhere from 75 to 85 on Metric-S, which would change the interpretation of GPT-5-high's 76.66 score from "trailing humans" to "within human range" or even "exceeding some human translators." Without reporting inter-translator variability (e.g., having 3–5 professionals each translate a subset and reporting score distributions), the human baseline is a point estimate with unknown variance.

What evidence exists in the paper. The paper acknowledges the scope limitation implicitly through its title and abstract, which specify "Chinese-English translation," but the analysis sections (5.1–5.4) often present findings without language-pair qualification. Table 3 reports a single human score (80.16) without confidence intervals or inter-translator statistics. Appendix A.1 (Table 6) reports multiple-run stability for models (standard deviations of 0.21–1.36) but does not report multiple-translator stability for the human baseline. The domain composition (Table 2) shows 121 Academic vs. 79 Non-Academic tasks, but the paper does not discuss whether results would change with different domain proportions.

Mitigation status. The paper does not address the language-pair limitation as a threat to generalizability. The framing positions DiscoX as "the first benchmark for discourse-level, expert-domain translation" (Section 1) without specifying that it is Chinese-English-specific in the core claim. The human baseline limitation is not acknowledged—the paper treats the human expert score of 80.16 as a fixed reference point rather than a sample from a distribution. Future work would need to replicate the findings on additional language pairs (at minimum, a linguistically similar pair like French-English and a distant pair like Japanese-English) and with multiple human translators per task to establish which findings are universal and which are language-pair-specific or translator-specific.

Metric-S's Segment-Level Consistency (54.8–56.4%) Remains Modest, Limiting Its Usefulness for Individual Translation Comparison

The assumption or constraint. The paper validates Metric-S using pairwise consistency with human judgments at both system and segment levels (Section 4.2, Table 4). While system-level consistency is strong (80.0–90.0%), segment-level consistency—the ability to correctly judge which of two individual translations is better—is much weaker: 54.8% for zh→en and 56.4% for en→zh. The paper treats scores as consistent if their difference is less than 0.05 (Section 4.2), meaning that Metric-S is effectively only slightly better than random at ranking individual translation pairs. (Chance for a binary preference task with a tie threshold is not exactly 50%, but with the 0.05 threshold, it is close.)

The fine-grained accuracy analysis in Appendix A.5 (Table 11) provides important context: the individual judges are highly accurate at identifying errors when they flag them (Accuracy Judge: 98.40%, Fluency Judge: 95.20%, Appropriateness Judge: 96.40%). The segment-level consistency problem is therefore primarily about false negatives—Metric-S misses errors that human evaluators catch—rather than false positives (inventing errors that don't exist). The metric is conservative: when it says something is wrong, it's almost certainly correct, but it fails to detect a substantial fraction of the problems that human evaluators notice.

The consequence. Metric-S is reliable for system-level comparison but unreliable for individual translation comparison. For a practitioner asking "Is Model A better than Model B for this translation task?", Metric-S provides a trustworthy answer (80–90% correct at the system level). For a practitioner asking "Which of these two translations of this specific document is better?", Metric-S is only marginally more reliable than flipping a coin (~55% correct vs. ~50% chance).

This limitation matters for several practical use cases:

Translation quality assurance: A human reviewer evaluating individual translations for publication or legal use cannot rely on Metric-S to reliably identify the better of two candidate translations. The metric may complement human review but cannot replace it for individual decisions.
Active learning / data filtering: If a practitioner wants to filter a large corpus of translations, keeping only those above a quality threshold, Metric-S's false-negative rate means many acceptable translations would be incorrectly rejected (or many high-quality translations would be scored similarly to lower-quality ones, making thresholding ineffective).
Fine-grained model diagnostics: The paper's strength is its dimension-level error profiles (Section 5.3, Table 3), but these profiles are aggregates over 200 tasks. For an individual task, the error profile may be inaccurate due to missed errors, so per-task diagnostics for model improvement are noisy.

The paper's finding that XCOMET-QE's segment-level consistency is below Metric-S's (26.4–32.4% vs. 54.8–56.4%) is important context—Metric-S is substantially better than the alternative, even at the segment level. But "substantially better than a metric that is worse than random" is a low bar, and the absolute performance remains modest.

What evidence exists in the paper. Table 4 reports segment-level consistency directly: 54.8% zh→en, 56.4% en→zh for Metric-S, vs. 26.4% and 32.4% for XCOMET-QE. Table 7 shows that segment-level consistency is relatively stable across ablations (ranging from 46.4% to 55.6%) while system-level consistency varies dramatically (20.0% to 85.0%), suggesting that segment-level evaluation is an intrinsically harder problem that the Metric-S architecture improves only modestly. Table 11 shows the high precision of individual judges, and Table 10 shows that segment-level consistency on WMT 2024 general tasks is comparable (54.6% for Metric-S vs. 57.6% for reference-based XCOMET), indicating the limitation is not specific to DiscoX but general to segment-level translation quality assessment.

Mitigation status. The paper partially acknowledges this limitation by reporting both system-level and segment-level consistency numbers, but it does not discuss the practical implications of modest segment-level performance. The Abstract and Section 1 frame Metric-S as providing "strong consistency with human judgments" and "a robust framework for more rigorous evaluation" without qualifying that this characterization primarily applies to system-level comparison. A more complete mitigation would be to combine Metric-S with complementary signals—e.g., reference-based metrics where references exist, or human spot-checking for high-stakes individual decisions—or to investigate whether ensembling multiple judge models (e.g., running Metric-S with both Gemini-2.5-Pro and DeepSeek-R1 and averaging) improves segment-level consistency. The paper's finding that different judge models produce different absolute scores but similar relative rankings (Table 8) suggests that ensembling might help with the false-negative problem by catching errors that individual judges miss.

The 60/20/20 Weighting Scheme and Severity Deduction Schedule Are Calibrated Against Human Judgments on Only 50 Cases

The assumption or constraint. Metric-S's final score formula (Section 3.4) involves two critical sets of hyperparameters: the dimension weights (Accuracy: 60, Fluency: 20, Appropriateness: 20) and the severity deduction schedule (Accuracy: minor = -2, major = -5, critical = -10, extremely critical = -50; Fluency: -2 per error; Appropriateness: -5 per error). The ablation study in Table 7 shows that changing dimension weights from 60/20/20 to equal weighting (33.3/33.3/33.3) causes a catastrophic drop in consistency from 70.3% to 48.2%, confirming that the specific weighting matters enormously. However, the paper does not report how the 60/20/20 weights were determined, whether they were optimized against the same 50-case validation set used for consistency measurement (creating potential overfitting), or whether different splits of that calibration data would produce different optimal weights.

Similarly, the severity deduction schedule—particularly the 50-point deduction for Extremely Critical Accuracy errors—is not ablated. A single Extremely Critical error (e.g., an untranslated segment) reduces the Accuracy score from 60 to at most 10, effectively deciding the overall score regardless of other performance. Whether this steep penalty matches human judgment (do humans really consider a single untranslated segment as disqualifying as 10 critical errors or 25 major errors?) is not validated beyond the aggregate consistency numbers, which measure ranking agreement rather than score calibration.

The consequence. The reported model rankings in Table 3 may be sensitive to the specific hyperparameter choices, particularly for models that differ primarily in their error severity distributions rather than their total error counts. For example:

A model that makes many minor Accuracy errors (e.g., -2 points each) but no Extremely Critical errors would be scored very differently than a model that makes one Extremely Critical error (-50) and few minor errors, even if human evaluators might consider the latter model's single catastrophic failure as less problematic than the former's pervasive low-level inaccuracy. The severity weights encode a specific tradeoff between error frequency and error severity that may not match all human evaluators' preferences.
The Fluency deduction of -2 points per error means that 10 fluency problems are scored equivalently to one major Accuracy error (-5) or one Appropriateness error (-5). This equivalence is asserted by the scoring formula but not validated against human judgments about the relative importance of different error types.
Claude-4's profile (Accuracy 39.38, Fluency 5.98) and Kimi-K2's profile (Accuracy 27.63, Fluency 16.44) produce very different dimension scores, and their overall ranking depends on the 60/20/20 weights. Different weights could reverse their relative ordering, changing the paper's conclusions about model complementarity.

The validation sample size (50 cases, Section 4.2) is small relative to the number of implicit hyperparameter degrees of freedom. If the 60/20/20 weights and severity schedule were tuned (even informally) against this same 50-case set, the reported 70.3% consistency may overestimate Metric-S's true generalization to new texts. The standard error on a proportion near 70% with n=50 is approximately 6.5 percentage points, meaning the true consistency could plausibly be anywhere from 64% to 77% at a 95% confidence level.

What evidence exists in the paper. Table 7 provides the only direct ablation of the weighting scheme, comparing 60/20/20 to equal weights and showing a 22.1-point consistency drop. This single comparison demonstrates that weights matter but does not explore the space of possible weightings—e.g., what if Accuracy were weighted at 50 or 70? Is 60/20/20 the optimum, or just one point that happens to work well? The paper does not report any systematic weight sweep or cross-validation procedure for hyperparameter selection. Table 8 shows that Metric-S's consistency varies with judge model choice (from 57.8% to 70.3%), but does not test whether the optimal weights are judge-model-dependent (e.g., Gemini-2.5-Pro might require different dimension weights than DeepSeek-R1 to best match human judgments).

The severity deduction schedule is not ablated at all. The paper specifies the per-error deductions without justifying the specific numbers or testing alternatives. The statement in Appendix E that "the details of severity levels in different domains" vary suggests domain-dependent adjustments, but these adjustments are not specified, and their effect on model rankings is not analyzed.

Mitigation status. The paper does not acknowledge the calibration sample size or the potential for hyperparameter overfitting. The ablation in Table 7 is presented as evidence that the 60/20/20 scheme is superior (it improves consistency), but without a description of how the weights were selected—e.g., were they chosen based on prior work, an independent calibration set, or the same 50 cases used for evaluation?—the risk of overfitting to the validation sample cannot be assessed. A standard mitigation would be to report weight selection on a separate calibration set (e.g., 25 cases) and evaluate consistency on a held-out set (the remaining 25), or to report consistency under a range of plausible weightings to show that the relative model rankings are robust to reasonable weight variation. Neither is done.

7. Implications and Future Directions

How This Work Changes the Landscape

DiscoX and Metric-S together establish discourse-level, expert-domain translation evaluation as a distinct capability assessment problem that cannot be approximated by sentence-level benchmarks or single-score metrics. This is not a paradigm shift—it does not propose new translation architectures, training objectives, or decoding strategies. Rather, it is a methodological reframing with three concrete consequences for how the field evaluates and develops translation systems.

First, DiscoX invalidates the premise that sentence-level benchmarks provide sufficient signal for professional translation quality. The paper does not argue that existing benchmarks like WMT and FLORES are wrong—it argues they are measuring a different, easier problem. The evidence for this is not merely the length difference (1,712 vs. 48 tokens), but the catastrophic failure of a state-of-the-art evaluation metric (XCOMET-QE) on DiscoX: 34.7% pairwise consistency vs. 70.3% for Metric-S, with XCOMET-QE scoring 10.0% system-level consistency on zh→en—essentially random (Table 4). If a metric that performs well on WMT-level tasks cannot distinguish better from worse translations on discourse-level texts, then sentence-level benchmarks provide literally no information about discourse-level translation quality. This finding makes it harder to claim that "LLMs are approaching human parity in translation" without specifying at what text scale—a qualification that was previously optional and is now empirically necessary.

Second, the paper resolves the tension between multi-dimensional evaluation and practical usability by making error de-duplication a first-class architectural component. Prior work recognized that translation quality has distinct facets (accuracy, fluency, style) but either (a) collapsed them into a single score, losing diagnostic signal, or (b) evaluated them independently, producing correlated error counts that inflated penalties. Metric-S demonstrates that a dedicated de-duplication agent with explicit priority rules can preserve the benefits of specialized evaluation while preventing double-counting. The ablation evidence (Table 7) shows this is a secondary but real effect: de-duplication improves consistency by 4.3 percentage points (70.3% → 66.0%), and the three-judge decomposition itself provides the primary gain (70.3% vs. 55.9% for a single merged-prompt judge). The architectural template—specialist judges + attribution agent—is generalizable to any multi-dimensional LLM evaluation task, not just translation.

Third, the difficulty-filtered benchmark construction methodology changes the conversation about what benchmarks are for. DiscoX explicitly rejects representative sampling (measuring average performance) in favor of difficulty filtering (measuring capability ceilings). The 30% selection rate from 665 candidates (Section 2.1) means DiscoX discards tasks where models already succeed, concentrating evaluation on the frontier of failure. This is a deliberate choice with a clear epistemological rationale: when a capability is economically critical but existing benchmarks are approaching saturation, difficulty-filtered benchmarks provide more diagnostic signal per evaluation dollar than representative ones. The paper does not argue that all benchmarks should be difficulty-filtered—only that for capabilities where underestimating the gap has real consequences (professional translation errors in legal contracts, technical manuals, scientific papers), the additional curation cost is justified.

The findings also make some research directions less attractive. The paper's demonstration that thinking-enhanced models systematically underperform non-thinking counterparts on discourse-level translation (Section 5.3, Table 5)—with Qwen-3-235B showing a 9.69-point thinking deficit—suggests that the straightforward application of chain-of-thought reasoning to translation is counterproductive. Research agendas premised on "better reasoning → better translation" need to contend with the specific failure mode identified: thinking models over-summarize and introduce extraneous content, treating the source as a prompt for analysis rather than a template for faithful reproduction. This doesn't mean thinking models are useless for translation, but it does mean that naive deployment without translation-specific prompting or fine-tuning will likely produce worse results, not better.

Similarly, the paper makes reference-free evaluation for discourse-level translation look necessary rather than optional. XCOMET-QE's 10.0% system-level consistency on zh→en DiscoX tasks (Table 4) demonstrates that even state-of-the-art neural metrics trained on sentence-level quality estimation cannot transfer to discourse-level texts. Since discourse-level texts have no single correct reference (Section 6.2), reference-based metrics are definitionally inadequate—they penalize valid variation. The path forward is either rubric-criterion-referenced evaluation (as Metric-S demonstrates) or training discourse-level quality estimators on discourse-level human judgments. The latter requires exactly the kind of expert-intensive annotation that DiscoX's construction pipeline exemplifies—1,330 person-hours for 200 test cases—making it clear that evaluation quality and annotation cost are tightly coupled in the discourse-level regime.

Follow-Up Research This Work Enables

Training discourse-level quality estimation models on Metric-S outputs. Metric-S produces rich, dimension-labeled, severity-weighted error annotations that are validated against human judgment at the error level (Appendix A.5, Table 11: 95.2–98.4% precision across dimensions). These annotations—error type, error location, error severity, dimension classification—constitute a high-quality training dataset for a specialized discourse-level quality estimation model. A strong follow-up would fine-tune a medium-sized LLM (e.g., 7–13B parameters) on the ~4,000 evaluation instances from the 20-system × 200-task DiscoX evaluation run, training it to predict per-dimension error counts and locations directly rather than executing a multi-agent workflow. The research question is whether the workflow's knowledge can be distilled into a single forward pass without catastrophic accuracy loss. The evaluation would compare the distilled model's pairwise consistency against Metric-S and human judgments on the same 50-case validation set, with particular attention to whether the distilled model reproduces the high-precision error detection (Table 11) or regresses to the false-positive-prone behavior of simpler LLM judges.

Investigating why thinking models over-summarize on translation tasks. Section 5.3 establishes the phenomenon—thinking models underperform non-thinking counterparts—but does not investigate the mechanism beyond observing that models "over-summarize the source text" and "introduce extraneous structural content." A mechanistic follow-up would systematically vary the thinking prompt to test whether the deficit is caused by (a) the chain-of-thought process itself consuming context window space that would otherwise be used for faithful reproduction, (b) the thinking model's training objective (reinforcement learning for reasoning) creating a bias toward compression and reorganization, or (c) the interaction between extended generation (thinking tokens + translation tokens) and the model's tendency to drift from the source over long outputs. The experiment would compare thinking models under different prompting regimes: standard translation prompt, translation prompt with explicit "do not summarize; translate each sentence faithfully" instructions, and a two-stage pipeline where thinking is used for terminology planning followed by non-thinking translation. If prompt modifications close the thinking deficit, the problem is remediable through instruction design; if they don't, the deficit is structural to how thinking models process translation tasks, implying that reasoning-capable models need translation-specific post-training.

Extending DiscoX to additional language pairs and measuring which findings replicate. The paper's findings—zh→en advantage over en→zh, academic > literary performance, thinking model deficit—are established for Chinese↔English only. A replication study on at minimum three language pairs would distinguish universal from language-pair-specific effects. The key design choices for such a study: one linguistically close pair (e.g., French↔English, where shared vocabulary and similar syntax minimize cross-lingual transfer difficulty), one linguistically distant pair with a similar resource asymmetry (e.g., Japanese↔English, where the target language differs radically in syntax and writing system), and one pair where both languages are non-English and have different resource profiles (e.g., Chinese↔Japanese, to test whether the en→zh deficit is specifically about generating Chinese or about translating into lower-resource languages generally). The study would replicate the DiscoX curation pipeline—domain expert sourcing, rubric authoring, difficulty filtering against current SOTA LLMs—at a smaller scale (e.g., 50–100 tasks per language pair) and evaluate the same 5–10 models across all pairs. The primary outcome would be a matrix of direction × domain × thinking/non-thinking effects, identifying which findings from DiscoX are universal properties of LLM translation and which are artifacts of the Chinese-English context.

Developing an online difficulty estimation method that eliminates the need for pre-filtering benchmarks. DiscoX's construction requires testing candidate texts against SOTA LLMs to establish difficulty before inclusion—a process that is expensive (Section 2.1: 665 texts evaluated, only 200 retained) and temporally fragile (as LLMs improve, tasks become too easy). A methodological follow-up would replace pre-filtering with dynamic difficulty estimation: a lightweight classifier trained to predict, from source text features alone (domain, length, terminology density, syntactic complexity metrics), whether a given text will be challenging for current LLMs. The training data exists: the 665 candidate texts with their known LLM failure counts (how many of the ~9.38 average rubrics each tested model failed). The evaluation would test whether a classifier trained on this data can predict which of a held-out set of new expert-domain texts would be retained by the full LLM-testing pipeline, and whether a benchmark constructed using the classifier's selections produces comparable model rankings to a benchmark constructed with full LLM testing. If successful, this would make DiscoX-style benchmarks extensible without the prohibitive cost of per-candidate LLM evaluation—a domain expert could submit a new text, the classifier would estimate its difficulty, and only texts above a threshold would enter the benchmark.

Stress-testing Metric-S against adversarial translations designed to exploit its failure modes. The fine-grained accuracy analysis (Table 11) shows Metric-S has high precision (95.2–98.4% correct when it flags an error) but modest segment-level pairwise consistency (54.8–56.4%, Table 4), implying false negatives—Metric-S misses errors that humans catch. A stress test would systematically construct translations designed to expose these blind spots: translations with subtle terminology errors that are lexically plausible but semantically wrong (e.g., translating a domain-specific acronym correctly in one context but incorrectly in another), translations with fluent but factually inverted content (negations removed, quantities altered), and translations with culturally inappropriate adaptations that are grammatically perfect and stylistically consistent but miss the source's pragmatic intent. For each adversarial category, the evaluation would compare Metric-S's error detection rate against human experts, identifying which error types are systematically missed. A negative result—Metric-S maintaining high precision across adversarial categories—would strengthen confidence in the metric for high-stakes use; a positive result—Metric-S failing on specific error types—would identify where human review remains necessary and what a next-generation metric needs to address.

Evaluating whether Metric-S's rubric mechanism generalizes to domains without pre-authored rubrics. The rubrics are central to Metric-S's Accuracy evaluation (Section 3.2) and are fed as explicit checkpoints to the Accuracy Judge. But rubrics require domain experts to pre-specify what must be translated correctly—a cost that may not be feasible in all deployment settings. A generalization study would test Metric-S on expert-domain texts where rubrics are not provided, comparing two configurations: (a) Metric-S with the Accuracy Judge operating purely on its general translation knowledge (no rubric checkpoints), and (b) Metric-S where an additional LLM agent first generates candidate rubrics from the source text (e.g., "identify all domain-specific terminology, proper names, and culture-loaded expressions that must be translated correctly") and feeds them to the Accuracy Judge. The evaluation would measure pairwise consistency with human judgments for both configurations against the full Metric-S with expert-authored rubrics. If auto-generated rubrics close most of the gap to expert-authored rubrics, the approach becomes much more scalable—domain experts would only be needed for calibration and spot-checking, not exhaustive rubric authoring. If auto-generated rubrics perform poorly, the scalability limitation of the rubric approach is confirmed, and future work should focus on making rubric authoring more efficient rather than automating it.

Practical Applications and Downstream Use Cases

Professional translation service benchmarking and procurement. Organizations that regularly contract professional translation services—multinational corporations, legal firms, scientific publishers, government agencies—need to evaluate and compare translation providers. Currently, this evaluation is either manual (expensive, slow, inconsistent across evaluators) or relies on sentence-level automated metrics (irrelevant to the discourse-level quality that matters for legal contracts, technical manuals, and scientific articles). DiscoX + Metric-S provide a standardized, automated, multi-dimensional quality assessment that aligns with human expert judgment at the system level (80.0–90.0% consistency, Table 4). A procurement team could run candidate translation providers' outputs through DiscoX and receive dimension-level scores showing which provider excels at accuracy (for legal documents where precision is paramount), which at fluency (for marketing materials where readability matters), and which at appropriateness (for culturally sensitive content). The 3.50-point gap between GPT-5-high and the human baseline (Table 3) establishes that human translation still has a measurable quality advantage, providing a data-driven justification for when to use human vs. machine translation rather than relying on intuition or cost alone.

Targeted model improvement through dimension-level diagnostics. The finding that models have complementary strengths (Claude-4: Accuracy 39.38, Fluency 5.98; Kimi-K2: Accuracy 27.63, Fluency 16.44, Table 3) means that Metric-S's output is directly actionable for model developers. A team working on a translation model can run DiscoX evaluation, identify which dimension is the bottleneck for their specific model, and target their improvement efforts accordingly. If their model scores 30 on Accuracy but 16 on Fluency (like Kimi-K2), investing in better terminology handling and omission detection will yield larger overall score improvements than further fluency work. If their model scores 39 on Accuracy but 6 on Fluency (like Claude-4), post-training for naturalness and target-language idiomaticity is the priority. This is a substantial practical advance over single-score metrics, which tell you that your model needs to improve but not what to improve. The fine-grained accuracy of Metric-S's error identification (95.2–98.4% precision, Table 11) means that when the metric flags a specific error type as frequent, developers can trust that it reflects a real weakness rather than metric noise.

Continuous quality monitoring for production translation systems. Organizations deploying LLM-based translation in production—for customer support, content localization, or internal document translation—need ongoing quality assurance to detect regressions, domain shifts, or prompt drift. DiscoX is not suitable for continuous monitoring (it is a fixed 200-task benchmark), but Metric-S's workflow can be applied to a rotating sample of production translations without requiring reference translations or pre-authored rubrics for every text. A monitoring system could sample 50–100 translations per week, run Metric-S (at a cost of 5 LLM calls per translation), and track per-dimension quality trends. A sudden drop in Accuracy scores could indicate a model update that introduced omission behavior; a gradual decline in Appropriateness could indicate domain drift as the input distribution shifts. The system-level consistency of 80.0–90.0% (Table 4) means that trend detection over aggregate samples is reliable even if individual translation judgments are noisy. The main barrier to adoption is cost—5 LLM calls per evaluation at frontier model pricing—but this could be reduced by using a cheaper judge model (Table 8 shows DeepSeek-R1 achieves 57.8% consistency vs. Gemini-2.5-Pro's 70.3%, a meaningful drop but still well above XCOMET-QE's 34.7%) or by the distillation approach discussed above.

Domain-specific translation model evaluation for regulated industries. In regulated industries—pharmaceuticals, medical devices, legal services, aerospace—translation errors carry compliance risk, and translation quality must be documented and defensible. Existing automated metrics (BLEU, COMET, ChrF) are insufficient for regulatory documentation because they provide a single opaque score with no error taxonomy and no explainability. Metric-S provides dimension-level scores with specific error types, locations, and severity classifications, plus a documented de-duplication process that ensures errors are not double-counted. For a pharmaceutical company submitting translated clinical trial documentation to regulators, Metric-S could generate a quality report showing: 3 Accuracy errors (2 major terminology issues in Section 3, corrected; 1 critical omission in Section 7, requires revision), 2 Fluency errors (minor grammatical issues, acceptable), 0 Appropriateness errors. This level of documentation—explainable, auditable, dimension-specific—is closer to what regulators and quality assurance teams actually need than a single "BLEU score of 0.45" which communicates nothing about whether the errors matter. The high precision of Metric-S's error detection (Table 11) is critical here: false positives (flagging non-errors as errors) would undermine trust in automated quality documentation, but Metric-S's 95.2–98.4% precision means flagged errors are almost certainly genuine, making the reports defensible in a regulatory context.