DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?

🎯 Pitch

Reasoning models don’t automatically make better NLG evaluators—DeepSeek-R1 trails its non‑reasoning variant on most translation metrics, yet it unexpectedly beats it by 70 % on summarization consistency. Distilled reasoning models hold up well down to 32B parameters but collapse at 8B, setting a sharp lower bound on practical deployment.

1. Executive Summary

This paper systematically studies whether reasoning-enabled large language models (LLMs) offer advantages over non-reasoning counterparts when used as evaluators of natural language generation. The experiments benchmark eight models—spanning DeepSeek-R1, OpenAI o3-mini, their distilled variants (8B–70B parameters), and equivalent non-reasoning LLMs—on the WMT23 dataset for machine translation evaluation and the SummEval dataset for text summarization evaluation, using GEMBA-MQM and G-Eval prompting frameworks, respectively. The analysis reveals architecture-dependent and task-dependent benefits: OpenAI o3-mini models show improved performance with increased reasoning effort on machine translation (e.g., o3-mini-high achieving a 0.577 segment-level Pearson correlation for en-de vs. 0.410 for GPT-4o-mini, a 40.7% improvement), while DeepSeek-R1 generally underperforms compared to its non-reasoning variant DeepSeek V3 except in summarization consistency evaluation (0.565 vs. 0.331 Kendall's τ, a 70.7% improvement). The paper further finds that distillation preserves reasonable performance up to 32B parameters but degrades substantially at the 8B scale, establishing that effective distillation of evaluation-relevant reasoning requires sufficient model capacity to maintain quality.

2. Context and Motivation

The Core Gap: Reasoning LLMs Are Unexplored as NLG Evaluators

The fundamental question this paper tackles is straightforward but unanswered: do reasoning-enabled LLMs make better evaluators of machine-generated text than standard LLMs? This matters because LLM-based evaluation has rapidly become the dominant paradigm for assessing the quality of machine translation, text summarization, and other natural language generation tasks, yet the recent wave of reasoning models—which have transformed performance on mathematical, logical, and multi-step inference problems—has never been systematically tested in this role.

The gap exists because reasoning LLMs and LLM-based evaluation metrics evolved along separate trajectories. Reasoning models like DeepSeek-R1 and OpenAI o3-mini emerged from reinforcement learning pipelines (Group Relative Policy Optimization, GRPO) and chain-of-thought (CoT) supervision designed to improve performance on benchmarks requiring explicit step-by-step deliberation—math competitions, coding challenges, and formal logic tasks. Meanwhile, the "LLM-as-a-Judge" paradigm developed its own set of prompting techniques (GEMBA-MQM, G-Eval) and evaluation frameworks optimized for standard instruction-tuned models like GPT-4, GPT-4o-mini, and LLaMA-3. The paper identifies this as a missed opportunity: if reasoning models truly excel at multi-step, nuanced analysis, they should, in principle, be able to evaluate translations and summaries more like human experts do—by systematically identifying semantic mismatches, weighing multiple quality dimensions, and producing fine-grained error assessments rather than relying on holistic pattern matching.

The paper does not assume reasoning will help. It explicitly acknowledges the counter-evidence: CoT prompting can degrade performance when verbal deliberation introduces unnecessary complexity (Liu et al., 2024) or when the task does not require it (Sprague et al., 2025). The research questions are therefore genuinely open: (1) whether reasoning models improve NLG evaluation quality at all, and (2) whether distilled reasoning models can preserve these capabilities at reduced computational cost, given that full-scale reasoning LLMs often have hundreds of billions of parameters.

Why This Problem Matters: Practical Deployment and Theoretical Understanding

Practical significance. LLM-based evaluation has become the backbone of NLG system development. When researchers and engineers train a new summarization model or deploy a machine translation system, they no longer rely solely on surface-level metrics like BLEU or ROUGE—they use prompted LLMs to assess fluency, coherence, factual consistency, and translation accuracy. The quality of these LLM judges directly determines the quality of the feedback loop: better evaluators lead to better systems. If reasoning LLMs can outperform standard models as judges—even selectively, on specific dimensions or for specific tasks—that has immediate implications for how evaluation pipelines should be constructed and which models should be deployed in production metric services.

The practical stakes are further amplified by the computational cost of these models. Full-scale reasoning LLMs (e.g., DeepSeek-R1 at 685B parameters, o3-mini with variable reasoning effort) are expensive to run. If distilled versions can preserve evaluation quality at a fraction of the size (e.g., R1 Qwen 32B achieving 97% of R1's performance on Eval4NLP with 21× fewer parameters), that enables deployment of reasoning-enhanced evaluation in resource-constrained settings—on-device quality checking, real-time translation assessment, or large-scale batch evaluation where cost per query matters.

Theoretical significance. Beyond the immediate practical question, this paper probes a deeper issue about the nature of reasoning in language models. The NLG evaluation task sits at an interesting boundary: it requires careful, multi-step analysis (identifying errors, categorizing their type, assessing severity, considering context) that should benefit from chain-of-thought deliberation, yet it also involves holistic judgments (overall fluency, naturalness) where over-analysis might introduce noise. Understanding where reasoning helps and where it hurts in this domain provides evidence about the transferability of reasoning capabilities—do the reasoning skills acquired through RL training on math and logic problems generalize to the qualitatively different task of critiquing text quality?

Moreover, the paper provides evidence about a phenomenon the field has only begun to study: test-time reasoning scaling for non-math tasks. By analyzing the correlation between reasoning token count and evaluation quality (Section 5), the paper tests whether "letting the model think longer" consistently improves evaluation accuracy—a finding with implications for how reasoning models should be deployed with controlled reasoning budgets.

Prior Approaches and Their Limitations

Traditional metrics: surface-level, semantically blind. Before LLM-based evaluators, MT and TS evaluation relied on reference-based metrics that compare system outputs to human-written references. The paper catalogs these in Section 2:

For MT: BLEU (n-gram overlap), METEOR (edit distance), BERTScore (contextualized embeddings), MoverScore (Word Mover's Distance), and trained metrics like COMET and xCOMET. The fundamental limitation is that these metrics measure surface similarity rather than semantic adequacy. A translation can be lexically different from the reference yet perfectly correct, or lexically similar yet semantically wrong—traditional metrics struggle with both cases.
For TS: ROUGE (n-gram overlap with reference summaries), Pyramid (content selection scoring across multiple references), Basic Elements (minimal semantic unit matching), and SUPERT (unsupervised semantic evaluation). The same limitation applies: these metrics reward lexical overlap with references, penalizing valid paraphrases and failing to detect factual inconsistencies that happen to use similar words.

LLM-as-a-Judge: powerful but built on non-reasoning models. The paradigm shift came with prompting-based evaluation, where instruction-tuned LLMs directly assess system outputs along specified quality dimensions. The paper highlights several influential frameworks:

GEMBA-MQM for MT: prompts an LLM to detect translation errors across dimensions (accuracy, fluency, terminology), assign severity scores, and produce a final quality rating. It achieved state-of-the-art performance on WMT22 by leveraging GPT-4's strong instruction-following capabilities.
G-Eval for TS: uses criteria-based prompting with chain-of-thought to evaluate summaries on coherence, consistency, relevance, and fluency, producing both numerical scores and explanations.
AutoMQM and FineSurE: extend the LLM-as-judge paradigm with more structured error detection and fact verification, respectively.

The critical limitation of ALL these approaches is that they were developed and tested exclusively on non-reasoning LLMs—standard instruction-tuned models that produce answers directly without explicit multi-step deliberation. Even when these methods incorporate chain-of-thought prompting (as G-Eval does), the underlying model does not have dedicated reasoning training; it simply follows the CoT instruction as part of its standard generation. The paper argues that this is a missed opportunity because human evaluation of translations and summaries is inherently a reasoning-heavy process: it requires comparing source and target texts, identifying discrepancies, categorizing error types, weighing severity, and sometimes trading off competing quality dimensions. Reasoning models, trained explicitly for multi-step analysis, might perform this evaluation more faithfully.

The efficiency gap: lightweight alternatives exist but don't incorporate reasoning. The paper acknowledges prior work on efficient evaluation metrics (FrugalScore, EffEval, COMETINHO, xCOMET-lite, PromptOptMe) that use distillation, quantization, or prompt optimization to reduce computational cost. However, none of these efforts have explored distilling reasoning capabilities for evaluation—they all work with standard non-reasoning model outputs. This leaves open the question of whether the specialized reasoning behaviors (long chains of deliberation, error identification and categorization, step-by-step justification) can be preserved in smaller distilled models.

The missing piece: no systematic comparison exists. Perhaps most critically, the paper identifies that prior work has never directly compared reasoning and non-reasoning LLMs as evaluators using the same prompting frameworks, same datasets, and same evaluation metrics. The field has accumulated results on standard models (GPT-4, LLaMA, Qwen) and separately accumulated results on reasoning models (DeepSeek-R1, o3-mini) for math and logic tasks, but no one has brought these two streams together. This means there is no evidence base for deciding whether to use a reasoning or non-reasoning model in an NLG evaluation pipeline—decisions are made based on intuition or convenience rather than systematic comparison.

How This Paper Positions Itself

The paper frames its contribution as filling this specific gap: it is "the first systematic comparison of reasoning versus non-reasoning LLMs as NLG evaluators across MT and TS tasks" (explicit claim in Sections 1 and 7). The positioning is explicitly empirical and comparative rather than methodological—the paper does not propose a new evaluation metric or a new reasoning approach. Instead, it takes established prompting frameworks (GEMBA-MQM and G-Eval) and applies them uniformly to a carefully constructed set of eight models that span the reasoning/non-reasoning spectrum across three parameter scales (8B, 32B, 70B, plus the frontier 600B+ models).

The model selection is deliberate and structured (Section 3, Models):

Frontier reasoning models (DeepSeek-R1, OpenAI o3-mini at three reasoning effort levels) to test whether state-of-the-art reasoning improves evaluation.
Non-reasoning control models (DeepSeek V3, GPT-4o-mini, Qwen-2.5 32B, LLaMA-3.3 70B, LLaMA-3.1 8B) matched as closely as possible to their reasoning counterparts by model family and/or parameter count.
Distilled reasoning models (R1 LLaMA 70B, R1 Qwen 32B, R1 LLaMA 8B) to test whether evaluation-relevant reasoning capabilities survive distillation.

This design enables the paper to disentangle three effects that are often confounded: (1) the effect of reasoning training itself (comparing R1 vs. V3, o3-mini vs. GPT-4o-mini), (2) the effect of model scale (comparing distilled variants at 8B, 32B, 70B, and full 600B+), and (3) the effect of model architecture/family (comparing DeepSeek-based vs. OpenAI-based reasoning approaches).

The paper positions its analysis as hypothesis-generating rather than conclusive: it aims to identify where reasoning helps (which tasks, which quality dimensions, which model families) and where it doesn't, providing actionable guidance for practitioners while opening research questions about why reasoning benefits are so architecture-dependent. The correlation analysis between reasoning token count and evaluation quality (Section 5) further positions the work as contributing to the emerging literature on test-time compute scaling, extending that conversation beyond math and code tasks to subjective quality assessment.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical comparison study that systematically evaluates whether reasoning-enabled large language models make better judges of machine-generated text quality than standard non-reasoning LLMs. It solves the problem of having no systematic evidence about when and where reasoning capabilities benefit NLG evaluation by constructing a controlled experimental framework: take established prompting-based evaluation protocols, apply them identically to a carefully matched set of reasoning and non-reasoning models across multiple parameter scales, and measure how well their quality assessments correlate with human judgments on standard benchmarks for machine translation and text summarization.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has five major stages, each building on the previous one:

Model Selection and Categorization — Eight models are organized into matched reasoning/non-reasoning pairs at comparable parameter scales (8B, 32B, 70B, and the 600B+ frontier tier) across two architectural families (DeepSeek and OpenAI), enabling controlled comparison that disentangles the effects of reasoning training, model scale, and model architecture.
Prompt-Based Evaluation Framework — Two established prompting protocols serve as the evaluation interface: GEMBA-MQM for machine translation (instructing models to detect translation errors across accuracy, fluency, and terminology dimensions, assign severity scores, and produce final quality ratings) and G-Eval for text summarization (instructing models to evaluate summaries along coherence, consistency, relevance, and fluency using explicit criteria and chain-of-thought prompting).
Data Preparation — Standard benchmark datasets provide the source texts, system outputs, and ground-truth human quality judgments: WMT23 Metrics Shared Task dataset for machine translation (covering en-de, he-en, and zh-en language pairs with segment-level human annotations) and SummEval plus Eval4NLP for summarization (covering 16 MT systems on 100 news articles with 4-dimensional quality ratings, plus a held-out dataset with unpublished human scores to prevent data contamination).
Inference and Score Collection — Each model receives identical prompts, generates its evaluation output (including reasoning traces for reasoning models), and produces numerical quality scores. For reasoning models, the complete chain-of-thought deliberation ("reasoning trace") is collected alongside the final score. For non-reasoning models, only direct scores are produced.
Meta-Evaluation and Analysis — Model-generated scores are compared against human judgments using appropriate correlation metrics: segment-level Pearson correlation and system-level pairwise accuracy for machine translation; segment-level Kendall's tau correlation across four quality dimensions for summarization. Additional analysis correlates reasoning token counts with evaluation error and model-predicted scores.

Information flows linearly: a source text and its machine-generated candidate (translation or summary) enter the system → the evaluation prompt template wraps them with task-specific instructions and quality criteria → the selected LLM processes the prompt and produces an evaluation (with or without explicit reasoning) → the output is parsed to extract numerical quality scores → these scores are correlated against human reference judgments → the correlation coefficients become the primary metrics for comparing models.

3.3 Roadmap for the Deep Dive

First, the model selection strategy and the reasoning behind each matched pair, because the entire study's validity depends on having comparable reasoning and non-reasoning models where the primary difference is the reasoning training itself, not scale, architecture, or training data.
Second, the prompt-based evaluation protocols (GEMBA-MQM and G-Eval), since these are the fixed interfaces through which all models interact with the evaluation task—understanding their structure is essential for interpreting what the models are actually being asked to do.
Third, the datasets and their human judgment annotations, because correlation with human judgments is the sole criterion for evaluation quality, and the choice of datasets determines which linguistic phenomena and quality dimensions are tested.
Fourth, the meta-evaluation metrics that convert model outputs into comparable performance numbers, since different correlation coefficients capture different aspects of alignment with human judgments.
Fifth, the reasoning token analysis methodology that probes how reasoning models use their deliberation capacity, moving beyond aggregate performance to mechanistic understanding.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical comparison paper whose core idea is that the utility of reasoning capabilities for NLG evaluation is architecture-dependent and task-dependent, and that a controlled experimental design matching reasoning models to non-reasoning counterparts at comparable scales can reveal where reasoning helps, where it doesn't, and how much of the benefit survives distillation.

Model Selection Strategy and the Matched-Pair Design

The central methodological challenge this paper faces is confounding variables: if you simply compare DeepSeek-R1 (685B parameters, reasoning-trained) against GPT-4o-mini (unknown size, non-reasoning), you cannot tell whether any performance difference comes from reasoning capability, model scale, training data, or architectural differences. The paper addresses this through a deliberately constructed set of matched pairs where the reasoning and non-reasoning variants share as much as possible—model family, parameter count, pretraining data—differing primarily in whether they underwent reasoning-specific post-training.

Frontier reasoning models and their non-reasoning counterparts. The paper selects two state-of-the-art reasoning models as the primary test subjects: DeepSeek-R1 (Guo et al., 2025) and OpenAI o3-mini (with three reasoning effort settings: high, medium, and low). Each is paired with the closest available non-reasoning model from the same family:

DeepSeek-R1 is the reasoning-enabled variant of DeepSeek-V3, fine-tuned using reinforcement learning (specifically Group Relative Policy Optimization, GRPO) to generate explicit chain-of-thought reasoning before producing answers. It has approximately 685B parameters. Its non-reasoning counterpart is DeepSeek-V3, the base model from which R1 was derived. This is the cleanest comparison in the study because both models share the same pretraining, the same architecture, and approximately the same parameter count—the only systematic difference is the RL-based reasoning post-training.
OpenAI o3-mini is a reasoning model with controllable reasoning effort. The paper uses three settings: o3-mini-high, o3-mini-medium, and o3-mini-low, which vary how many reasoning tokens the model generates before producing its final answer. Its presumed non-reasoning counterpart is GPT-4o-mini, which the authors describe as "presumed non-reasoning variant from the GPT-4 family." This pairing is acknowledged as less clean than the DeepSeek pairing because "there is no public confirmed information about the size of either of these models, thus this cannot be verified" (Limitations, point 1). The assumption is that they are comparable in capability tier, but this remains an assumption.

Distilled reasoning models and their non-reasoning counterparts. To test whether evaluation-relevant reasoning capabilities survive distillation to smaller scales, the paper includes three distilled variants of DeepSeek-R1 paired with non-reasoning models of comparable size:

R1 LLaMA 70B (distilled from DeepSeek-R1 into the LLaMA architecture at 70B parameters) compared against LLaMA 3.3 70B, a non-reasoning instruction-tuned model at the same parameter scale and from the same model family (LLaMA). The comparison isolates the effect of reasoning distillation within the LLaMA architecture at 70B scale.
R1 Qwen 32B (distilled from DeepSeek-R1 into the Qwen architecture at 32B parameters) compared against Qwen-2.5 32B, a non-reasoning instruction-tuned model at the same scale and from the same Qwen family. This tests whether reasoning distillation works at the 32B scale within the Qwen architecture.
R1 LLaMA 8B (distilled into LLaMA at 8B parameters) compared against LLaMA 3.1 8B, a non-reasoning instruction-tuned model at the smallest tested scale. This pair tests whether there is a lower bound on model capacity below which reasoning distillation fails to preserve evaluation capabilities.

Why this particular selection. The paper's model selection strategy is designed to disentangle three effects that are typically confounded in comparisons between reasoning and non-reasoning models:

The effect of reasoning training itself: By comparing DeepSeek-R1 vs. DeepSeek-V3 (same architecture, same scale, different post-training) and o3-mini vs. GPT-4o-mini (same family, similar tier, different post-training), the paper can attribute performance differences to the reasoning training rather than to architecture or scale.
The effect of model scale on reasoning distillation: By comparing distilled variants at 8B, 32B, and 70B against both the original R1 (685B) and against their same-scale non-reasoning counterparts, the paper can identify the scale threshold below which distillation fails. The finding that R1 Qwen 32B achieves "97% of original R1 quality on Eval4NLP" while R1 LLaMA 8B shows "50.4% reduction" (Section 6) is only meaningful because the 32B and 8B comparisons use the same distillation source (DeepSeek-R1) with comparable non-reasoning baselines.
The effect of model architecture/family on reasoning effectiveness: By comparing DeepSeek-based reasoning models (R1, R1-LLaMA-70B, R1-Qwen-32B) against OpenAI-based reasoning models (o3-mini variants), the paper can test whether the benefits of reasoning are architecture-specific. The finding that "OpenAI o3-mini models demonstrate superior performance over their non-reasoning equivalent" while "DeepSeek-R1 mostly underperforms compared to its non-reasoning counterpart" (Section 6) is the key architecture-dependent result that this design enables.

What the paper does NOT control for. Several important confounds remain that the authors acknowledge: (1) The exact parameter counts of OpenAI models are unknown, so the o3-mini/GPT-4o-mini size comparison is unverified. (2) The pretraining data composition for different model families differs in unknown ways, and multilingual pretraining data in particular could affect MT evaluation performance independently of reasoning capability. (3) The specific reinforcement learning algorithms and reward functions used for reasoning post-training differ between DeepSeek (GRPO) and OpenAI (proprietary), so "reasoning" is not a uniform treatment. (4) The distilled models use different base architectures (LLaMA vs. Qwen), introducing architectural variation alongside scale variation in the distillation comparisons.

Prompt-Based Evaluation Protocols

The paper uses two established prompting frameworks as the fixed interface through which all models perform evaluation. By keeping the prompts identical across all models, any performance differences must be attributed to the models themselves rather than to prompt engineering variations.

GEMBA-MQM for Machine Translation Evaluation. GEMBA-MQM (Kocmi and Federmann, 2023a) is a prompting-based metric that instructs an LLM to evaluate machine translation quality by detecting and classifying translation errors. The paper follows the template described in the original GEMBA-MQM paper without modification.

The prompt provides the model with:

The source text in the original language
The machine-translated hypothesis text (the candidate translation to be evaluated)
Instructions to identify errors across specified quality dimensions
A structured output format requiring error categorization and severity assignment

The model's task is to compare the source and hypothesis texts, identify any discrepancies that constitute translation errors, classify each error into a category (accuracy, fluency, terminology, style, etc.), assign a severity level (critical, major, minor), and produce an overall quality score based on the severity and count of identified errors.

What "MQM" means and why it matters. MQM (Multidimensional Quality Metrics) is a framework for translation quality assessment that decomposes quality into fine-grained error categories rather than producing a single holistic score. By prompting LLMs to perform MQM-style evaluation, GEMBA-MQM asks the model to do something structurally similar to what human annotators do: find specific errors, name their type, and rate their severity. This is inherently a multi-step reasoning task—you cannot correctly identify that "weiter südlich" mistranslates "furthest south" without (1) understanding both phrases, (2) recognizing the semantic mismatch, (3) categorizing it as an accuracy error, and (4) judging its severity. This makes it a natural testbed for reasoning models because the task structure matches what reasoning training aims to improve.

G-Eval for Text Summarization Evaluation. G-Eval (Liu et al., 2023) is a prompting-based metric that instructs an LLM to evaluate text summaries along multiple quality dimensions using explicit evaluation criteria. The paper uses the prompts detailed in the original G-Eval paper.

The prompt provides the model with:

The original source text (a news article)
The system-generated summary to be evaluated
Dimension-specific evaluation criteria (e.g., for coherence: "the summary should be well-structured and well-organized, not just a heap of related information")
A request for both a numerical score (typically 1–5) and a textual explanation

The model evaluates the summary independently on four dimensions:

Coherence: how well the summary flows as a coherent piece of text
Consistency: whether the summary contains information that contradicts or is not supported by the source text (i.e., factual faithfulness)
Relevance: whether the summary includes only important information from the source
Fluency: the grammatical and stylistic quality of the summary text

Why G-Eval is a strong test for reasoning. Unlike GEMBA-MQM, which already has an error-detection structure that naturally maps to multi-step reasoning, G-Eval's chain-of-thought prompting is more open-ended. The model must generate its own reasoning structure to assess each quality dimension. The paper notes that G-Eval "provides explicit criteria for each dimension and asks the model to generate both numerical scores and explanations, making it particularly suitable for testing the impact of reasoning capabilities." This is because reasoning models trained to produce step-by-step deliberation might naturally produce better-structured explanations and more carefully justified scores than non-reasoning models that simply generate a score token.

Prompt template consistency. The paper emphasizes that "all models were evaluated using the same prompting templates to ensure fair comparison." This is a critical methodological choice: if different models received different prompts optimized for their specific capabilities, it would be impossible to tell whether performance differences came from the models or from prompt quality. By holding prompts constant, the paper isolates model capability as the independent variable. However, this also means that prompts are not optimized for reasoning models specifically—the GEMBA-MQM and G-Eval templates were designed for standard instruction-tuned models, and reasoning models might perform differently (possibly better) with prompts that explicitly leverage their reasoning capabilities (e.g., asking them to "think step by step about each potential error" rather than simply following the existing instruction format).

Dataset Selection and Human Judgment Ground Truth

The paper's evaluation quality assessment depends entirely on how well model-generated scores correlate with human judgments. The choice of datasets with reliable, fine-grained human annotations is therefore central to the methodology.

WMT23 Metrics Shared Task Dataset for Machine Translation. The paper uses the dataset from the WMT23 Metrics Shared Task (Freitag et al., 2023), which provides human quality judgments for machine translation outputs. The dataset includes:

Source texts in multiple languages
Reference translations (human-produced)
System translations from multiple MT systems (the candidates to evaluate)
Human quality judgments at the segment level (individual sentence pairs)
Span-level error annotations (human annotators marked specific spans containing errors)

The paper selects three language pairs from this dataset to "evaluate across diverse linguistic structures": en-de (English to German, a high-resource pair with relatively similar language structures), he-en (Hebrew to English, a lower-resource pair with typologically distinct languages and different scripts), and zh-en (Chinese to English, a pair with fundamentally different writing systems and grammatical structures). These three pairs test different aspects of multilingual evaluation capability:

En-de requires understanding of German morphology (case marking, compound nouns, verb-final clause structure) and how English semantic content maps onto German syntactic constructions.
He-en requires handling of a right-to-left script, Semitic morphology (root-and-pattern system), and substantial structural differences in how tense, definiteness, and possession are expressed.
Zh-en requires handling of a logographic writing system, radically different information structuring (topic-comment rather than subject-predicate), and zero morphology for tense and number.

SummEval Dataset for Summarization. The paper uses the SummEval dataset (Fabbri et al., 2021), which is a standard benchmark for summarization evaluation. It contains:

100 news articles from the CNN/DailyMail dataset (the source texts)
Summaries from 16 different summarization systems (the candidates to evaluate)
Human judgments on four quality dimensions: coherence, consistency, relevance, and fluency
Each dimension is rated on a 1–5 Likert scale by human annotators (1 = worst, 5 = best)

The use of 16 different systems creates substantial diversity in summary quality and error types: some systems may produce fluent but factually inconsistent summaries, others may be factually faithful but poorly organized, and human evaluation across all four dimensions disentangles these quality aspects.

Eval4NLP Dataset for Held-Out Summarization Evaluation. The paper additionally employs the Eval4NLP dataset (Leiter et al., 2023) for summarization evaluation. The critical methodological feature of this dataset is that "the human judgement scores for Eval4NLP are not published online, which allows us to evaluate reasoning and non-reasoning models with no data contamination." This addresses a significant concern in LLM evaluation: if models are trained on internet text that includes benchmark evaluation data (including human judgments and discussions of those judgments), their apparent performance on that benchmark may reflect memorization rather than genuine evaluation capability.

By using Eval4NLP as a held-out test set with unpublished human scores, the paper can check whether performance patterns observed on SummEval generalize to unseen evaluation data. The fact that model rankings are largely consistent between SummEval and Eval4NLP (e.g., DeepSeek V3 outperforms DeepSeek R1 on both, o3-mini variants achieve the highest Eval4NLP scores) strengthens confidence that the results reflect genuine evaluation capability rather than benchmark contamination.

What constitutes "ground truth" in this setting. Human quality judgments serve as the ground truth for meta-evaluation, but it is important to understand what these judgments represent. For MT, the human annotations are segment-level quality scores derived from MQM error analysis—human annotators identified specific errors, categorized them, and assigned severity, from which an overall score is computed. For summarization, the judgments are direct Likert-scale ratings on each quality dimension by multiple annotators. The "ground truth" is therefore the aggregate of human annotator judgments, which themselves have some variance (inter-annotator agreement is not perfect). The meta-evaluation metrics measure how well model scores correlate with this human consensus, not with some objective "true quality."

Meta-Evaluation Metrics: Measuring Alignment with Human Judgments

Once each model produces quality scores for translations or summaries, the paper must quantify how well those scores align with human judgments. Different meta-evaluation metrics capture different aspects of this alignment.

Segment-Level Pearson Correlation for Machine Translation. For MT evaluation on WMT23, the paper computes segment-level Pearson correlation ( $\rho$ ) between model-predicted scores and human judgment scores for individual sentence pairs (segments).

The Pearson correlation coefficient is computed as:

$\rho = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2} \sqrt{\sum_{i=1}^{n} (y_i - \bar{y})^2}}$

where $x_i$ is the model-predicted score for segment $i$ , $y_i$ is the human judgment score for segment $i$ , $\bar{x}$ and $\bar{y}$ are the respective means, and $n$ is the number of segments.

What it computes: Pearson's $\rho$ measures the linear relationship between two variables on a scale from -1 to +1. A value of +1 means the model scores and human scores move in perfect lockstep (when the model scores a segment higher, humans also scored it higher); 0 means no linear relationship; -1 means perfect inverse relationship. The statistic uses the covariance between model and human scores (numerator) normalized by the product of their individual standard deviations (denominator).

Why this form: Pearson correlation assumes a linear relationship between model scores and human judgments, which is a reasonable assumption if both the model and humans are measuring the same underlying quality construct. It is sensitive to the relative ordering of segments: if model A consistently rates segment X higher than segment Y while humans rate Y higher than X, the correlation will be penalized. Alternative metrics like Spearman rank correlation would only care about ordering (not linearity), but Pearson is standard in WMT meta-evaluation because it captures both ranking and calibration.

Segment-Level Kendall's Tau for Summarization. For TS evaluation on SummEval, the paper computes segment-level Kendall's tau ( $\tau$ ) correlation between model scores and human judgments across each quality dimension (coherence, consistency, relevance, fluency).

Kendall's tau is a rank correlation coefficient computed as:

$\tau = \frac{C - D}{\binom{n}{2}}$

where $C$ is the number of concordant pairs (pairs of segments where the model and human agree on which segment scores higher), $D$ is the number of discordant pairs (pairs where they disagree), and $\binom{n}{2}$ is the total number of segment pairs.

What it computes: Kendall's tau measures the ordinal association between two rankings—it asks: if you take any two segments, how often does the model correctly predict which one humans preferred? The statistic ranges from -1 (perfect disagreement on every pair) to +1 (perfect agreement). It counts each pair of segments exactly once, comparing their relative ordering in model scores vs. human scores.

Why this form for summarization: Unlike Pearson, Kendall's tau does not assume linearity or interval-scale properties of the scores—it only cares about relative rankings. This is appropriate for the SummEval Likert-scale ratings (1–5) because the distance between a score of 2 and 3 may not represent the same quality difference as between 4 and 5. Rank correlation is more robust to this non-linearity. The paper's choice of Kendall's tau specifically (rather than the more common Spearman's rho) follows the convention established in the G-Eval paper and SummEval benchmark, enabling direct comparison with prior work.

System-Level Pairwise Accuracy for Machine Translation. In addition to segment-level correlation, the paper computes system-level pairwise accuracy for MT evaluation, following Deutsch et al. (2023). This metric measures how often the evaluation metric correctly ranks two translation systems relative to each other.

Pairwise accuracy is computed as:

$\text{Accuracy} = \frac{\text{Number of correctly ordered system pairs}}{\text{Total number of system pairs compared}}$

where a system pair is correctly ordered if the metric assigns a higher aggregate score to the system that human judges preferred.

What it computes: Instead of evaluating how well the metric scores individual segments, pairwise accuracy evaluates whether the metric can answer the practical question: "which of these two MT systems is better overall?" For each pair of systems, the metric's aggregate scores (computed by averaging segment-level scores across the test set) are compared to human aggregate judgments. If the metric and humans agree on which system performs better, the pair is counted as correct.

Why this matters alongside segment-level correlation: Segment-level correlation measures fine-grained alignment (does the metric track quality variations within a single system's output?), while system-level pairwise accuracy measures coarse-grained utility (can the metric tell good systems from bad ones?). A metric can have high segment-level correlation but poor system-level accuracy if its errors are systematically biased in ways that cancel out at the aggregate level, or vice versa. The paper reports both because they capture different aspects of evaluation quality relevant to different use cases: segment-level correlation matters for diagnostic analysis of specific translations; system-level accuracy matters for benchmarking and model selection.

The mt-metrics-eval package. The paper states that "all measures were computed using mt-metrics-eval package," referencing the Google Research repository at https://github.com/google-research/mt-metrics-eval. This is important for reproducibility: different implementations of Pearson correlation can handle edge cases (e.g., what to do when all scores are identical) differently, and using a standard package ensures consistency with the broader MT evaluation literature.

Reasoning Token Analysis Methodology

Beyond aggregate performance comparisons, the paper conducts a mechanistic analysis of how reasoning models use their deliberation capacity. This analysis operates on the reasoning traces that reasoning models generate before producing their final evaluation scores.

What reasoning tokens are. When a reasoning model like DeepSeek-R1 or o3-mini processes an evaluation prompt, it does not immediately produce a score. Instead, it first generates a sequence of "thinking" tokens—visible as text enclosed in special delimiters (e.g., thinking and response for DeepSeek-R1)—that represents the model's step-by-step deliberation. These tokens are not shown to end users but are available in the API response. The paper counts these tokens as a measure of "how much reasoning" the model performed on a given input.

Correlation with evaluation error. For each reasoning model, the paper computes the Pearson correlation between the number of reasoning tokens generated for a particular segment and the absolute difference between the model's predicted score and the human judgment score for that segment:

$\rho_{\text{error}} = \text{Pearson}(\text{reasoning\_token\_count}, |\text{model\_score} - \text{human\_score}|)$

What it computes: This correlation answers the question: "when the model spends more tokens thinking, does it tend to get closer to the human judgment?" A negative correlation indicates that longer reasoning is associated with smaller errors—the model benefits from additional deliberation. A near-zero correlation indicates that reasoning length is unrelated to accuracy.

Why this is important: This analysis directly tests the test-time compute scaling hypothesis for NLG evaluation. In domains like mathematics and coding, allowing reasoning models to generate more reasoning tokens consistently improves performance (Muennighoff et al., 2025). Whether this transfers to subjective quality assessment is unknown: evaluation requires judgment rather than deduction, and there may be diminishing returns or even negative effects from over-analyzing text quality. By measuring this correlation across different model architectures and tasks, the paper provides evidence about where test-time scaling helps and where it doesn't for NLG evaluation.

Correlation with model-predicted scores. The paper also computes the Pearson correlation between reasoning token count and the model's own predicted quality score:

$\rho_{\text{score}} = \text{Pearson}(\text{reasoning\_token\_count}, \text{model\_score})$

What it computes: This correlation answers the question: "do models tend to think more when they encounter text they perceive as lower quality?" A negative correlation (more tokens → lower score) means the model engages in more extensive deliberation when it identifies problems—it spends more effort justifying why a particular translation or summary is flawed.

Why this is informative about model behavior: This analysis reveals something about the model's internal decision process that aggregate performance metrics cannot. If a model consistently generates more reasoning tokens for lower-quality translations, it suggests that the model's reasoning process is genuinely about error detection and justification—the model is "thinking harder" about problematic cases because there is more to analyze. If there is no such correlation, the reasoning trace might be largely boilerplate or unrelated to the actual quality assessment. The paper finds strong negative correlations for o3-mini models (e.g., -0.474 for o3-mini-high on MT) and moderate correlations for DeepSeek distillations, but near-zero correlation for the original DeepSeek-R1—a pattern that informs the architecture-dependent effectiveness conclusion.

Model-specific handling of reasoning traces. The paper notes an important methodological asymmetry: "While the inference API of o3-mini allows specifying the reasoning effort (either 'low', 'medium' or 'high'), there is no such option for DeepSeek R1 and its distilled variants." This means that the o3-mini analysis can compare different reasoning budgets within the same model architecture (controlling for everything except how long the model thinks), while the DeepSeek analysis can only observe the model's naturally chosen reasoning length. The paper suggests in Limitations that "further studies might examine R1 performance at different reasoning efforts by implementing forced generation of the closing response token upon generating a predetermined amount of reasoning tokens," but this was not done in the current study.

Experimental Protocol and Reproducibility Considerations

Inference temperature and decoding. The paper does not explicitly state the decoding parameters (temperature, top-p, top-k) used for model inference, which is a notable omission for reproducibility. In evaluation tasks, deterministic decoding (temperature = 0) is typically preferred to ensure consistent scores across runs, but this is not confirmed. The three o3-mini reasoning effort settings (high, medium, low) likely affect not just the presence of reasoning but also sampling parameters, though the paper does not detail how these settings translate to specific generation parameters.

Cross-model consistency. All models received the same prompts, but the models differ in their instruction-following capabilities and output formats. The paper does not describe any output parsing failures or cases where models failed to produce valid scores in the expected format. This is relevant because if reasoning models occasionally produce scores in non-standard formats or include scores within their reasoning traces rather than in the designated response section, systematic parsing errors could bias the results.

Statistical significance. The paper reports that for the reasoning token correlation analysis (Table 2), correlations marked with ‡ are "not statistically significant (p > 0.05)" while all other correlations are "significant at p < 0.001." The paper does not report confidence intervals or significance tests for the main evaluation quality comparisons (Tables 1), which makes it difficult to assess whether observed differences between models (e.g., o3-mini-high at 0.577 vs. GPT-4o-mini at 0.410 for en-de) are statistically reliable or could arise from sampling variability in the 500-segment test sets.

Computational cost accounting. The paper does not report inference cost (FLOPs, API cost, or wall-clock time) for the different models, although the abstract mentions this as relevant context ("distilled reasoning models can maintain evaluation capabilities while reducing computational costs"). The reasoning token analysis in Section 5 provides token counts as a proxy for cost, but absolute cost comparisons between, say, o3-mini-high and GPT-4o-mini would depend on their respective parameter counts (unknown for both models) and per-token pricing, which the paper does not address.

Summary of Design Choices and Their Justifications

Matched-pair model selection over independent model evaluation: enables causal attribution of performance differences to reasoning training by controlling for scale, architecture, and family, though the matching is imperfect for OpenAI models due to undisclosed parameter counts.
Identical prompts across all models over model-specific prompt optimization: ensures that performance differences come from models rather than prompt quality, but may underestimate reasoning model performance if reasoning-specific prompts would be more effective.
Three language pairs (en-de, he-en, zh-en) for MT over a single pair: tests whether reasoning benefits are consistent across diverse linguistic structures, high-resource vs. low-resource settings, and different script systems.
Both SummEval and Eval4NLP for TS over a single dataset: uses Eval4NLP's unpublished human scores to control for potential data contamination effects that could inflate SummEval performance.
Four quality dimensions for summarization (coherence, consistency, relevance, fluency) over a single aggregate score: enables fine-grained analysis of which specific evaluation capabilities reasoning improves or degrades, revealing that reasoning selectively benefits consistency and fluency while sometimes hurting relevance.
Both segment-level correlation and system-level pairwise accuracy for MT over a single meta-evaluation metric: captures both fine-grained quality tracking (segment-level) and practical system ranking utility (system-level).
Kendall's tau for summarization over Pearson: appropriate for Likert-scale ordinal data where the distance between adjacent scores may not represent equal quality intervals.
Reasoning token correlation analysis as a mechanistic probe rather than just aggregate performance comparison: moves beyond "does reasoning help?" to "how does reasoning help?" by testing whether thinking longer reduces error and whether models allocate more thinking to lower-quality inputs.
No modification of existing prompting frameworks (GEMBA-MQM, G-Eval): tests reasoning models as drop-in replacements for existing evaluation pipelines rather than requiring new prompt engineering, maximizing practical applicability of the findings.

4. Key Insights and Innovations

Innovation 1: Reasoning Capability Is Neither Uniformly Beneficial Nor Uniformly Harmful for NLG Evaluation — It Is Architecture-Dependent and Dimension-Selective

The paper's most fundamental intellectual contribution is not a new method but a diagnostic finding that refutes two opposing simplistic narratives about reasoning in LLMs. Before this work, one could reasonably hold either of two positions: (1) reasoning models, having demonstrated superiority on math, logic, and coding benchmarks, should naturally excel at any task requiring careful analysis — and NLG evaluation, which demands error detection, severity assessment, and multi-dimensional quality weighing, fits that description; or (2) reasoning models, trained via reinforcement learning on deductive tasks with objective correctness criteria, might perform poorly on subjective quality assessment where "correctness" is a matter of human judgment and over-analysis could introduce noise (consistent with Liu et al., 2024 and Sprague et al., 2025 showing that CoT can hurt on non-symbolic tasks).

The paper shows that both narratives are wrong because the truth is architecture-specific and dimension-specific. The OpenAI o3-mini family demonstrates the first narrative: o3-mini-high achieves a segment-level Pearson correlation of 0.577 on en-de MT evaluation versus 0.410 for its non-reasoning counterpart GPT-4o-mini — a 40.7% improvement (Table 1). This is a large effect that supports the intuition that systematic deliberation helps identify translation errors. Yet the DeepSeek family simultaneously demonstrates the second narrative: DeepSeek-R1 achieves 0.364 on the same en-de task versus 0.490 for its non-reasoning counterpart DeepSeek-V3 — a 25.7% degradation (Table 1). The same reasoning training that produced gains elsewhere produces losses here.

What makes this genuinely novel rather than a simple "mixed results" finding is the patterned selectivity of when reasoning helps within each architecture. On summarization, DeepSeek-R1 dramatically outperforms DeepSeek-V3 on consistency evaluation (0.565 vs. 0.331 Kendall's τ, a 70.7% improvement) while substantially underperforming on relevance (0.303 vs. 0.446, a 32.1% drop) and coherence (0.381 vs. 0.462, a 17.5% drop). The reasoning model is not simply better or worse — it is selectively better at detecting factual contradictions (consistency checks require comparing the summary against the source text for unsupported claims, a multi-step verification task that maps naturally to reasoning) while being worse at holistic quality assessment (coherence and relevance involve more gestalt judgments that may resist step-by-step decomposition). This dimension-selectivity is a conceptual advance because it suggests that reasoning capabilities are not a generic "better analysis" upgrade but rather a specific cognitive tool that helps with certain sub-tasks and interferes with others.

This finding fundamentally changes how the field should think about deploying reasoning models for evaluation: the question is not "should we use reasoning models as judges?" but rather "for which dimensions, in which architectures, on which tasks should we use reasoning models as judges?" The paper does not answer this question fully — it provides evidence for two architectures on two tasks — but it establishes the question as the right one to ask and provides the first systematic evidence that the answer is non-trivial.

Prior work context. Prior to this study, the LLM-as-a-Judge literature (Gu et al., 2024; Huang et al., 2024; Li et al., 2025; Kocmi and Federmann, 2023a; Liu et al., 2023) evaluated exclusively non-reasoning models. The reasoning LLM literature (Guo et al., 2025; Arora and Zanette, 2025; Shao et al., 2024) focused on math, coding, and formal logic — domains with objective ground truth. Neither literature had reason to predict the architecture-dependent, dimension-selective pattern this paper documents. The paper therefore establishes a new empirical phenomenon that neither community's prior findings anticipated.

Evidence anchoring. The architecture-dependent claim is anchored in Table 1, specifically the within-family comparisons (R1 vs. V3, o3-mini vs. GPT-4o-mini, each distilled variant vs. its non-reasoning counterpart). The dimension-selectivity claim is anchored in the SummEval per-dimension results where R1's consistency advantage (0.565 vs. 0.331) coexists with relevance disadvantage (0.303 vs. 0.446). The reasoning effort gradient supporting the test-time scaling interpretation is anchored in the o3-mini variants: for en-de MT, o3-mini-high (0.577) > o3-mini-medium (0.517) > o3-mini-low (0.471), showing monotonic improvement with reasoning effort.

Innovation 2: Reasoning Token Count Analysis as a Mechanistic Probe Reveals Two Distinct Modes of Deliberation — Accuracy-Improving vs. Effort-Allocating

The paper's second conceptual contribution is the introduction and application of a simple but powerful diagnostic tool: correlating reasoning token counts with evaluation error and with model-assigned scores to reveal how reasoning models use their deliberation budget. This analysis (Section 5, Table 2) uncovers two qualitatively different behaviors that would be invisible from aggregate performance metrics alone.

First mode: thinking longer reduces error (accuracy-improving deliberation). For OpenAI o3-mini models on MT evaluation, there is a statistically significant negative correlation between reasoning token count and evaluation error: -0.118 (high), -0.110 (medium), -0.091 (low). When the model generates more reasoning tokens, its predicted scores get closer to human judgments. This is evidence for the test-time compute scaling hypothesis transferring from math/code domains to subjective quality assessment — allowing the model to deliberate longer consistently improves accuracy. The monotonic relationship with reasoning effort setting (stronger correlation at higher effort) strengthens this interpretation.

Second mode: thinking harder about perceived problems without necessarily being more accurate (effort-allocating deliberation). Across nearly all reasoning models, there is a strong negative correlation between reasoning token count and model-assigned quality scores: o3-mini-high shows -0.474 on MT and -0.209 on TS; R1 LLaMA 70B shows -0.208 on MT and -0.221 on TS; R1 Qwen 32B shows -0.150 on MT and -0.216 on TS. Models consistently allocate more reasoning tokens when they identify (or perceive) lower-quality texts. This is not the same as being more accurate — the effort-allocation pattern exists even for models like DeepSeek-R1 (where the error correlation is effectively zero: -0.015 on MT, -0.054 on TS), meaning R1 thinks longer about texts it considers low-quality without that extra thinking improving its alignment with human judgments.

The conceptual significance is that these two modes are empirically separable. A model can exhibit effort-allocation (thinking more about problematic cases) without accuracy-improvement (that thinking doesn't make its judgments more human-like). This distinction matters because effort-allocation alone might be mistaken for effective reasoning — a model that generates longer, more detailed critiques of poor translations might seem to be doing better analysis, but if those critiques don't improve score accuracy, the extra tokens are computationally wasteful for the evaluation task. The paper's correlation framework surfaces this distinction quantitatively.

What makes this a diagnostic innovation rather than just a correlation report is that it operationalizes a mechanistic hypothesis about reasoning model behavior. Prior work on test-time scaling (Muennighoff et al., 2025) measured whether "more thinking improves performance" but did not separately measure "more thinking is allocated to harder cases." The NLG evaluation setting naturally produces both signals because (a) some texts are genuinely harder to evaluate accurately, and (b) models might recognize difficulty and allocate more tokens accordingly. By measuring both correlations, the paper can diagnose whether the allocation-to-difficulty actually translates to improved accuracy. The finding that DeepSeek-R1 allocates tokens to perceived problems (-0.010 score correlation on TS) without error reduction (-0.054 error correlation, essentially zero) suggests that R1's reasoning process is engaging in analysis that is internally coherent but externally misaligned with human quality standards.

Prior work context. Token-level correlation analysis of this type has no direct precedent in the LLM-as-a-judge or reasoning LLM literatures. The closest prior work is the broader test-time scaling literature (Muennighoff et al., 2025), which measures accuracy as a function of reasoning budget but does not decompose the relationship into separate effort-allocation and accuracy-improvement signals.

Evidence anchoring. The two-mode distinction is anchored in Table 2, specifically the contrast between the Error and Score columns. The OpenAI models show both negative error correlations AND strong negative score correlations, indicating that effort-allocation translates into accuracy-improvement. The original DeepSeek-R1 shows effectively zero error correlation (-0.015 MT, -0.054 TS) despite showing score correlations (0.019 MT, -0.010 TS), indicating effort-allocation without accuracy-improvement. The distilled DeepSeek models show an intermediate pattern (moderate score correlations, weak-to-moderate error correlations), suggesting partial transfer of effective reasoning.

Innovation 3: Distillation of Reasoning for NLG Evaluation Has a Sharp Scale Threshold — Sufficient Capacity Exists at 32B Parameters but Not at 8B

The paper's third conceptual contribution is the identification of a scale cliff in reasoning distillation for evaluation tasks. This is not simply "bigger models are better" — the finding is that distilled reasoning models at 32B parameters maintain nearly full evaluation capability (97% of original DeepSeek-R1 quality on Eval4NLP) while those at 8B parameters suffer catastrophic degradation (50.4% reduction in SummEval average, Table 1). The existence of such a sharp threshold between 8B and 32B parameters, rather than a smooth degradation, is the conceptually interesting finding.

What makes this a "cliff" rather than a gradual decline. The performance drop from 32B to 8B is disproportionate to the parameter reduction. R1 Qwen 32B achieves a SummEval average of 0.355 versus the original R1 at 0.351 — essentially no loss. R1 LLaMA 8B achieves 0.174 versus the original R1 at 0.351 — a 50.4% reduction. The parameter reduction from 32B to 8B is only a factor of 4×, yet the capability loss is total: the 8B model loses not just some absolute performance but specifically the reasoning advantages that define the larger models. On consistency evaluation — the dimension where R1 and R1 Qwen 32B excel over their non-reasoning counterparts (0.565 and 0.540 respectively) — the 8B model falls to 0.251, actually below its non-reasoning counterpart LLaMA 3.1 8B (0.312). The distillation has not just compressed the model; it has qualitatively changed its evaluation behavior, eliminating the specific strength that made reasoning valuable.

Why this matters beyond the obvious "bigger is better." Prior work on efficient evaluation metrics (Eddine et al., 2022; Larionov et al., 2023; Rei et al., 2022) established that distillation can reduce model size while preserving evaluation quality for non-reasoning models. The novel contribution here is that reasoning-specific capabilities impose a different, higher capacity requirement than general evaluation quality. A non-reasoning LLaMA 3.1 8B achieves a SummEval average of 0.228 (Table 1) — modest but functional. Its reasoning-distilled counterpart R1 LLaMA 8B achieves 0.174 — worse. The distillation process has not just failed to transfer reasoning advantages; it has actively degraded the base model's evaluation capability. This suggests that forcing a small model to emulate the reasoning patterns of a much larger model (generating extensive chain-of-thought before scoring) creates an interference effect: the model allocates its limited capacity to producing reasoning traces, leaving insufficient representational capacity for the actual quality assessment.

This finding has immediate practical implications for the deployment of reasoning-enhanced evaluation. It establishes 32B as an empirically supported lower bound for models where reasoning distillation preserves evaluation quality, and 8B as a scale where reasoning distillation is actively harmful. For practitioners building cost-efficient evaluation pipelines, this means that efforts to use distilled reasoning models should target the 32B tier or above; below that threshold, a non-reasoning model of the same size may outperform its reasoning-distilled counterpart.

Prior work context. The reasoning distillation literature (Guo et al., 2025) demonstrated that distilled R1 models preserve reasoning capabilities on math and coding benchmarks down to relatively small scales. This paper shows that the transfer does not hold uniformly: evaluation-relevant reasoning has different capacity requirements. The efficient metrics literature (FrugalScore, COMETINHO, xCOMET-lite) showed that non-reasoning evaluation models can be compressed effectively; this paper shows that reasoning evaluation models cannot be compressed to the same degree without losing the specific capabilities that make reasoning valuable.

Evidence anchoring. The scale cliff is anchored in Table 1, comparing three parameter tiers: R1 LLaMA 8B (0.174 SummEval average, 0.368 Eval4NLP) vs. R1 Qwen 32B (0.355 SummEval average, 0.564 Eval4NLP) vs. original DeepSeek-R1 (0.351 SummEval average, 0.583 Eval4NLP). The near-identity of 32B and original R1 performance contrasts sharply with the 8B collapse. The loss of the consistency advantage at 8B is visible in the SummEval per-dimension breakdown: R1 LLaMA 8B consistency at 0.251 vs. LLaMA 3.1 8B at 0.312, a reversal of the pattern seen at larger scales.

Innovation 4: Reasoning Effort Control as a Coarse-Grained Intervention Reveals That Deliberation Quantity Has Architecture-Dependent Effects on Evaluation Quality

The paper's fourth conceptual contribution leverages the controllable reasoning effort feature of OpenAI o3-mini (high, medium, low settings) as a natural experiment to study whether "thinking longer" causally improves evaluation quality within a single model architecture. This is methodologically distinct from correlating naturally-occurring reasoning length with performance (Innovation 2 above) because the effort setting is an experimental intervention rather than an observational variable: the same model, with the same training, on the same inputs, is instructed to generate more or fewer reasoning tokens.

What the causal interpretation reveals that correlation cannot. The correlation analysis shows that within a given effort setting, O3-mini models show negative error correlations (longer reasoning associated with less error). But this could reflect confounding: maybe the model naturally generates more tokens on easier-to-evaluate segments, creating a spurious correlation. The effort-setting intervention addresses this by showing that forcing the model to reason more (switching from low to high effort) produces a monotonic improvement in evaluation quality for MT: o3-mini-high (0.577 en-de) > o3-mini-medium (0.517) > o3-mini-low (0.471) > GPT-4o-mini (0.410, the non-reasoning baseline). Each step of increased reasoning effort yields a measurable gain, with the high-effort setting outperforming the non-reasoning baseline by 40.7%.

The task-dependence of the causal effect. Notably, this causal benefit of increased reasoning effort is strong for MT and weak-to-absent for summarization. On SummEval, o3-mini-high, medium, and low achieve nearly identical average scores (0.337, 0.333, 0.335), and all three slightly underperform GPT-4o-mini (0.346). Increasing reasoning effort provides essentially no benefit for summarization evaluation within the o3-mini architecture. This task-dependence reinforces the paper's central thesis that reasoning benefits are selective, not universal. The causal evidence strengthens this conclusion because it rules out the possibility that the MT benefit is merely correlational — the model genuinely becomes a better translation evaluator when instructed to deliberate more, but does not become a better summarization evaluator.

Why this is a conceptual contribution rather than just a hyperparameter sweep. The three effort settings are not arbitrary hyperparameter choices — they represent a practical instantiation of the test-time compute scaling paradigm that has been primarily studied in math and code domains. By showing that this paradigm transfers to MT evaluation but not to summarization evaluation, the paper establishes a boundary condition on test-time scaling: it helps when the evaluation task decomposes naturally into discrete, identifiable sub-problems (translation errors can be located in specific spans, categorized into well-defined types, and assigned severity based on impact). It may not help when the evaluation task requires more holistic, gestalt judgments that resist decomposition into explicit reasoning steps (summarization coherence and fluency, which require assessing overall textual quality rather than catching discrete errors).

Prior work context. The test-time scaling literature (Muennighoff et al., 2025) demonstrated that controlling reasoning budget improves performance on reasoning benchmarks, but did not test this on subjective evaluation tasks. The LLM-as-a-judge literature has not explored reasoning effort as an experimental variable — prior work using chain-of-thought prompting for evaluation (G-Eval, Liu et al., 2023) used CoT as a fixed prompt feature rather than a variable to be manipulated and studied causally.

Evidence anchoring. The causal effect of reasoning effort on MT evaluation is anchored in Table 1: o3-mini-high (0.577 en-de, 0.568 zh-en) > o3-mini-medium (0.517, 0.505) > o3-mini-low (0.471, 0.491) > GPT-4o-mini (0.410, 0.487). The absence of a comparable effect on summarization is anchored in the SummEval average column: o3-mini-high (0.337) ≈ o3-mini-medium (0.333) ≈ o3-mini-low (0.335) < GPT-4o-mini (0.346). The he-en language pair is an interesting partial exception: o3-mini-high (0.421) and o3-mini-low (0.413) both underperform GPT-4o-mini (0.435), suggesting that even for MT, the reasoning benefit may not hold for all language pairs — a nuance the paper does not fully explore but that is visible in the data.

5. Experimental Analysis

Evaluation Methodology

Dataset. For MT evaluation, the paper uses the WMT23 Metrics Shared Task dataset (Freitag et al., 2023), spanning three language pairs: en-de, he-en, and zh-en. The dataset includes source texts, reference translations, system translations from multiple MT systems, and human quality judgments at the segment level, including span-level error annotations. For TS evaluation, the paper uses the SummEval dataset (Fabbri et al., 2021), which contains human judgments for summaries from 16 different summarization systems on 100 news articles from the CNN/DailyMail dataset, with each summary evaluated along four dimensions (coherence, consistency, relevance, fluency) on a 1–5 Likert scale. Additionally, the paper uses the Eval4NLP dataset (Leiter et al., 2023) as a held-out summarization test set whose human judgment scores are not publicly available, enabling evaluation with no data contamination.
Base model(s). The paper evaluates eight models organized into three categories: (1) state-of-the-art reasoning models — DeepSeek-R1 (685B parameters, RL-trained reasoning variant of DeepSeek-V3 using GRPO) and OpenAI o3-mini (with three controllable reasoning effort settings: high, medium, low); (2) distilled variants of DeepSeek-R1 — R1 LLaMA 70B, R1 Qwen 32B, and R1 LLaMA 8B, representing three model size tiers; (3) non-reasoning control models — DeepSeek-V3 (non-reasoning counterpart to R1), GPT-4o-mini (presumed non-reasoning variant from the GPT-4 family), Qwen-2.5 32B, LLaMA-3.3 70B, and LLaMA-3.1 8B. The selection deliberately creates matched reasoning/non-reasoning pairs at comparable parameter scales to disentangle the effects of reasoning training, model scale, and model architecture.
Metrics. For MT evaluation, the paper computes segment-level Pearson correlation (ρ) between model-predicted scores and human judgment scores for individual sentence pairs, measuring linear alignment. Additionally, it computes system-level pairwise accuracy following Deutsch et al. (2023), which measures how often the metric correctly predicts which of two translation systems is better according to human judgments — computed as the fraction of correctly ordered system pairs over all system pairs. For TS evaluation, the paper computes segment-level Kendall's tau (τ) correlation between model scores and human judgments across the four quality dimensions (coherence, consistency, relevance, fluency). Kendall's tau is a rank correlation coefficient that measures ordinal association: it counts concordant vs. discordant segment pairs without assuming linearity or interval-scale properties of the scores. All MT measures were computed using the mt-metrics-eval package.
Baselines. The primary baselines are the non-reasoning control models matched by model family and/or parameter count: DeepSeek-V3 serves as the baseline for DeepSeek-R1; GPT-4o-mini serves as the presumed baseline for OpenAI o3-mini variants; LLaMA-3.3 70B serves as the baseline for R1 LLaMA 70B; Qwen-2.5 32B serves as the baseline for R1 Qwen 32B; and LLaMA-3.1 8B serves as the baseline for R1 LLaMA 8B. These within-family comparisons are the primary mechanism for isolating the effect of reasoning training.
Generation budget / compute accounting. The paper does not report inference cost in terms of FLOPs, API cost, or wall-clock time. The reasoning token analysis in Section 5 provides reasoning token counts as a proxy for computational effort, measuring the number of "thinking" tokens generated by reasoning models before producing their final evaluation. For o3-mini, the reasoning effort settings (high, medium, low) serve as a controlled intervention on reasoning budget, though the paper does not map these settings to specific token counts or cost figures.
Cross-validation / statistical protocol. The paper does not report cross-validation or train/validation/test splits for strategy selection, as this is not a method development paper that requires hyperparameter tuning. For the reasoning token correlation analysis (Table 2), the paper reports statistical significance: correlations marked with ‡ are "not statistically significant (p > 0.05)" while all other correlations are "significant at p < 0.001." The paper does not report confidence intervals or significance tests for the main evaluation quality comparisons in Table 1. The paper uses identical prompting templates across all models to ensure fair comparison, and all models were evaluated on the same test data.

Main Quantitative Results

Summarization Evaluation: Reasoning Provides Selective, Dimension-Specific Benefits with DeepSeek R1 Excelling Only at Consistency While Non-Reasoning Models Dominate Overall

Table 1 presents the segment-level Kendall's τ correlations between model evaluations and human judgments on SummEval across four quality dimensions (coherence, consistency, relevance, fluency) plus a combined average, and on Eval4NLP as a single summarization quality score.

DeepSeek R1 vs. DeepSeek V3: the consistency exception. DeepSeek-R1 achieves a substantially higher correlation on consistency evaluation compared to its non-reasoning counterpart: 0.565 vs. 0.331 — a 70.7% improvement. This is the largest positive reasoning effect observed anywhere in the study. However, this advantage is confined to a single dimension. On the remaining three SummEval dimensions, DeepSeek-R1 underperforms DeepSeek-V3: coherence (0.381 vs. 0.462, a 17.5% decline), relevance (0.303 vs. 0.446, a 32.1% decline), and fluency (0.157 vs. 0.356, a 55.9% decline). Consequently, DeepSeek-V3 achieves a higher SummEval average (0.399) than DeepSeek-R1 (0.351), and a higher Eval4NLP score (0.630 vs. 0.583). The reasoning model is decisively better at detecting factual contradictions between summary and source (consistency) but substantially worse at holistic quality assessment (coherence, relevance, fluency).

OpenAI o3-mini vs. GPT-4o-mini: reasoning excels at coherence and fluency but not consistency. The OpenAI reasoning models show the opposite dimension-specific pattern to DeepSeek. O3-mini-high dramatically outperforms GPT-4o-mini on coherence (0.482 vs. 0.321, approximately 50% improvement) and on fluency (0.313 vs. 0.263, approximately 19% improvement). However, it substantially underperforms on consistency (0.242 vs. 0.430, a 43.7% decline) and relevance (0.311 vs. 0.370, a 15.9% decline). The SummEval averages are comparable: o3-mini-high (0.337) vs. GPT-4o-mini (0.346), with GPT-4o-mini holding a slight edge. On Eval4NLP, o3-mini-high achieves 0.644 vs. GPT-4o-mini's 0.634 — a small advantage for the reasoning model.

Reasoning effort does not improve summarization evaluation. Across o3-mini variants, there is essentially no variation in SummEval performance with reasoning effort: high (0.337), medium (0.333), low (0.335). The Eval4NLP scores are similarly invariant: high (0.644), medium (0.644), low (0.645). This is a striking null result — increased reasoning effort, which the correlation analysis shows is causally beneficial for MT evaluation (discussed below), has no effect on summarization evaluation quality within the o3-mini architecture.

Distilled models preserve evaluation capability at 32B but collapse at 8B. R1 Qwen 32B essentially matches the original DeepSeek-R1: SummEval average 0.355 vs. 0.351 (R1 actually slightly higher) and Eval4NLP 0.564 vs. 0.583 (97% of original performance). Importantly, R1 Qwen 32B preserves the reasoning advantage on consistency evaluation — it achieves 0.540 vs. Qwen-2.5 32B's 0.449, a 20.3% improvement — showing that the consistency-detection capability survives distillation at 32B scale. In contrast, R1 LLaMA 8B shows catastrophic degradation: SummEval average 0.174 vs. original R1's 0.351 (a 50.4% reduction) and Eval4NLP 0.368 vs. 0.583 (a 36.9% reduction). Critically, the consistency advantage disappears entirely: R1 LLaMA 8B achieves 0.251 vs. LLaMA 3.1 8B's 0.312, meaning the distilled reasoning model is actually worse at consistency evaluation than its non-reasoning counterpart of the same size.

Machine Translation Evaluation: Architecture-Dependent Benefits with OpenAI o3-mini Showing Clear Reasoning Gains and DeepSeek-R1 Showing Consistent Degradation

Table 1 presents segment-level Pearson correlations for three language pairs (en-de, he-en, zh-en) and system-level pairwise accuracy on WMT23.

OpenAI o3-mini: reasoning effort improves MT evaluation monotonically. O3-mini-high achieves the highest segment-level correlations for en-de (0.577) and zh-en (0.568) among all models tested. The three reasoning effort settings show a clear monotonic improvement with increased reasoning for en-de: high (0.577) > medium (0.517) > low (0.471), each substantially exceeding GPT-4o-mini (0.410). The same gradient appears for zh-en: high (0.568) > medium (0.505) > low (0.491) > GPT-4o-mini (0.487). For he-en, the pattern reverses: GPT-4o-mini (0.435) outperforms all o3-mini variants (high: 0.421, medium: 0.404, low: 0.413), suggesting that the reasoning benefit may not extend to all language pairs — specifically, the low-resource and typologically distinct he-en pair shows no reasoning advantage. System-level pairwise accuracy is identical across all OpenAI models (0.928).

DeepSeek-R1 vs. DeepSeek-V3: reasoning degrades MT evaluation across all language pairs. DeepSeek-V3 consistently and substantially outperforms DeepSeek-R1: en-de (0.490 vs. 0.364, a 25.7% degradation), zh-en (0.512 vs. 0.441, a 13.9% degradation), and he-en (0.394 vs. 0.398, essentially tied). System-level accuracy is comparable: R1 at 0.908 vs. V3 at 0.904. The reasoning model is not just failing to improve — it is actively worse at segment-level quality assessment for two of three language pairs.

Distilled models at 70B and 32B sometimes outperform the original R1 on MT. R1 LLaMA 70B achieves better segment-level correlations than the original DeepSeek-R1 on all three language pairs: en-de (0.421 vs. 0.364, a 15.7% improvement), zh-en (0.451 vs. 0.441, a 2.3% improvement), and he-en (0.365 vs. 0.398, a slight decline). R1 Qwen 32B similarly outperforms original R1 on en-de (0.388 vs. 0.364) and zh-en (0.465 vs. 0.441), though it declines on he-en (0.338 vs. 0.398). However, both distilled models still substantially underperform their non-reasoning counterparts: LLaMA 3.3 70B achieves 0.590 en-de vs. R1 LLaMA 70B at 0.421; Qwen-2.5 32B achieves 0.521 en-de vs. R1 Qwen 32B at 0.388. The distillation preserves some evaluation capability relative to the original R1 but does not close the gap with non-reasoning models of the same scale. R1 LLaMA 8B shows the expected degradation: 0.310 en-de, 0.325 he-en, 0.410 zh-en, all below its non-reasoning counterpart LLaMA 3.1 8B (0.476, 0.335, 0.421 respectively).

System-level accuracy shows minimal reasoning/non-reasoning differences. System-level pairwise accuracy is consistently high across all models (0.904–0.944), with Qwen-2.5 32B achieving the highest (0.944) and DeepSeek-V3 the lowest among the top performers (0.904). The small spread (0.04 between highest and lowest) suggests that system-level ranking is a relatively easy task on WMT23 — all models can reliably distinguish good MT systems from bad ones, even when their segment-level quality assessments differ substantially.

Reasoning Token Analysis: Two Distinct Modes of Deliberation — Accuracy-Improving for o3-mini, Effort-Allocating Without Accuracy for DeepSeek-R1

Table 2 presents Pearson correlations between reasoning token count and (1) evaluation error (absolute difference between model-predicted and ground truth scores) and (2) model-assigned scores, computed separately for MT evaluation on WMT23 and TS evaluation on SummEval.

Error correlations: o3-mini shows test-time scaling benefits; DeepSeek-R1 does not. OpenAI o3-mini models exhibit statistically significant negative correlations between reasoning token count and evaluation error for MT: -0.118 (high), -0.110 (medium), and -0.091 (low). When these models generate more reasoning tokens, their predicted scores tend to be closer to human judgments — evidence that increased deliberation improves accuracy. For TS, the error correlations are also negative but smaller: -0.156 (high), -0.090 (medium), and -0.006 (low), with the low-effort correlation not achieving statistical significance (p > 0.05). In contrast, DeepSeek-R1 shows effectively zero error correlations: -0.015 (MT), -0.054 (TS). The distilled models show intermediate patterns: R1 Qwen 32B achieves -0.003 (MT, not significant) and -0.122 (TS, significant); R1 LLaMA 70B achieves -0.042 (MT) and 0.026 (TS, a weak positive correlation).

Score correlations: models consistently allocate more reasoning to lower-quality texts. Nearly all reasoning models show substantial negative correlations between reasoning token count and model-assigned quality scores, meaning they generate more reasoning tokens when they perceive the text as lower quality. The strongest correlations come from o3-mini-high on MT (-0.474), followed by o3-mini-medium (-0.415) and o3-mini-low (-0.347). For TS, the same pattern holds: o3-mini-high (-0.209), o3-mini-medium (-0.189), o3-mini-low (-0.149). DeepSeek distillations show moderate score correlations: R1 LLaMA 70B at -0.208 (MT) and -0.221 (TS); R1 Qwen 32B at -0.150 (MT) and -0.216 (TS). The original DeepSeek-R1 is again the outlier: 0.019 (MT) and -0.010 (TS) — essentially no correlation between how much it thinks and what score it assigns.

The joint pattern reveals an efficiency insight. The combination of strong score correlations with strong error correlations (as seen in o3-mini for MT) indicates that when the model allocates more reasoning to texts it perceives as lower-quality, this additional deliberation genuinely improves alignment with human judgments. The combination of moderate score correlations with weak error correlations (as seen in DeepSeek distillations) indicates that the model allocates more reasoning to perceived problems but this allocation does not consistently translate into more human-aligned scores. The combination of near-zero score correlations with near-zero error correlations (as seen in original DeepSeek-R1) indicates that the model's reasoning process is neither responding to perceived quality differences nor improving evaluation accuracy — the reasoning trace appears disconnected from the evaluation task.

Ablation Studies and Robustness Checks

Language pair diversity as an implicit robustness check: The paper evaluates MT across three language pairs (en-de, he-en, zh-en) spanning different resource levels, script systems, and linguistic structures. The finding that o3-mini reasoning benefits are strongest for en-de and zh-en but absent or reversed for he-en (Table 1) serves as a robustness check against the claim that reasoning universally improves MT evaluation — it does not. This language-pair dependence is a non-trivial finding: he-en involves Hebrew (right-to-left script, Semitic morphology, lower resource), and the failure of reasoning to help on this pair suggests that multilingual evaluation capability may depend more on pretraining data coverage than on reasoning ability.

Summarization dimension diversity as an implicit ablation of task structure: The four SummEval dimensions (coherence, consistency, relevance, fluency) probe qualitatively different evaluation sub-tasks. The finding that reasoning models show dimension-specific patterns — R1 excels at consistency but fails at relevance, o3-mini excels at coherence but fails at consistency (Table 1) — serves as evidence that the effect of reasoning is not a generic "better analysis" improvement but is tightly coupled to the specific cognitive demands of each evaluation dimension. The fact that these dimension-specific patterns are mirrored across model families (both DeepSeek and OpenAI reasoning models show dimension-selectivity, though on different dimensions) strengthens the interpretation that reasoning capabilities transfer selectively rather than uniformly.

Held-out dataset (Eval4NLP) as a contamination check: The Eval4NLP dataset has unpublished human judgment scores, meaning models cannot have memorized these scores during training. The consistency of model rankings between SummEval and Eval4NLP — DeepSeek-V3 outperforms DeepSeek-R1 on both (0.399 vs. 0.351 on SummEval; 0.630 vs. 0.583 on Eval4NLP), o3-mini variants achieve the highest Eval4NLP scores (0.644–0.645) — serves as a robustness check that the SummEval results reflect genuine evaluation capability rather than benchmark contamination. The paper presents this as "allows us to evaluate reasoning and non-reasoning models with no data contamination" but does not report whether model rankings differ between the two datasets, which would be the relevant test of contamination effects.

Reasoning effort as a within-model ablation of deliberation quantity: The three o3-mini reasoning effort settings (high, medium, low) serve as an ablation of how much the model deliberates, controlling for all other aspects of the model (architecture, training data, parameters). The monotonic improvement with reasoning effort on MT en-de (0.577 > 0.517 > 0.471, Table 1) and zh-en (0.568 > 0.505 > 0.491) serves as causal evidence that deliberation quantity itself — not just the presence of reasoning training — determines evaluation quality. The absence of this gradient for he-en, for SummEval, and for Eval4NLP serves as evidence that the benefit of additional deliberation is task-dependent.

Distillation scale as an ablation of capacity requirements for reasoning transfer: The three distilled model sizes (8B, 32B, 70B) serve as an ablation of the relationship between model capacity and the preservation of reasoning-derived evaluation capabilities. The finding that R1 Qwen 32B maintains 97% of original R1 Eval4NLP performance (0.564 vs. 0.583) while R1 LLaMA 8B loses over 50% of SummEval performance (0.174 vs. 0.351, Table 1) identifies a sharp threshold — somewhere between 8B and 32B parameters — below which reasoning distillation becomes actively harmful rather than merely less beneficial. The paper does not test intermediate scales (e.g., 13B, 20B), so the exact threshold location is not precisely identified.

Model family as an implicit ablation of reasoning training methodology: The comparison between DeepSeek-based reasoning models (R1 and its distillates, trained via GRPO) and OpenAI o3-mini (trained via proprietary methods) serves as an ablation of how reasoning is implemented. The finding that these two families show qualitatively different patterns on the same evaluation tasks — R1 degrades MT performance while o3-mini improves it, R1 excels at consistency while o3-mini excels at coherence (Table 1) — establishes that "reasoning" is not a uniform capability. The specific reinforcement learning algorithm, reward function, and training data used for reasoning post-training determine which evaluation capabilities are enhanced and which are degraded.

Critical Assessment

Claim 1: "OpenAI o3-mini models show improved performance with increased reasoning on MT." This claim is well-supported for en-de and zh-en, where the monotonic gradient (high > medium > low > non-reasoning baseline) is clear from Table 1. However, it does not hold for he-en, where GPT-4o-mini (0.435) outperforms all o3-mini variants (0.421 high, 0.404 medium, 0.413 low). The paper acknowledges this in the text: "interestingly, GPT-4o-mini performs comparatively well on low-resource language pair he-en (0.435), outperforming its reasoning counterparts" (Section 4.2), but the claim as stated in the abstract and conclusions generalizes beyond what the data support. The correct statement is that o3-mini reasoning improves MT evaluation for high-resource language pairs (en-de, zh-en) but not for the low-resource he-en pair. The paper does not test enough language pairs to characterize which linguistic properties determine whether reasoning helps — is it resource level, script similarity, morphological complexity, or something else? Three language pairs (two showing benefit, one not) is insufficient to draw conclusions about the boundary conditions.

Claim 2: "DeepSeek-R1 generally underperforms compared to its non-reasoning variant except in summarization consistency evaluation." This claim is strongly supported. Table 1 shows DeepSeek-R1 underperforming DeepSeek-V3 on all MT language pairs (en-de: 0.364 vs. 0.490; zh-en: 0.441 vs. 0.512; he-en: 0.398 vs. 0.394, the one near-tie) and on three of four SummEval dimensions (coherence, relevance, fluency). The consistency exception is dramatic (0.565 vs. 0.331, a 70.7% improvement) and the paper correctly highlights it. However, the paper does not investigate why consistency is the exception. Is it because consistency evaluation requires fact-verification across documents (a reasoning-heavy sub-task), while coherence and fluency require more holistic stylistic judgment? Is it because the consistency evaluation in SummEval specifically rewards a particular reasoning pattern that R1's training incidentally produces? The paper identifies the pattern but leaves the mechanism unexplained, which limits the actionable guidance for practitioners — if you don't know why R1 is good at consistency, you cannot predict whether it will be good at consistency evaluation on other datasets or for other text types.

Claim 3: "Reasoning token usage correlates with evaluation quality only in specific models." This claim is supported by Table 2 but requires careful specification. For MT evaluation, the error correlations are significant for o3-mini variants (-0.118, -0.110, -0.091) and essentially zero for DeepSeek-R1 (-0.015). For TS evaluation, the pattern is more mixed: o3-mini-high shows -0.156, R1 Qwen 32B shows -0.122, but o3-mini-low shows -0.006 (not significant). The claim that correlation exists "only in specific models" is accurate but the paper could have been more precise about which models show which pattern for which task. Additionally, the paper does not address a potential confound: reasoning token count correlates with text length (longer source texts or longer translations might naturally elicit both more reasoning tokens and different evaluation difficulty), which could create a spurious correlation between token count and error that is not causal. Controlling for text length would strengthen this analysis.

Claim 4: "Distillation maintains reasonable performance up to 32B parameter models but degrades substantially at 8B scale." This claim is well-supported by Table 1 for the specific models and tasks tested. R1 Qwen 32B achieves SummEval average 0.355 (vs. 0.351 for original R1) and Eval4NLP 0.564 (vs. 0.583, 97% of original). R1 LLaMA 8B achieves SummEval average 0.174 (50.4% reduction) and Eval4NLP 0.368 (36.9% reduction). However, two important caveats weaken the generalizability of this finding. First, the 32B and 8B distilled models use different base architectures — Qwen for the 32B, LLaMA for the 8B — so the performance difference confounds scale with architecture. It is possible that Qwen is simply a better base model for distillation than LLaMA at any given scale, and that a hypothetical R1 Qwen 8B might perform better than R1 LLaMA 8B, or that R1 LLaMA 32B might perform worse than R1 Qwen 32B. The paper's experimental design does not disentangle architecture from scale in the distillation comparisons. Second, the 70B distilled model (R1 LLaMA 70B) actually shows mixed results: it underperforms original R1 on SummEval average (0.315 vs. 0.351) and Eval4NLP (0.556 vs. 0.583), and it underperforms its own non-reasoning counterpart LLaMA 3.3 70B (0.375 SummEval, 0.624 Eval4NLP) — so "reasonable performance up to 32B" is more accurately stated as "the 32B Qwen-distilled variant specifically maintains performance, while the 70B LLaMA-distilled variant and the 8B LLaMA-distilled variant both show degradation, but we cannot tell whether this is due to scale or architecture."

Methodological weaknesses that qualify all claims:

No statistical significance testing for the main results. Table 1 reports correlation coefficients without confidence intervals or significance tests. The 500-segment WMT23 test sets and 100-article SummEval test set are relatively small for correlation analysis, and observed differences between models (e.g., o3-mini-high at 0.577 vs. GPT-4o-mini at 0.410 for en-de) may or may not be statistically reliable. In meta-evaluation research, confidence intervals on correlation differences are standard practice because the test sets are small and correlation estimates have substantial variance.
Unknown model sizes for OpenAI models. The paper acknowledges in Limitations that "there is no public confirmed information about the size of either of these models, thus this cannot be verified." This means the o3-mini/GPT-4o-mini comparison does not control for model scale — if o3-mini is substantially larger than GPT-4o-mini, the observed reasoning benefits could be partly or entirely attributable to scale rather than reasoning capability. The paper's central claim about architecture-dependent reasoning benefits rests partly on a comparison where basic model properties are unknown.
Single prompt template with no optimization for reasoning models. All models use the same GEMBA-MQM and G-Eval prompts that were designed for standard instruction-tuned models. Reasoning models might respond differently — possibly better — to prompts that explicitly structure the reasoning process (e.g., "First, identify all potential errors. Second, categorize each error. Third, assign severity..."). The paper's finding that reasoning models underperform on some dimensions might reflect prompt-model mismatch rather than genuine capability limitations.
Limited language pair coverage for MT. Three language pairs (en-de, he-en, zh-en) provide some diversity but cannot support general claims about "multilingual evaluation." The finding that reasoning helps on en-de and zh-en but not he-en is provocative but unexplained — is the determining factor resource level, script, morphological type, or something specific about the WMT23 test sets for these languages? Testing additional typologically diverse pairs (e.g., ja-en, ar-en, fi-en) would be needed to characterize the conditions under which reasoning helps.
No cost-benefit analysis. The paper mentions computational cost as a motivation (distilled models "reducing computational costs") but never quantifies the inference cost of reasoning vs. non-reasoning models. A reader cannot determine from this paper whether the 40.7% improvement in en-de correlation from using o3-mini-high instead of GPT-4o-mini is worth the additional cost. The reasoning token analysis provides relative costs (token counts) but not absolute costs, and the paper does not report whether o3-mini-high generates 2×, 10×, or 100× more tokens than GPT-4o-mini for the same evaluation task.

Missing experiments that would strengthen the paper:

A reasoning-specific prompt variant. Testing whether prompts that explicitly scaffold the reasoning process (structured error detection steps, explicit comparison instructions) change the performance gap between reasoning and non-reasoning models would clarify whether the observed architecture-dependent effects reflect genuine capability differences or prompt compatibility issues.
Intermediate distillation scales. Testing a 13B or 20B distilled variant would help locate the scale threshold for reasoning transfer more precisely than "somewhere between 8B and 32B."
A wider set of NLG tasks. The paper frames itself as studying "NLG evaluation" but tests only MT and TS. Tasks like dialogue response evaluation, image caption evaluation, or data-to-text evaluation might show different reasoning benefit patterns. The title's claim about "Evaluating MT and Summarization" is accurate, but the broader framing in the abstract and introduction about "natural language generation" should be tempered.
Inter-annotator agreement as an upper bound. Human judgments are not perfectly reliable — inter-annotator agreement on SummEval and WMT23 imposes a ceiling on how well any automated metric can correlate with (noisy) human judgments. Reporting this ceiling would contextualize the observed correlations and show how close the best models come to human-level agreement.

6. Limitations and Trade-offs

The Matched-Pair Design Between OpenAI Models Rests on an Unverifiable Size Assumption

The assumption or constraint. The paper's central experimental design matches reasoning models to non-reasoning counterparts at comparable parameter scales so that performance differences can be attributed to reasoning training rather than model size. For the DeepSeek family, this matching is credible: DeepSeek-R1 is explicitly the RL-fine-tuned variant of DeepSeek-V3, sharing the same ~685B parameter architecture and pretraining. For the OpenAI family, however, the paper acknowledges a critical gap:

"We make an assumption that the GPT-4o-mini model is the most likely non-reasoning equivalent, by capabilities and/or size, to the reasoning model o3-mini. At the same time, there is no public confirmed information about the size of either of these models, thus this cannot be verified." (Limitations, point 1)

This is not a minor disclosure issue — it means one of the two primary architecture families in the study lacks the controlled comparison that the paper's conclusions depend on. If o3-mini is substantially larger than GPT-4o-mini (e.g., 100B vs. 8B parameters), the observed performance advantage of o3-mini-high over GPT-4o-mini on en-de MT evaluation (0.577 vs. 0.410, a 40.7% improvement, Table 1) could be partly or entirely attributable to model scale rather than reasoning capability. If o3-mini is actually smaller than GPT-4o-mini but outperforms it due to reasoning, that would strengthen the paper's claims — but we have no way to know which scenario holds.

The consequence. The paper's most striking positive finding for reasoning — that OpenAI o3-mini models demonstrate superior MT evaluation quality over their non-reasoning equivalent — cannot be confidently attributed to reasoning training. A practitioner deciding between deploying o3-mini or GPT-4o-mini for MT evaluation cannot determine from this paper whether the performance difference reflects reasoning capability (a potentially generalizable property of reasoning models), model scale (a property that would also benefit other tasks), or an interaction between scale and reasoning (a property that might not transfer to other model families). The architecture-dependent performance conclusion — "reasoning capabilities alone do not guarantee improved evaluation quality, rather the specific implementation and post-training approach for enhancing reasoning capabilities determines effectiveness" (Section 6) — is weakened because the OpenAI "implementation" advantage is confounded with unknown scale differences.

What evidence exists in the paper. The limitation is explicitly stated in the Limitations section (point 1) but the main results sections (Section 4, Section 6) discuss the o3-mini/GPT-4o-mini comparison without qualifying this uncertainty. Table 1 reports the performance numbers side by side with the DeepSeek matched pairs, visually implying comparable experimental control. The reasoning effort gradient within o3-mini (high > medium > low for en-de MT in Table 1) partially mitigates this concern by showing that within a fixed architecture, more reasoning improves performance — this provides some evidence that reasoning itself matters independent of scale, since the effort manipulation operates within the same model. However, the comparison to the non-reasoning baseline (GPT-4o-mini) is where the scale confound becomes critical, and the effort gradient does not resolve it: a larger model with more reasoning could outperform a smaller model without reasoning even if reasoning itself were neutral or harmful.

Mitigation status. The authors acknowledge this limitation transparently in the dedicated Limitations section but do not attempt to address it experimentally. No alternative non-reasoning baselines at verified comparable scales are tested. The paper does not propose proxy methods to estimate model size (e.g., inference latency scaling, per-token cost ratios) that might provide circumstantial evidence about the o3-mini/GPT-4o-mini size relationship. The limitation is left as a caveat for the reader to weigh.

Prompt Templates Are Fixed Without Reasoning-Specific Optimization, Potentially Undercounting Reasoning Model Capability

The assumption or constraint. The paper evaluates all models using "the same prompting templates to ensure fair comparison" (Section 3, Evaluation Protocol), specifically the standard GEMBA-MQM template for MT and the standard G-Eval template for summarization. Both templates were originally designed and validated for non-reasoning instruction-tuned LLMs. The paper does not test any reasoning-optimized prompt variants that might better leverage the structured deliberation capabilities of reasoning models.

This design choice is methodologically defensible as a starting point — it tests whether reasoning models can serve as drop-in replacements in existing evaluation pipelines. However, it also means that the paper is testing a specific interaction between reasoning models and prompts optimized for a different model class. If reasoning models perform differently (better or worse) with prompts that explicitly scaffold their reasoning process, the reported results reflect prompt-model compatibility rather than the models' intrinsic evaluation capability.

The consequence. The paper's negative findings for DeepSeek-R1 — that it "mostly underperforms compared to its non-reasoning counterpart DeepSeek V3 in both MT and TS evaluation" (Section 6) — could reflect either genuine evaluation capability limitations or suboptimal prompting. A reasoning model trained to produce step-by-step analysis might perform substantially better with a prompt that says "First, list all potential translation errors in the hypothesis. Second, for each error, identify its category (accuracy, fluency, terminology). Third, assign severity (critical, major, minor). Finally, produce your assessment." rather than the standard GEMBA-MQM prompt that presents the task in a more holistic instruction-following format. The paper provides no evidence to distinguish between "the model lacks evaluation capability" and "the model has evaluation capability but the prompt doesn't elicit it effectively."

This is particularly relevant for the DeepSeek-R1 consistency result on summarization (0.565 Kendall's τ, a 70.7% improvement over DeepSeek-V3 at 0.331, Table 1). If this dimension-selective benefit arises because the G-Eval consistency prompt happens to align well with R1's reasoning patterns (fact-verification across documents), then other dimensions where R1 underperforms might benefit from prompt restructuring that better maps to the model's reasoning style. The paper's framing implies that R1 is genuinely bad at coherence and relevance evaluation, but it could equally be that the standard G-Eval prompts for those dimensions don't elicit effective reasoning from this particular model.

What evidence exists in the paper. The paper does not test any prompt variants. There is no ablation comparing standard prompts to reasoning-structured prompts for any model. The paper acknowledges this implicitly in the discussion: "Future work should focus on specifically aligning reasoning approaches with evaluation requirements through fine-tuning and architectural modifications" (Section 6), but this framing focuses on model-side modifications rather than prompt-side optimization, which is a lower-cost intervention. The reasoning token analysis (Section 5, Table 2) shows that DeepSeek-R1 generates reasoning traces but these traces don't correlate with evaluation accuracy — however, this doesn't tell us whether better-structured prompts would produce more effective reasoning traces.

Mitigation status. The authors do not acknowledge prompt compatibility as a limitation. The "same prompt" design is presented as a feature ("to ensure fair comparison") rather than as a potential confound. The Limitations section does not mention this issue. The paper's conclusions are stated as findings about model capability rather than findings about model-prompt interaction.

Difficulty Estimation and Strategy Selection Costs Are Not Accounted For in Any Efficiency Analysis

The assumption or constraint. The paper frames its work partly around practical deployment considerations — "whether distilled reasoning models can maintain evaluation capabilities while reducing computational costs, given that full-scale reasoning LLMs often have several hundreds of billions of parameters" (Section 1). Yet the paper provides no computational cost accounting whatsoever: no FLOPs estimates, no API cost comparisons, no wall-clock time measurements, no token count comparisons between reasoning and non-reasoning models at equivalent tasks. The reasoning token analysis (Section 5, Table 2) provides relative token counts within reasoning models but never compares the absolute number of tokens generated by, say, o3-mini-high versus GPT-4o-mini for the same evaluation task.

The consequence. A practitioner reading this paper cannot determine whether the observed performance improvements justify the additional cost. If o3-mini-high achieves a 40.7% improvement in en-de Pearson correlation over GPT-4o-mini (0.577 vs. 0.410, Table 1) but generates 10× more tokens per evaluation (due to reasoning traces that can be thousands of tokens long), the practical value proposition depends on the practitioner's cost-sensitivity — a fact the paper provides no evidence to assess. Similarly, the claim that "distillation up to 32B parameters preserves most evaluation quality" (Sections 1, 6) is meaningful for efficiency only if the reader knows the baseline cost of running the 685B original versus the 32B distilled model — the paper never quantifies this.

The reasoning token analysis makes the efficiency question more acute, not less. Table 2 shows that o3-mini models generate more reasoning tokens when evaluating lower-quality texts (strong negative score correlations: -0.474, -0.415, -0.347 for MT). This means that in a batch evaluation setting where translation quality varies, the average cost per evaluation may be substantially higher than the cost on high-quality texts. The paper reports this as an interesting behavioral finding ("models dedicate substantially more reasoning tokens when they identify quality issues," Section 5) but does not connect it to the cost implications for deployment.

What evidence exists in the paper. The paper provides no cost data. The reasoning token counts used in Table 2 are not reported in absolute terms (e.g., mean tokens per segment, distribution of token counts across the test set) — only as correlation coefficients. The reader cannot determine how many tokens o3-mini-high typically generates versus o3-mini-low versus GPT-4o-mini. The paper's GitHub repository may contain this data, but the paper itself does not report it.

Mitigation status. The authors do not acknowledge this as a limitation. The Limitations section (points 1–3) addresses model size disclosure, reasoning effort control, and hybrid reasoning model coverage — not cost accounting. The paper's framing around efficiency ("distillation [...] reducing computational costs," Section 1) implies cost is a relevant consideration, but the absence of any cost quantification means the efficiency claims are asserted rather than demonstrated.

Single-Task Scope Provides No Evidence About Generalization Beyond Machine Translation and Summarization

The assumption or constraint. The paper's title and abstract claim to study "NLG evaluation" broadly, but the experiments cover exactly two tasks: machine translation (via GEMBA-MQM on WMT23) and text summarization (via G-Eval on SummEval and Eval4NLP). These are well-chosen tasks with established benchmarks and reliable human judgments, but they represent a narrow slice of the NLG evaluation landscape. The paper provides no evidence about how reasoning models perform on evaluation of dialogue responses, image captions, data-to-text generation, story generation, instruction-following outputs, or any other NLG task.

This matters because the paper's central finding — that reasoning benefits are architecture-dependent and task-dependent — implies that the specific properties of MT and TS evaluation tasks shape whether reasoning helps. MT evaluation (via MQM error detection) involves identifying discrete, localizable errors (mistranslations, omissions, additions) with well-defined categories and severity levels — a structure that maps naturally to step-by-step analysis. Summarization evaluation involves both similar discrete checking (consistency: does the summary contradict the source?) and holistic quality assessment (coherence, fluency). Other NLG tasks with different evaluation structures — dialogue response appropriateness, creative writing quality, instruction-following accuracy — may show entirely different reasoning benefit patterns that cannot be inferred from MT and TS results.

The consequence. The paper cannot support claims about reasoning models as "NLG evaluators" in general. A practitioner evaluating a dialogue system or an image captioning model cannot look to this paper for guidance on whether to use a reasoning or non-reasoning LLM as a judge. The task-dependence the paper documents within its narrow scope (o3-mini reasoning helps MT but not summarization; R1 reasoning helps consistency but not relevance) suggests — but cannot confirm — that task-dependence would be even more pronounced across a broader range of evaluation tasks.

What evidence exists in the paper. The tasks are explicitly stated in the abstract and Section 3: "Our evaluation focuses on two primary NLG tasks: MT evaluation and TS evaluation." The paper does not claim broader task coverage. However, the framing in the abstract ("evaluating natural language generation") and Section 1 ("A critical but underexplored application is Natural Language Generation (NLG) evaluation") is broader than the experimental scope. The task-dependence evidence within the paper — MT vs. TS showing different patterns, individual quality dimensions showing different patterns — actually strengthens the argument that broader generalization cannot be assumed.

Mitigation status. The authors do not explicitly flag task scope as a limitation. The Limitations section does not mention the restriction to MT and TS. However, the Discussion (Section 6) frames findings as specific to the studied tasks rather than claiming universality: "MT evaluation appears to benefit more from the particular reasoning approach employed by o3-mini models." This phrasing appropriately limits the scope but the broader framing throughout the paper could mislead readers about generalizability.

The 8B Distillation Failure Is Confounded with Architecture Change, Preventing a Clean Scale Threshold Identification

The assumption or constraint. The paper's distillation analysis compares three model sizes: 70B (R1 LLaMA 70B), 32B (R1 Qwen 32B), and 8B (R1 LLaMA 8B). The key finding is that the 32B model "maintains performance comparable to the original DeepSeek-R1" while the 8B model "shows substantial degradation" (Section 6), establishing a scale cliff somewhere between 8B and 32B parameters. However, the 32B and 8B distilled models use different base architectures: Qwen for the 32B, LLaMA for the 8B. The 70B model, which also uses LLaMA, shows mixed results — it underperforms the original R1 on SummEval average (0.315 vs. 0.351, Table 1) and Eval4NLP (0.556 vs. 0.583), and its SummEval performance is below its own non-reasoning counterpart LLaMA 3.3 70B (0.375). This means the distillation performance pattern could reflect architecture effects (Qwen distillations preserve capability better than LLaMA distillations) rather than pure scale effects.

The consequence. The paper's practical guidance — "models of at least 32B parameters maintaining performance while 8B models lose critical evaluation capabilities" (Section 6) — may be incorrect if the determining factor is architecture rather than scale. A hypothetical R1 Qwen 8B might perform substantially better than R1 LLaMA 8B, and a hypothetical R1 LLaMA 32B might perform worse than R1 Qwen 32B. The paper's claim about a "sharp scale threshold" (Innovation 3 in Section 4) is undermined by the architecture confound: the threshold might actually be between Qwen and LLaMA architectures rather than between 32B and 8B parameters.

This matters practically because model providers considering distillation of reasoning models for evaluation tasks need to know whether to invest in larger LLaMA-based distillation or in Qwen-based distillation at any given scale. The current results provide no way to disentangle these factors. A provider with existing LLaMA-based infrastructure who reads this paper might conclude that 70B is sufficient (since R1 LLaMA 70B outperforms R1 LLaMA 8B on most metrics in Table 1) without realizing that a 32B Qwen-based model might outperform both.

What evidence exists in the paper. The architecture confound is visible in Table 1: R1 LLaMA 70B achieves lower SummEval average (0.315) than R1 Qwen 32B (0.355) despite being more than twice the size, and lower than its own non-reasoning baseline LLaMA 3.3 70B (0.375). The 8B degradation is dramatic (0.174 SummEval vs. 0.228 for non-reasoning LLaMA 3.1 8B), but whether this reflects the 8B scale or the LLaMA architecture (or an interaction) is unanswerable from the available data. The paper does not test R1 Qwen 8B or R1 LLaMA 32B, which would be needed to isolate scale from architecture.

Mitigation status. The authors do not acknowledge this confound. The Discussion (Section 6) treats the distillation findings as purely about parameter count: "models of at least 32B parameters maintaining performance while 8B models lose critical evaluation capabilities." The Limitations section does not mention the architecture confound. This is an omission that weakens the specificity of the practical guidance the paper offers.

Evaluation Quality Meta-Metrics Are Reported Without Confidence Intervals on a Modest-Sized Test Set, Making Performance Differences Difficult to Interpret

The assumption or constraint. The paper's primary results (Table 1) report point estimates of Pearson and Kendall correlation coefficients between model scores and human judgments. No confidence intervals, standard errors, or statistical significance tests are reported for these main comparisons. The test sets are modest in size: WMT23 provides segment-level human judgments for three language pairs (the exact number of segments per language pair is not specified in the paper, but WMT23 Metrics Shared Task test sets typically contain 500–1000 segments per language pair; Freitag et al., 2023). SummEval provides exactly 100 articles × 16 systems = 1,600 summary evaluations, each rated on four dimensions. The Eval4NLP dataset size is not specified.

Correlation coefficients estimated from these sample sizes have non-trivial variance. A difference of 0.05 or even 0.10 in Pearson correlation may not be statistically significant, depending on sample size and the correlation magnitude. Without confidence intervals, the reader cannot determine whether o3-mini-high (0.577 en-de) is reliably different from GPT-4o-mini (0.410), or whether the ordering of o3-mini-high > o3-mini-medium > o3-mini-low for en-de represents a genuine monotonic trend or sampling noise.

The consequence. Many of the paper's specific comparative claims lack statistical grounding. The claim that "o3-mini-high achieves the highest correlation scores for en-de (0.577) and zh-en (0.568) translation pairs, substantially outperforming the non-reasoning GPT-4o-mini (0.410 and 0.487, ↑40.7% and ↑16.6% respectively)" (Section 6) interprets point estimates as precisely measured quantities. If the 95% confidence interval for o3-mini-high on en-de is [0.52, 0.63] and for GPT-4o-mini is [0.35, 0.47], the difference is clearly significant. If the intervals overlap substantially (e.g., [0.48, 0.67] vs. [0.33, 0.49]), the conclusion is weaker. The paper provides no way to distinguish these scenarios.

The problem is compounded by the multiple comparisons implicit in the analysis: 8 models × 3 language pairs × segment-level correlation + system-level accuracy for MT, plus 8 models × 4 dimensions × average for TS, plus 8 models × Eval4NLP, plus cross-task comparisons. With this many comparisons, some "significant-looking" differences will arise by chance. Without confidence intervals or multiplicity correction, the paper's narrative about which differences are meaningful and which are noise relies entirely on author judgment.

What evidence exists in the paper. The paper reports statistical significance only for the reasoning token correlations in Table 2, noting that correlations marked with ‡ are "not statistically significant (p > 0.05)" while others are "significant at p < 0.001." This is the only significance testing in the paper. For the main evaluation quality results (Table 1), no significance information is provided. The paper uses the mt-metrics-eval package which can compute confidence intervals for Pearson correlations via bootstrap resampling — this capability exists but was not used or not reported.

Mitigation status. The authors do not acknowledge the absence of confidence intervals as a limitation. The Limitations section focuses on model disclosure issues, reasoning effort control, and task scope — not statistical rigor. Meta-evaluation research in the WMT metrics shared tasks has standardized reporting confidence intervals (Freitag et al., 2022, 2023), making their absence in this paper a departure from community norms that is not discussed or justified.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new method, metric, or model — it is a diagnostic study that reframes how the field should think about deploying reasoning LLMs for evaluation tasks. Its primary contribution is replacing a binary question with a conditional one. Before this work, the natural question was "should we use reasoning models as NLG evaluators?" — a question that invited a yes/no answer based on whether reasoning training, on average, improved correlation with human judgments. The paper's central empirical finding is that this question is ill-posed: the answer depends on model architecture, task, quality dimension, language pair, and reasoning effort setting simultaneously. The paper replaces the binary question with "under what conditions do reasoning models improve evaluation quality, and when do they degrade it?" — and provides the first systematic evidence to begin answering that conditional question.

The magnitude of the shift is modest but the conceptual realignment is important. The paper does not establish a new scaling law or propose a novel architecture that will transform evaluation pipelines overnight. Instead, it performs a service that is often undervalued: it prevents the field from prematurely converging on a simple narrative ("reasoning models are better evaluators" or "reasoning models are worse evaluators") by demonstrating that both narratives are true under different conditions. The 40.7% improvement of o3-mini-high over GPT-4o-mini on en-de MT evaluation (0.577 vs. 0.410 Pearson ρ, Table 1) coexists in the same table with DeepSeek-R1's 25.7% degradation relative to DeepSeek-V3 on the same task (0.364 vs. 0.490). A researcher who tested only one of these model families could have reached opposite conclusions; the paper's side-by-side comparison makes the architecture-dependence impossible to ignore.

Reconciling prior contradictions. The paper provides a framework for understanding why prior work on chain-of-thought evaluation produced mixed findings. Liu et al. (2024) showed that CoT can degrade performance on tasks where thinking is unnecessary; Sprague et al. (2025) showed that CoT helps primarily on math and symbolic reasoning. The paper's findings extend this pattern to NLG evaluation with greater specificity: CoT-style deliberation (as embodied in reasoning-trained models) helps on evaluation sub-tasks that decompose naturally into discrete, identifiable sub-problems — detecting translation errors in specific text spans with well-defined error categories (MT evaluation), verifying whether a summary's claims are supported by the source text (consistency evaluation). It helps less or actively hurts on evaluation sub-tasks that require holistic, gestalt quality judgments — assessing whether a summary flows well as a coherent text (coherence), whether it captures the right information (relevance), or whether the translation reads naturally in the target language (fluency). This decomposition explains why different prior studies reached different conclusions: they were testing on implicitly different mixtures of sub-tasks.

Which research directions become more attractive. The paper strongly suggests that the most productive research direction is not "build better reasoning models for evaluation" in general, but rather understand the mapping between specific reasoning architectures and specific evaluation sub-tasks. The fact that DeepSeek-R1 excels at consistency evaluation while o3-mini excels at coherence evaluation (Table 1) implies that different reasoning training procedures produce different evaluation specializations — a hypothesis the paper does not test directly but makes newly tractable. Relatedly, the paper makes verifier robustness for evaluation tasks an attractive research direction: the finding that reasoning models allocate more tokens to texts they perceive as lower-quality without necessarily improving accuracy (Table 2, DeepSeek-R1 shows score correlations of -0.010 on TS but error correlations of -0.054, essentially zero) suggests that reasoning traces can be internally coherent but externally misaligned with human quality standards. This is analogous to reward hacking in RLHF, adapted to the evaluation domain.

Which directions become less attractive. The paper reduces the appeal of uniform deployment of reasoning models as drop-in replacements in existing evaluation pipelines. The finding that the non-reasoning DeepSeek-V3 achieves the highest SummEval average (0.399, Table 1) among all tested models — outperforming every reasoning model — means that reasoning is not a Pareto improvement for evaluation quality. Practitioners who were considering switching to reasoning models wholesale should instead consider a more nuanced strategy. The paper also reduces enthusiasm for small-scale distillation of reasoning models (below 32B parameters) for evaluation deployment, given the catastrophic 50.4% degradation at 8B scale (Table 1, R1 LLaMA 8B at 0.174 SummEval average vs. original R1 at 0.351).

Follow-Up Research This Work Enables

Mapping reasoning training procedures to evaluation sub-task specializations. The paper's most striking result is the complementary dimension-selectivity of DeepSeek-R1 and o3-mini: R1 dominates consistency (0.565 vs. 0.331, Table 1) while o3-mini dominates coherence (0.482 vs. 0.321) and fluency (0.313 vs. 0.263). These models were trained via different reinforcement learning algorithms (GRPO for DeepSeek, proprietary for OpenAI) with potentially different reward structures, training data compositions, and reasoning trace formats. A direct follow-up would systematically compare models trained with the same base architecture but different reasoning rewards — for example, training one variant with a reward function that emphasizes factual verification (predicting consistency strengths) and another with a reward function that emphasizes stylistic assessment (predicting coherence/fluency strengths) — and measuring whether the evaluation dimension specializations shift accordingly. This would test whether the architecture-dependent patterns in Table 1 reflect training-induced specializations or fixed architectural properties.

Prompt adaptation for reasoning models in evaluation. The paper uses standard GEMBA-MQM and G-Eval prompts designed for non-reasoning models, applied identically to all models. This tests reasoning models as drop-in replacements but leaves open whether reasoning-optimized prompts would yield different results. A concrete experiment would compare three prompt conditions for each reasoning model on MT and TS evaluation: (a) the standard prompt (as in the paper), (b) a structured reasoning prompt that explicitly decomposes the evaluation into steps ("Step 1: Identify all potential errors in the translation. Step 2: Categorize each error type. Step 3: Assign severity to each error..."), and (c) an open-ended reasoning prompt that instructs the model to "think carefully and explain your reasoning before providing your evaluation." If DeepSeek-R1's MT evaluation performance improves substantially with structured prompting — closing some or all of the gap with DeepSeek-V3 (currently 0.364 vs. 0.490 on en-de, Table 1) — that would demonstrate that R1's weakness is not in evaluation capability but in prompt alignment. If performance is unchanged, it would strengthen the conclusion that R1's reasoning training genuinely impairs MT evaluation.

Language-pair-scale characterization of reasoning benefits for MT evaluation. The paper tests three language pairs (en-de, he-en, zh-en) and finds that o3-mini reasoning benefits are clearest for en-de and zh-en but absent or reversed for he-en (Table 1). However, three language pairs with different resource levels, script systems, and linguistic typologies cannot isolate which factor determines reasoning benefit. A systematic study testing 10–15 language pairs varying along three axes — resource level (high-resource vs. low-resource), script similarity to English (Latin script vs. non-Latin script), and morphological typology (analytic, agglutinative, fusional) — could identify whether reasoning helps primarily on high-resource languages (where the model has sufficient pretraining exposure to reason effectively about translation quality), on script-similar languages (where tokenization and representation are better), or on languages with specific structural properties. The he-en finding — the one pair where reasoning did not help — involves a lower-resource language with a non-Latin script and rich morphology, confounding all three factors. Disentangling them would provide actionable guidance: if resource level is the critical factor, reasoning models should be deployed for evaluating high-resource language pairs only; if script is critical, the boundary would be different.

Controlled distillation scale sweep within a single architecture. The paper finds a performance cliff between 32B (R1 Qwen 32B at 0.355 SummEval average, Table 1) and 8B (R1 LLaMA 8B at 0.174) but confounds architecture with scale — the 32B model uses Qwen while the 8B uses LLaMA. A clean follow-up would distill DeepSeek-R1 into a single architecture (e.g., Qwen or LLaMA) at 8B, 14B, 20B, 32B, and 70B scales, evaluating all on the same MT and TS benchmarks. This would identify whether the degradation is gradual (each halving of parameters reduces performance by X%) or exhibits a sharp threshold (performance is flat above some critical parameter count and collapses below it). If the degradation is gradual, the practical takeaway would be a cost-quality tradeoff curve from which practitioners can choose based on their computational budget. If the degradation is sharp, the threshold would represent a genuine capacity requirement for reasoning transfer to evaluation, with implications for model design (architectures below the threshold cannot support effective evaluation reasoning regardless of training).

Negative result: does explicit reasoning always hurt on holistic quality dimensions? The paper finds that DeepSeek-R1 substantially underperforms DeepSeek-V3 on coherence (0.381 vs. 0.462, Table 1), relevance (0.303 vs. 0.446), and fluency (0.157 vs. 0.356) for summarization evaluation — dimensions that require holistic quality assessment rather than discrete error detection. A targeted experiment would test whether this degradation is caused by the reasoning process itself or by some other aspect of R1's training. The experiment: take a base model (e.g., DeepSeek-V3), prompt it to evaluate summaries with and without enforced chain-of-thought (using the same model, not a reasoning-trained variant), and measure whether adding explicit reasoning causes similar degradation on holistic dimensions. If forcing step-by-step analysis on a non-reasoning model also degrades coherence/fluency assessment, the mechanism is likely cognitive: decomposing a holistic quality judgment into steps introduces noise or shifts attention to the wrong features. If the non-reasoning model with CoT does not degrade, the R1 degradation is specific to its training and potentially fixable. This would clarify whether the paper's dimension-selectivity findings reflect a fundamental limitation of reasoning for holistic assessment or a contingent artifact of current reasoning training methods.

Combined reasoning and non-reasoning evaluation ensembles. The paper shows that reasoning and non-reasoning models have complementary strengths on specific evaluation dimensions: R1 excels at consistency while V3 excels at coherence and relevance (Table 1); o3-mini excels at coherence and fluency while GPT-4o-mini excels at consistency and relevance. This naturally suggests ensemble approaches where different models evaluate different dimensions based on their demonstrated strengths. A concrete follow-up would construct a dimension-specific ensemble: use DeepSeek-R1 for consistency scoring, DeepSeek-V3 for coherence and relevance scoring, and the best available model for fluency, then aggregate into a final quality score. The ensemble's correlation with human judgments across all dimensions would test whether combining complementary model strengths yields better overall evaluation than any single model. The baseline would be the best single model from Table 1 (DeepSeek-V3 with 0.399 SummEval average). If the ensemble significantly exceeds this, it would provide a practical deployment strategy that leverages the architecture-dependent findings rather than being limited by them.

Practical Applications and Downstream Use Cases

Cost-aware model selection for evaluation-as-a-service pipelines. Organizations that provide NLG evaluation as a service (e.g., companies offering automated translation quality assessment, summarization benchmarking platforms) can use the paper's findings to make architecture-specific deployment decisions. The paper shows that DeepSeek-V3 achieves the highest SummEval average among all tested models (0.399, Table 1) and the highest en-de Pearson correlation among non-OpenAI models (0.490) — all without generating reasoning tokens. For a service evaluating mostly summarization quality or English-German translation, DeepSeek-V3 provides the best quality at lower cost than any reasoning model (since reasoning traces add token overhead without benefit in these cases). For a service evaluating Chinese-English translation specifically, o3-mini-high provides the best quality (0.568 zh-en Pearson ρ) but at the cost of generating reasoning tokens — the service could offer it as a premium tier with the 16.6% quality improvement over GPT-4o-mini (0.487) as the value proposition. The paper's architecture-dependent findings translate directly to a tiered service model where different evaluation tasks route to different models based on empirically demonstrated strengths.

Dimension-specific evaluation in summarization system development. Teams developing summarization models typically need to track multiple quality dimensions during iterative development — coherence to ensure outputs read well, consistency to catch hallucinations, relevance to avoid information omission, fluency for surface quality. The paper shows that no single model excels at all dimensions: DeepSeek-R1 dominates consistency (0.565, Table 1) but is weak on relevance (0.303) and fluency (0.157); DeepSeek-V3 is strong on coherence (0.462) and relevance (0.446) but moderate on consistency (0.331); o3-mini-high dominates coherence (0.482) and fluency (0.313). A development pipeline could use DeepSeek-R1 specifically to flag factual inconsistencies in candidate summaries (the 70.7% improvement over V3 on consistency means catching substantially more hallucinations), while using DeepSeek-V3 or o3-mini-high to assess the other dimensions. This dimension-specific deployment is more cost-effective than running all models on all dimensions, and the paper provides the evidence needed to justify the allocation: Table 1's per-dimension breakdown tells developers exactly which model to trust for which quality aspect.

Distillation scale targeting for on-device or edge evaluation. The paper's finding that R1 Qwen 32B achieves 97% of original DeepSeek-R1's Eval4NLP performance (0.564 vs. 0.583, Table 1) while the 8B model exhibits catastrophic degradation provides a concrete parameter target for practitioners building lightweight evaluation systems. For applications where running a 600B+ parameter model is infeasible — browser-based evaluation plugins, on-device quality checking in translation apps, real-time summarization quality assessment in low-latency environments — the 32B scale is an empirically validated sweet spot. The paper shows that at this scale, the distilled reasoning model preserves both the overall evaluation quality and the specific reasoning advantage on consistency evaluation (R1 Qwen 32B consistency at 0.540 vs. Qwen-2.5 32B at 0.449, a 20.3% improvement). A deployment team can confidently invest in R1 Qwen 32B knowing that (a) it will not be worse than a same-sized non-reasoning model (unlike the 8B case, where R1 LLaMA 8B underperforms LLaMA 3.1 8B on SummEval average 0.174 vs. 0.228), and (b) it will provide specific benefits on consistency evaluation that non-reasoning models of the same size cannot match.

When to Prefer This Method

The paper does not propose a new method — it provides empirical guidance for choosing among existing models for evaluation tasks. The decision framework implied by the results is:

Prefer DeepSeek-V3 (non-reasoning) for summarization evaluation when overall quality across all dimensions matters. DeepSeek-V3 achieves the highest SummEval average among all tested models (0.399, Table 1) with strengths in coherence (0.462), relevance (0.446), and fluency (0.356). It outperforms all reasoning models on these holistic quality dimensions and does so without reasoning token overhead.
Prefer DeepSeek-R1 specifically for summarization consistency evaluation when factual faithfulness is the primary concern. DeepSeek-R1 achieves 0.565 Kendall's τ on consistency (vs. 0.331 for V3, a 70.7% improvement, Table 1) — the largest positive reasoning effect in the study. If the downstream application is detecting hallucinations in summaries (e.g., for medical summarization, news verification), R1's consistency advantage justifies the reasoning cost.
Prefer OpenAI o3-mini-high for machine translation evaluation on high-resource language pairs (en-de, zh-en). O3-mini-high achieves the highest segment-level Pearson correlations for en-de (0.577) and zh-en (0.568) among all tested models (Table 1), with clear monotonic improvement from increased reasoning effort. The 40.7% improvement over GPT-4o-mini on en-de (0.410) is the strongest reasoning benefit documented in the paper.
Prefer GPT-4o-mini for MT evaluation on the he-en language pair, and exercise caution about generalizing reasoning benefits to all language pairs. GPT-4o-mini achieves 0.435 on he-en, outperforming all o3-mini variants (high: 0.421, medium: 0.404, low: 0.413, Table 1). The reasoning advantage does not hold for this lower-resource, typologically distinct pair.
Prefer distilled reasoning models at 32B scale (R1 Qwen 32B) for cost-sensitive deployments that still need some reasoning benefits. R1 Qwen 32B achieves 97% of original R1 Eval4NLP performance (0.564 vs. 0.583, Table 1) and preserves the consistency advantage over non-reasoning models of the same scale (0.540 vs. 0.449). Avoid distilled reasoning models at 8B scale — R1 LLaMA 8B underperforms its non-reasoning counterpart on SummEval average (0.174 vs. 0.228) and loses the consistency advantage entirely (0.251 vs. 0.312).
Prefer non-reasoning LLaMA 3.3 70B over its reasoning-distilled counterpart R1 LLaMA 70B for most evaluation tasks. LLaMA 3.3 70B achieves higher SummEval average (0.375 vs. 0.315, Table 1), higher Eval4NLP (0.624 vs. 0.556), and higher MT correlations on all three language pairs (en-de: 0.590 vs. 0.421; he-en: 0.420 vs. 0.365; zh-en: 0.522 vs. 0.451). The 70B distillation does not preserve enough reasoning benefit to justify choosing it over the non-reasoning model of the same scale.