UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?

🎯 Pitch

Even the best MLLMs trail human evaluators by over 30 accuracy points at spotting flaws in AI-generated videos, despite dramatically outperforming all specialized automatic metrics across 15 diverse aspects. By simply prompting models like Qwen2VL-72B without any training, we unlock zero-shot evaluation capabilities that leave purpose-built methods like VBench narrowly confined to their pre-defined dimensions.

1. Executive Summary

This paper investigates whether multimodal large language models (MLLMs) can serve as unified evaluators for AI-generated videos (AIGVs), capable of assessing any evaluation aspect through prompting alone rather than requiring aspect-specific specialized models. Using their introduced benchmark, UVE-Bench—which provides pairwise human preference annotations across 15 fine-grained evaluation aspects on videos from eight state-of-the-art video generative models—the authors extensively evaluate 18 MLLMs in both single video rating (via token-probability scoring) and video pair comparison (via direct selection from four preference options). Advanced MLLMs such as Qwen2-VL-72B and InternVL2.5-78B significantly outperform existing specialized evaluators like VBench and VideoScore across nearly all aspects, achieving average single-rating scores exceeding 75 out of 100 while specialized methods remain narrowly constrained to their trained aspects, yet a substantial gap of over 30 accuracy points persists between these MLLMs and human evaluators in pairwise comparison, establishing that MLLMs show promising versatility as unified AIGV evaluators but cannot yet replace human judgment, particularly on aspects requiring fine-grained temporal understanding such as motion naturalness and temporal flickering.

2. Context and Motivation

The Core Problem: AIGV Evaluation Has Not Kept Pace with Video Generation

The fundamental problem this paper addresses is that automatic evaluation methods for AI-generated videos are fragmented, narrow, and difficult to extend, even as video generative models (VGMs) advance rapidly. In the past two to three years, models like SORA, Kling, Gen-3, Movie Gen Video, and HunyuanVideo have progressed from producing obviously flawed outputs to generating videos that can be difficult for non-experts to distinguish from real footage. Yet the tools we use to measure their quality have not undergone a corresponding evolution. This creates a bottleneck: researchers developing VGMs need reliable, fine-grained feedback on what their models do well and poorly, but the available automatic metrics are limited to a handful of pre-specified aspects and cannot easily adapt to new evaluation needs.

The gap is both practical and structural. Practically, without comprehensive automatic evaluation, VGM development depends heavily on human assessment—slow, expensive, and difficult to standardize across different research groups. Structurally, existing evaluation paradigms assume that the set of evaluation aspects is fixed and known in advance, which is fundamentally at odds with how generative models evolve: each new generation of VGMs introduces new failure modes (e.g., SORA's physically implausible object interactions, Movie Gen's inconsistent lighting across frames) that existing metrics were never designed to detect.

This is not merely an inconvenience. The paper argues in Section 1 that effective evaluation is essential for advancing VGMs and enabling real-world deployment of AIGVs. If a model developer cannot quickly and accurately measure whether a training change improves motion naturalness or introduces new structural artifacts, the iteration cycle slows dramatically. The alternative—relying on qualitative human inspection or coarse aggregate metrics—obscures the very signals that drive model improvement.

Why This Matters Now: The Urgency of Finer-Grained, Scalable Evaluation

The paper identifies a specific historical moment that makes this problem urgent. Section 1 states:

"with the rapid development of VGMs and AIGV applications, there is an urgent need for finer-grained and more comprehensive evaluations. Continuously collecting human assessments and training models to accommodate emerging evaluation aspects and frequently changing standards is cost-intensive and difficult to scale."

Three converging trends make this urgency concrete:

First, the failure modes of modern VGMs are increasingly subtle. Earlier models (e.g., VideoCrafter1 from 2023) produced obvious artifacts—severe flickering, grossly inconsistent subject appearance, clearly unnatural motion. Modern models like Movie Gen Video and HunyuanVideo (both released in late 2024) produce videos where these artifacts are borderline imperceptible to casual viewing. Distinguishing a slightly awkward walking gait from a natural one, or detecting that lighting changes inconsistently across a 5-second clip, requires finer perceptual discrimination than earlier metrics were designed for. This is reflected in UVE-Bench's construction: the benchmark deliberately uses videos from the latest VGMs to "challenge the evaluators to identify the weaknesses of such videos" (Section 3.1).

Second, the evaluation surface area is expanding. As VGMs are deployed in more diverse applications—animation, advertising, education, entertainment—the aspects that matter for evaluation multiply. A video that is perfectly adequate for one use case may fail for another on a dimension that was never previously evaluated (e.g., cultural appropriateness of generated content, compliance with brand safety guidelines, emotional tone). Each new application potentially requires new evaluation aspects, and building specialized evaluators for each one is unsustainable.

Third, evaluation standards themselves are moving targets. What counts as "good" temporal consistency or "acceptable" motion naturalness shifts as the state of the art advances. A metric trained on human judgments collected in early 2024 may have been calibrated to a quality ceiling that newer models have already surpassed, making it unable to discriminate meaningfully among top-tier contemporary models. The paper's data collection strategy—using videos from eight different VGMs spanning a wide quality range, from Open-Sora 1.2 to Movie Gen Video—is designed specifically to ensure that UVE-Bench remains discriminative even as model quality improves.

Where Existing Approaches Fall Short: The Two-Category Taxonomy and Their Limitations

The paper's most important conceptual move in contextualizing prior work is its taxonomy of existing AIGV evaluation methods into two categories (Section 1), each with distinct structural weaknesses:

Category 1: Off-the-shelf models with heuristic rules. These methods repurpose models trained for other tasks—typically CLIP for video-text alignment, DINO for frame-to-frame consistency, RAFT for optical flow (to measure motion smoothness), and aesthetic predictors trained on real photographs—and apply hand-designed rules to aggregate their outputs into evaluation scores. The representative examples are VBench (Huang et al., 2024) and EvalCrafter (Liu et al., 2024). VBench, for instance, uses CLIP-based ViCLIP for overall consistency, DINO for subject consistency, AMT (a frame interpolation model) for motion smoothness, MUSIQ (an image quality model) for imaging quality, and a separately trained aesthetic predictor for aesthetic quality—each mapped to a different evaluation aspect.

The limitations of this approach are fundamental, not incidental:

Coverage is capped by available models. You can only evaluate an aspect if there exists an off-the-shelf model that roughly corresponds to it. Aspects like "does the motion align with the text description?" have no natural off-the-shelf model, so VBench's 16 aspects simply omit motion-alignment evaluation entirely. In UVE-Bench's taxonomy (Section 3.2), this means Category 1 methods can assess roughly half of the 15 subaspects—and even then, only approximately, since the models were trained on real videos or images, not AIGVs with their distinctive artifact distributions.
No mechanism for aspect-specific prompting or instruction. An off-the-shelf CLIP model embeds the video and text into a shared space and computes cosine similarity. It cannot be told "pay attention to whether the dog's fur color matches the prompt, but ignore the background." The evaluation is a black box that conflates all forms of correspondence into a single scalar. This makes it impossible to diagnose why a video scores poorly on alignment.
Distribution shift. Models trained on real photographs (e.g., LAION aesthetic predictors, MUSIQ) or real videos (e.g., RAFT for optical flow) encounter AIGV artifacts—synthetic textures, physically implausible motion patterns, unnatural lighting—that were absent from their training distribution. Their responses to these artifacts are unpredictable and may not correlate with human judgments of quality.

Category 2: Human-assessment-trained specialized evaluators. These methods collect human ratings for specific aspects of interest and train models—increasingly MLLMs—to predict those ratings. Representative examples include VideoScore (He et al., 2024), which fine-tunes Mantis-Idefics2-8B on human ratings (1–4 scale) for five aspects; VIDEOCON-PHYSICS (Bansal et al., 2024), which trains a model on binary human annotations for physical commonsense adherence and text alignment; Q-Align (Wu et al., 2024), which teaches LMMs to map visual quality to discrete text-defined levels; and AIGV-Assessor (Wang et al., 2024), which trains LMMs for perceptual quality assessment of text-to-video generation.

This approach partially addresses the coverage problem—you can train an evaluator for any aspect you're willing to pay humans to annotate—but introduces a different, equally fundamental limitation: scalability to new aspects. The paper articulates this crisply in Section 1:

"Continuously collecting human assessments and training models to accommodate emerging evaluation aspects and frequently changing standards is cost-intensive and difficult to scale."

This is not a hypothetical concern. Consider the lifecycle: a new VGM is released → users and researchers notice a new failure mode (e.g., "objects morphing into different objects mid-video") → someone must define this as an evaluation aspect, write annotation guidelines, collect thousands of human ratings, train a new evaluator or fine-tune an existing one → by the time this is done, an even newer VGM may have emerged with different failure modes. This pipeline is fundamentally reactive and cannot keep pace with the rate of model advancement. The paper's UVE-Bench, with its 15 aspects, already exceeds what any single Category 2 method covers—VideoScore covers at most 8 of UVE-Bench's 15 aspects by the authors' mapping (Table 8 in the appendix), VBench covers 6, and most others cover 1–3.

A deeper issue that the paper implies but does not state explicitly: human ratings for AIGV quality are inherently noisy and standards-dependent. Section 3.4 notes that the paper uses pairwise preference annotation rather than absolute rating specifically because pairwise comparisons "avoid the inconsistent standards of absolute ratings between humans." This is an important methodological choice with theoretical grounding—it is well-established in psychophysics and the image quality assessment literature that humans are far more reliable at comparing two stimuli than at mapping a single stimulus to an absolute scale (Thurstone, 1927; the classic "Law of Comparative Judgment"). Category 2 methods that train on absolute ratings inherit this inconsistency, and their learned scales may not generalize to videos of significantly different quality levels than those in the training set.

The Key Insight: MLLMs as a Potential Universal Evaluation Interface

The conceptual pivot of the paper is the recognition that the limitations of both existing approaches share a common root: the lack of a general-purpose visual-language understanding engine that can be instructed to evaluate any aspect on demand. The paper articulates this by analogy to human evaluation:

"Unlike existing automatic metrics, we humans can assess any aspect of AIGVs as long as a proper guideline is provided. This ability stems from our robust visual perception and language understanding."

The question the paper poses is whether MLLMs—trained on vast amounts of visual and language data and demonstrating increasingly sophisticated joint vision-language understanding—have developed analogous capabilities. This is not an obvious question. MLLMs are trained primarily on real-world images and videos, not AI-generated content. Their understanding of concepts like "motion naturalness" or "structural correctness" is learned indirectly through language descriptions and real-world visual statistics. Whether this knowledge transfers to evaluating synthetic videos—which may violate physical laws in ways never seen in training data while maintaining superficially realistic textures—is an empirical question that requires systematic investigation.

The paper's framing of this question has several important theoretical implications:

It reframes evaluation from a model-training problem to a prompting problem. Instead of collecting human data and training a specialized model for each new aspect, the vision is to write a prompt describing what to evaluate and have the MLLM produce a judgment. If this works, the cost of adding a new evaluation aspect drops from thousands of human annotations plus model training to the cost of writing a few sentences of natural language instruction.

It enables evaluation at arbitrary granularity. The paper's taxonomy of 15 aspects (Section 3.2) is itself illustrative: Static Quality breaks into Aesthetic Quality, Technical Quality, and Structural Correctness; Temporal Quality breaks into Appearance Consistency, Temporal Flickering, and Motion Naturalness; and so on. A truly unified evaluator could support even finer-grained distinctions—evaluating not just "motion naturalness" overall but "naturalness of arm movements" versus "naturalness of gait" separately, simply by modifying the prompt. This is impossible with either off-the-shelf models (which have fixed output semantics) or human-trained evaluators (which require new annotation campaigns for each new subdivision).

It separates evaluation capability from evaluation methodology. In the proposed framework, the same MLLM can perform both single video rating and video pair comparison (Section 2), and the choice between them is a matter of prompt design and output processing rather than model architecture. This is significant because different evaluation contexts prefer different formats: model developers doing large-scale hyperparameter sweeps may want scalar ratings for ranking, while qualitative analysis of specific failure cases benefits from pairwise comparisons that highlight relative strengths. A unified evaluator accommodates both without retraining.

How This Paper Positions Itself: A Benchmark-Driven Empirical Investigation, Not a Method Proposal

A crucial aspect of this paper's positioning—and one that differentiates it from much work in the AIGV evaluation space—is that it does not propose a new evaluation method. The approach described in Section 2 (prompting MLLMs and using token probabilities or direct generation for evaluation) is presented as a baseline framework to be tested, not as a novel contribution. The paper's contributions are instead:

A benchmark (UVE-Bench) that enables systematic comparison of MLLMs as unified evaluators across a broad range of aspects, using videos from state-of-the-art VGMs and human pairwise preference annotations.
An empirical characterization of where current MLLMs succeed and fail as AIGV evaluators, establishing both the promise and the current limitations of the approach.
Analytical insights into design choices (prompting strategy, scoring mechanism, frame count, single-vs-pair comparison) that affect MLLM evaluation performance.

This benchmarking-first approach is appropriate because the core question—"can MLLMs serve as unified AIGV evaluators?"—requires measurement infrastructure that did not previously exist. Existing AIGV datasets, as the paper documents in Table 3, cover few aspects (FETV: 9, VBench: 16 but with off-the-shelf models not human annotations, VideoFeedback: 5), use outdated VGMs (the most recent model in prior benchmarks is typically SORA-class or earlier, from early 2024), and provide only single-video ratings rather than pairwise preferences. The paper therefore must build the evaluation infrastructure before it can answer the question, and the infrastructure itself—UVE-Bench—is a significant contribution.

The paper also positions itself relative to prior work that has used MLLMs for AIGV evaluation but in a limited way. Section 1 notes:

"While using MLLMs for AIGV evaluation is a straightforward concept and has been explored in previous works [17, 2, 56], these studies are limited by evaluating only a few aspects and relying on human-annotated ratings to train the MLLMs."

The key distinction is zero-shot versus fine-tuned evaluation. Prior work (VideoScore, VIDEOCON-PHYSICS, AIGV-Assessor) uses MLLMs as a backbone for training specialized evaluators—still requiring human data and aspect-specific training. This paper investigates whether MLLMs can evaluate without any AIGV-specific training, purely by leveraging their pre-trained visual-language understanding. This is a crucial distinction because it determines whether the "unified" vision is actually feasible: if MLLMs require fine-tuning on human ratings for each new aspect, they suffer from the same scalability limitations as Category 2 methods. If they can evaluate zero-shot, the vision of a truly unified evaluator—capable of assessing any aspect described in natural language—becomes plausible.

The paper's empirical results partially validate this vision while also revealing its current limits. Advanced MLLMs do outperform specialized methods without any AIGV-specific training (Tables 4 and 5), supporting the zero-shot evaluation hypothesis. But the 30+ point accuracy gap to human evaluators in pairwise comparison (Table 5) and the specific weakness in temporal understanding (§4.2) indicate that zero-shot MLLM evaluation is not yet reliable enough to replace human judgment—particularly for aspects requiring fine-grained temporal dynamics understanding.

Summary: The Gap, The Urgency, and The Approach

To synthesize: the paper identifies a structural gap between the rapid advancement of VGMs and the static, narrow nature of existing AIGV evaluation methods. This gap is becoming increasingly costly as VGMs improve and their failure modes become more subtle and aspect-specific. Prior approaches—repurposing off-the-shelf models and training specialized evaluators on human ratings—each address part of the problem but cannot provide the combination of broad coverage, fine granularity, and zero-cost extensibility to new aspects that the field needs.

The paper's central hypothesis is that MLLMs, by virtue of their joint visual-language understanding, can fill this gap if they can be instructed to evaluate arbitrary aspects through prompting alone. UVE-Bench is the instrument designed to test this hypothesis systematically, and the paper's empirical findings provide the first comprehensive evidence on where this approach works (static quality, video-text alignment, dynamic degree assessment), where it falls short (temporal quality, structural correctness), and what design choices matter most for closing the gap.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a benchmarking and empirical analysis study, not a method proposal. The system being evaluated is remarkably simple: take a pre-trained MLLM, give it a video and a text description of what evaluation aspect to assess, and extract a judgment either by measuring the probability the model assigns to positive/negative tokens or by having it directly generate a preference choice. The core problem this system solves is evaluation fragmentation — instead of needing a separate specialized model for each of 15 different video quality aspects, a single MLLM can (in principle) evaluate any aspect by simply changing the text prompt. The "shape" of the solution is a prompting interface that maps natural language evaluation guidelines to structured judgments, enabling zero-shot evaluation across arbitrary aspects without any AIGV-specific training.

3.2 Big-Picture Architecture (Diagram in Words)

The unified evaluation framework has five major components, organized into two evaluation modes:

MLLM Backbone — the pre-trained multimodal model (e.g., Qwen2-VL-72B, InternVL2.5-78B) that processes video frames and text prompts jointly. This is the sole computation engine; no fine-tuning or adaptation is performed.
Prompt Template — a structured text format (Table 1) that wraps an aspect-specific evaluation description, a question targeting that aspect, and answer format instructions into a single input. The template is identical in structure across all 15 evaluation aspects; only the aspect-specific description and question change.
Video Frame Sampler — a pre-processing step that uniformly samples a fixed number of frames from each video (16 frames for single video rating, 12 frames per video for pair comparison) to fit within the MLLM's context window while providing temporal coverage.
Scoring/Selection Mechanism — the component that converts MLLM outputs into evaluation results. For single video rating, this is the token-probability scoring function (Equation 1) that computes a normalized score from the relative probabilities of positive and negative tokens. For video pair comparison, this is direct constrained generation (Equation 2) that maps the MLLM's text output to one of four preference categories.
Evaluation Criterion — the metric that compares the MLLM's output against human preference annotations to produce a performance score. For single video rating, this is Asingle (Equation 3), which handles both "better/worse" and "same good/same bad" human judgments through a piecewise function with exponential decay. For video pair comparison, this is simple accuracy Apair.

Information flows as follows: a video (or video pair) enters the system → the frame sampler extracts a uniform temporal grid of frames → the aspect-specific prompt is constructed by filling the template with the appropriate description and question → the MLLM processes frames and text jointly → the scoring/selection mechanism extracts a structured judgment → the evaluation criterion compares this judgment to the human annotation to produce a performance metric.

3.3 Roadmap for the Deep Dive

First, the single video rating mechanism (Section 2.2, Equation 1): the token-probability scoring formulation that converts MLLM output likelihoods into numerical quality scores. This is the simpler of the two evaluation modes and introduces the core idea of using MLLM token probabilities as a quality signal.
Second, the video pair comparison mechanism (Section 2.2, Equation 2): the direct generation approach where the MLLM selects from four preference options. This builds on the rating mechanism but introduces the challenge of multi-video input and constrained output parsing.
Third, UVE-Bench construction (Section 3): the video collection pipeline, the 15-aspect taxonomy and its rationale, the human annotation procedure, and the evaluation criteria that connect MLLM outputs to human judgments. Understanding the benchmark is essential because every experimental result is measured against it.
Fourth, the evaluation criteria mathematics (Equations 3–6): the detailed scoring functions that handle both pairwise preference ("better/worse") and tie ("same good/bad") cases, including the exponential decay mechanism and its parameterization. This is where many subtle design choices live.
Fifth, the design choices and their justifications: why pairwise preference over absolute ratings, why 15 specific aspects, why the particular frame counts, why yes/no as default scoring tokens. These choices are not arbitrary and understanding them reveals the paper's methodological commitments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmarking and empirical analysis paper whose core idea is that MLLMs can serve as zero-shot unified evaluators for AI-generated videos when prompted with aspect-specific evaluation guidelines, and that the quality of this evaluation can be systematically measured against human pairwise preferences across a diverse set of aspects and video generative models.

Single Video Rating via Token-Probability Scoring

The single video rating mechanism formulates evaluation as a token-probability extraction problem. Rather than asking the MLLM to generate a free-form quality assessment and then parsing that text (which introduces parsing ambiguity and generation stochasticity), the approach directly reads out the model's internal belief about video quality from the probabilities it assigns to pre-defined positive and negative tokens.

The core equation is:

$S = \frac{P_\theta(t_{\text{pos}} \mid V, T, G_a)}{P_\theta(t_{\text{pos}} \mid V, T, G_a) + P_\theta(t_{\text{neg}} \mid V, T, G_a)}$

where $S \in [0, 1]$ is the predicted quality score, $P_\theta(t \mid \cdot)$ is the probability assigned to token $t$ by MLLM $\theta$ given the input context, $t_{\text{pos}}$ and $t_{\text{neg}}$ are predefined positive and negative scoring tokens (by default "yes" and "no"), $V$ is the input video (represented as sampled frames), $T$ is the text-to-video prompt used to generate the video, and $G_a$ is the evaluation guideline text specific to aspect $a$ .

What this equation computes: the normalized probability that the MLLM assigns to the positive token relative to the sum of positive and negative token probabilities. Think of it as asking the MLLM a yes/no question about video quality and reading out how strongly it believes "yes" versus "no." If the model assigns probability 0.8 to "yes" and 0.2 to "no," the score is 0.8/(0.8+0.2) = 0.8. If it assigns equal probability, the score is 0.5. The normalization by the sum ensures the output is always in [0, 1] and represents the model's relative confidence, not just the absolute probability of one token (which could be affected by the overall probability mass assigned to other tokens in the vocabulary).

Why this form: this is a softmax over two tokens, which has two important properties. First, it is invariant to the absolute scale of the probabilities—if the model is generally uncertain and assigns low probabilities to all tokens, the ratio still correctly reflects relative preference. Second, it naturally handles model calibration differences: a model that is systematically overconfident (assigning very high probabilities to its preferred token) and one that is underconfident (assigning moderate probabilities) will produce similar scores as long as the relative ordering of "yes" versus "no" is preserved. This is motivated by prior work on image quality evaluation (Zhang et al., 2024; Lin et al., 2024) which demonstrated that token-probability ratios from MLLMs correlate well with human quality judgments.

The guideline $G_a$ is constructed by filling a template (Table 1, top) with aspect-specific content. The template structure is:

<video> Watch the above frames of an AI-generated video and evaluate <aspect-specific description> Complete your evaluation by answering this question: <aspect-specific question>? <answer prompt>

The aspect-specific description provides a detailed definition of what to evaluate. For example, for Technical Quality, the description might explain that the evaluator should examine the presence of unwanted noise, blur, and distortion. The aspect-specific question is typically of the form "Is the video good in terms of [aspect]?" and the answer prompt constrains the response format, typically "Please directly answer yes or no:".

The tokens $t_{\text{pos}}$ and $t_{\text{neg}}$ are the first tokens the model would generate following the answer prompt. By default, the paper uses "yes" and "no," but Section 4.3.2 explores alternatives including "good"/"bad," an adaptive strategy that selects between these two pairs per aspect, and direct numeric score generation (0–100). The probability extraction relies on the MLLM providing access to token-level log probabilities at the generation position, which is supported by most modern open-source MLLMs.

A critical implementation detail: the video $V$ is not fed to the MLLM as raw pixels but as a set of uniformly sampled frames. For single video rating, 16 frames are sampled per video (except for Video-LLaVA, which has an 8-frame hardware limitation). The frames are arranged temporally and presented to the MLLM as a sequence of images, relying on the MLLM's native multi-image understanding capability to process them as a coherent video. This frame-based approach is used because most MLLMs do not have native video encoders; they process videos by treating frames as individual images and relying on the LLM backbone's positional encoding and attention mechanism to capture temporal relationships.

Video Pair Comparison via Direct Preference Elicitation

Video pair comparison takes a fundamentally different approach: rather than extracting scalar scores and comparing them, the MLLM is shown both videos simultaneously and asked to directly select a preference from four options. The formulation is:

$C = f_\theta(V_1, V_2, T_1, T_2, G_a) \in \mathcal{O}$

where $C$ is the predicted choice, $f_\theta$ is the MLLM's text generation function (operating on the concatenated multi-video, multi-text input), $V_1$ and $V_2$ are the two videos being compared, $T_1$ and $T_2$ are their respective text-to-video prompts (included when relevant to the evaluation aspect, such as for video-text alignment), $G_a$ is the aspect-specific evaluation guideline, and $\mathcal{O} = \{\text{"V1 better"}, \text{"V2 better"}, \text{"same good"}, \text{"same bad"}\}$ is the set of four possible preference outcomes.

What this equation describes: the MLLM receives both videos (each as a sequence of frames), their generation prompts if needed, and the evaluation guideline, then generates text that is parsed into one of the four preference categories. The phrase " $f_\theta$ is the MLLM's text generation function" means the full autoregressive decoding process: the MLLM conditions on the multi-modal input and generates tokens one by one until it produces a complete preference statement, which is then matched to the closest option in $\mathcal{O}$ via string matching.

Why this form: direct pairwise comparison has two advantages over comparing independently computed single-video scores. First, it allows the MLLM to attend jointly to both videos, potentially noticing relative differences that would be missed when evaluating each video in isolation (e.g., a subtle lighting inconsistency that is only apparent when seeing the well-lit and poorly-lit versions side by side). Second, it directly outputs a discrete choice rather than a continuous score, avoiding the thresholding logic required to convert score differences into "better"/"same" decisions (which requires setting parameters $\alpha$ , $\beta$ , and $\tau$ as done in the single-to-pairwise adaptation in Section 4.3.3). However, this advantage comes at a cost: the MLLM must have the capability to process multiple videos in a single context window, and the two videos together consume approximately twice the context budget of a single-video evaluation.

The prompting template for video pair comparison (Table 1, bottom) is:

The first video: <video> The second video: <video> Watch the above two AI-generated videos and evaluate <aspect-specific description> Complete your evaluation by answering this question: Which video is <aspect-specific question>? You should make your judgment based on the following rules: <instructions on how to make the choice> Now give your judgment:

The rules section explicitly instructs the model on how to choose among the four options, for instance: "If both videos are good, answer 'same good'. If both videos are bad, answer 'same bad'." This is important because MLLMs may not naturally produce these specific phrases without explicit instruction.

For video pair comparison, 12 frames are sampled per video (rather than 16 for single rating), totaling 24 frames across the pair. This reduction is necessary because most MLLMs have context window limitations that constrain the total number of images that can be processed simultaneously, and two 16-frame videos (32 frames total) would exceed the practical input limits for many models. The paper notes (Section 3.3) that for video pair comparison evaluation, they use a subset of 2,411 samples from UVE-Bench with a more balanced distribution across the four preference categories, since the original dataset has more "V1/V2 better" examples which would bias four-way accuracy metrics toward those categories.

UVE-Bench: Video Collection Pipeline

UVE-Bench is constructed through a multi-stage pipeline designed to produce a diverse set of video pairs with known quality differences and reliable human preference annotations. The construction involves four sequential stages: video source selection, prompt collection, video generation/collection, and pairing/annotation.

Stage 1: Video source selection. The benchmark includes videos from eight video generative models, selected to span a wide quality range and represent both proprietary and open-source systems. The models and their release dates form a temporal progression: Gen-3 (June 2024), Kling 1.5 (September 2024), LUMA 1.6 (September 2024), Movie Gen Video (October 2024), Open-Sora 1.2 (June 2024), CogVideoX-5B (August 2024), HunyuanVideo (December 2024), and Mochi 1 (October 2024). This temporal spread is deliberate: earlier models like Open-Sora 1.2 and CogVideoX-5B tend to produce more obvious artifacts, while later models like Movie Gen Video and HunyuanVideo produce higher-quality outputs where flaws are subtler. Including both makes the benchmark discriminative across quality levels—some pairs will have clear quality differences (old vs. new model) while others will require fine-grained discrimination (two modern models). Table 2 documents the specific characteristics: resolutions range from 596×336 (360p) to 1920×1080 (1080p), durations cluster around 5.0–5.4 seconds, and frame rates vary from 8 to 60 fps.

Stage 2: Prompt collection from three sources. The text prompts used to generate videos come from three distinct sources to ensure content diversity:

VideoGen-Eval (Zeng et al., 2024): provides over 700 prompts targeting various application scenarios (animation, advertisement) and key video generation capabilities (text alignment, motion diversity). This source was used to generate or collect videos from Gen-3, Kling 1.5, LUMA 1.6, Open-Sora 1.2, and CogVideoX-5B.
Movie Gen Video Bench (Polyak et al., 2024): offers 1,003 prompts covering categories including human activity, animals, and scenery. This source was used for Open-Sora 1.2, CogVideoX-5B, Mochi 1, and HunyuanVideo generations, as well as for directly collecting the Movie Gen Video model's released outputs.
ShutterStock captions: a small portion of video captions from the ShutterStock platform were modified to generate videos with different degrees of Light & Color change—one of the 15 evaluation aspects. For this source, videos were generated using Open-Sora 1.2, CogVideoX-5B, Mochi 1, and HunyuanVideo. Additionally, some non-AIGV real videos related to Light & Color were collected directly from ShutterStock for the real-vs-generated comparison in Section 4.4.

Using three prompt sources rather than one ensures that the benchmark's videos are not biased toward the prompt distribution of any single evaluation dataset, which could inadvertently favor evaluators that were trained or validated on similar prompt styles.

Stage 3: Video generation and collection. For VideoGen-Eval, the authors directly collected videos already generated and released by Zeng et al. (2024) for Gen-3, Kling 1.5, LUMA 1.6, Open-Sora 1.2, and CogVideoX-5B. For Movie Gen Video Bench and ShutterStock, the authors generated videos themselves using Open-Sora 1.2, CogVideoX-5B, Mochi 1, and HunyuanVideo. For Movie Gen Video, they collected the generated videos released by Polyak et al. (2024) and selected the first five seconds to standardize duration.

Stage 4: Pairing and filtering. Videos generated from the same text prompt (across different VGMs) were presented in a pairwise manner. This pairing strategy is important: by comparing videos generated from the same prompt, the evaluation isolates differences attributable to model quality rather than prompt difficulty. If video A from model X and video B from model Y were generated from the same prompt, and humans consistently prefer A, this indicates model X performs better on that prompt—controlling for prompt-specific confounds.

UVE-Bench: The 15-Aspect Taxonomy

The evaluation aspects in UVE-Bench are organized into a hierarchical taxonomy with four top-level categories, each subdivided into 3–4 subaspects, totaling 15 fine-grained evaluation dimensions. This structure (Section 3.2) is inspired by prior work in AIGV evaluation (VBench, FETV, EvalCrafter) but extends coverage significantly—Table 3 shows that existing datasets cover at most 9 aspects, and those that claim more (e.g., VBench's 16) do so using off-the-shelf models rather than human annotations, meaning the aspects are defined by available models rather than by human-judged quality dimensions.

Static Quality (4 subaspects): evaluates the visual quality of individual frames without considering temporal relationships.

Aesthetic Quality: assesses aesthetic elements including frame layout, lighting, and color harmony. This is about whether the frame looks pleasing, not whether it looks realistic.
Technical Quality: examines the presence of unwanted noise, blur, and distortion. This is about technical artifacts that degrade visual clarity, independent of content.
Structural Correctness: checks for abnormal subject structures that contradict common sense, such as a person with three eyes or a hand with six fingers. This is the closest subaspect to detecting "AI weirdness" in generated content.
Overall Static Quality: jointly considers the above three subaspects into a holistic static quality judgment.

Temporal Quality (4 subaspects): evaluates visual quality from a temporal perspective, requiring the evaluator to track how visual properties change across frames.

Appearance Consistency: evaluates whether the appearance and identity of subjects or backgrounds remain consistent across frames. A person's clothing should not change color mid-video; a building should not morph into a different structure.
Temporal Flickering: identifies undesirable flickering and jittering that degrade visual quality. This is a purely technical artifact—rapid, unnatural changes in pixel values that create a strobe-like effect.
Motion Naturalness: determines if subject motions and interactions appear natural and adhere to physical laws. This is arguably the most challenging subaspect because it requires understanding of real-world physics, biomechanics, and object interactions—knowledge that must be learned from training data rather than computed from low-level image statistics.
Overall Temporal Quality: comprehensively assesses temporal visual quality by integrating the three subaspects above.

Dynamic Degree (4 subaspects): evaluates the amount of change in the video, as opposed to the quality of that change (which is the domain of Temporal Quality).

Subject Motion: focuses on how much the subjects in the video move. Two videos could both have natural motion but differ in whether the subject is static or highly active.
Camera Motion: assesses how much the camera moves (panning, zooming, tilting). This is independent of subject motion—a video can have static subjects but dynamic camera work.
Light & Color: considers changes in lighting conditions and color across the video. This includes both natural lighting changes (sun moving behind clouds) and unnatural ones (inconsistent color grading).
Overall Dynamic Degree: combines the three subaspects to provide a comprehensive measure of the video's dynamic degree.

Video-Text Alignment (3 subaspects): evaluates how well the generated video matches the text prompt used to generate it.

Appearance Alignment: focuses on whether subject or scene appearance matches the text description. If the prompt says "a red car" but the video shows a blue car, this is an appearance alignment failure.
Motion Alignment: assesses whether subject or camera motions match the text description. If the prompt says "a person walking slowly" but the video shows running, this is a motion alignment failure.
Overall Alignment: considers how faithfully the entire video content reflects the complete text prompt, integrating both appearance and motion alignment.

This 15-aspect taxonomy is not arbitrary—it reflects a deliberate decomposition of video quality into orthogonal dimensions where possible. The four top-level categories separate "how does it look frame-by-frame?" (Static Quality) from "how does it change?" (Temporal Quality and Dynamic Degree) from "does it match what was requested?" (Video-Text Alignment). Within each category, the subaspects further separate technical/objective measures (Technical Quality, Temporal Flickering) from perceptual/subjective measures (Aesthetic Quality, Motion Naturalness) and from content-specific measures (Subject Motion, Appearance Alignment).

The paper does not claim this taxonomy is exhaustive. Section 5 explicitly identifies future extensions including safety, ethics, bias assessments, and additional generation settings (image-to-video, video-to-video). The 15 aspects represent a substantial expansion over prior work while remaining manageable for human annotation.

UVE-Bench: Human Annotation Procedure

The human annotation procedure is designed to produce pairwise preferences rather than absolute ratings, a critical methodological choice with both theoretical and practical justifications.

Annotation format. Annotators—who are the authors of the paper, described as having "rich research experience in VGMs and MLLMs and are proficient in English"—are presented with pairs of videos generated from the same text prompt. For each pair, they must make a choice from the set $\mathcal{O} = \{\text{"V1 better"}, \text{"V2 better"}, \text{"same good"}, \text{"same bad"}\}$ and identify which evaluation aspect influenced their decision. The four-way choice is more informative than a simple binary "A vs. B" because it distinguishes cases where both videos are good (and the difference is negligible) from cases where both are bad (and neither meets quality standards). This distinction matters for evaluation: an evaluator that correctly identifies "same good" should be rewarded differently from one that incorrectly claims one is better.

Why pairwise preference over absolute rating. The paper explicitly states (Section 3.4) that pairwise comparisons "avoid the inconsistent standards of absolute ratings between humans." This is grounded in established psychophysics: humans are more reliable at comparative judgments than absolute ones because comparison eliminates the need to maintain a consistent internal scale across different stimuli. In absolute rating, annotator A might consider a video "4 out of 5" while annotator B considers the same video "3 out of 5," not because they disagree about the video's quality but because they calibrate their scales differently. In pairwise comparison, both annotators would agree that video X is better than video Y (or vice versa), since the judgment is relative. This methodological choice directly impacts evaluation: the metrics Asingle and Apair are designed to compare MLLM outputs against pairwise preferences, not absolute ratings.

Annotation filtering. To minimize subjectivity, annotators were instructed to "only annotate video pairs for which they could confidently make a preference judgment, and the remaining video pairs are discarded" (Section 3.4). This introduces a deliberate selection bias: the benchmark contains video pairs where quality differences are perceptible and annotatable, excluding pairs where the two videos are truly indistinguishable. This is appropriate because the goal is to measure evaluator discrimination ability; including pairs where even humans cannot distinguish would add noise without signal.

Scale of annotation. Following this procedure, the authors obtained 4,042 pairwise preference annotations across 1,230 individual videos. The distribution (Figure 2c) shows that "V1 better" and "V2 better" annotations are more common than "same good" and "same bad," which is expected given that different VGMs produce systematically different quality levels and pairs from the same prompt often have a clearly superior video. For the video pair comparison evaluation in Section 4, the authors create a balanced subset of 2,411 samples to mitigate bias toward "better" categories in accuracy computation.

Quality verification. To verify annotation reliability, the authors randomly sampled 50 samples per subaspect (approximately 750 samples total, though the exact number depends on how many subaspects have at least 50 samples) and had a different group of three annotators label the same pairs. Including the original annotations, inter-annotator agreement measured by Fleiss' Kappa was 0.803, which indicates "substantial agreement" on the standard interpretation scale (0.61–0.80 is "substantial"; above 0.80 is "almost perfect"). This level of agreement validates that the annotation guidelines produce consistent judgments across different annotators.

UVE-Bench: Evaluation Criteria Connecting MLLM Outputs to Human Judgments

The evaluation criteria are the mathematical bridge between what MLLMs output (scores or choices) and what humans annotated (preferences from $\mathcal{O}$ ). They must handle the asymmetry between the two evaluation modes: single video rating produces continuous scores, while human annotations are discrete preferences.

Single video rating criterion (Asingle). For single video rating, the MLLM produces two scalar scores $S_1, S_2 \in [0, 1]$ for the two videos in a pair. The criterion Asingle maps these scores to a correctness value based on the human preference $P$ :

$A_{\text{single}} = \begin{cases} \mathbb{1}(S_1 > S_2) & \text{if } P = \text{"V1 better"} \\ \mathbb{1}(S_1 < S_2) & \text{elif } P = \text{"V2 better"} \\ f_c(S_1 \mid \beta) \cdot f_c(S_2 \mid \beta) & \text{elif } P = \text{"same good"} \\ f'_c(S_1 \mid \alpha) \cdot f_c(S_2 \mid \alpha) & \text{elif } P = \text{"same bad"} \end{cases}$

where $\mathbb{1}(\cdot) \in \{0, 1\}$ is a binary indicator function, $\alpha < \beta$ are thresholds for "bad" and "good" respectively (by default $\alpha = 0.4$ , $\beta = 0.8$ ), and $f_c$ and $f'_c$ are piecewise functions that handle the continuous-to-discrete mapping for "same" cases.

What this equation computes: a per-sample score between 0 and 1 measuring how well the MLLM's single-video ratings agree with the human pairwise preference. For "better" cases, it is a binary 0/1 based on whether the MLLM's scores correctly order the two videos. For "same good" cases, it is the product of two compatibility scores: $f_c(S_1 \mid \beta) \cdot f_c(S_2 \mid \beta)$ , where each $f_c$ measures how compatible the score is with the judgment "good." For "same bad" cases, it uses $f'_c$ for the first video, measuring compatibility with "bad," while still using $f_c$ for the second—the exact asymmetry is specified in Equations 5 and 6.

Why this form: the piecewise structure separately handles two fundamentally different types of human judgments. For "V1/V2 better," the criterion is an ordinal comparison—did the model correctly identify which video was superior? This is a strict test of relative quality perception. For "same good/bad," the criterion is a calibration test—did the model assign scores in the appropriate range (high for "same good," low for "same bad")? The product form for "same" cases requires both videos to be correctly calibrated; if one video is scored as good but the other as poor when humans said "same good," the product will be low, penalizing the inconsistency. This is more stringent than, for example, averaging the two compatibility scores, which would give partial credit for getting one video right.

The piecewise compatibility functions are defined as:

$f_c(S \mid \beta) = \begin{cases} 1 & \text{if } \beta < S \leq 1 \\ e^{-s \cdot (\beta - S)} & \text{elif } 0 \leq S \leq \beta \end{cases}$

$f'_c(S \mid \alpha) = \begin{cases} 1 & \text{if } 0 \leq S < \alpha \\ e^{-s \cdot (S - \alpha)} & \text{elif } \alpha \leq S \leq 1 \end{cases}$

where $s$ is the coefficient that controls the speed of exponential decay (the paper does not specify the exact value of $s$ , but from the shape of Figure 7, it appears to be set so that scores at the boundary $\alpha$ or $\beta$ decay smoothly but rapidly).

What $f_c$ computes: compatibility of a score with the judgment "good." If the score is above $\beta$ (the "good" threshold), compatibility is 1—the model's score perfectly aligns with human judgment. If the score is below $\beta$ , compatibility decays exponentially from 1 to 0 as the score decreases, meaning a score just below $\beta$ is mostly compatible (the model was close to calling it good) while a score near 0 is completely incompatible.

What $f'_c$ computes: compatibility of a score with the judgment "bad." It is the mirror image: scores below $\alpha$ (the "bad" threshold) get compatibility 1; scores above $\alpha$ decay exponentially toward 0.

Why exponential decay: a linear penalty would treat all deviations from the threshold equally, but this is psychologically wrong. A score of 0.79 when $\beta = 0.8$ should be penalized much less than a score of 0.2 when $\beta = 0.8$ , because the former represents near-agreement (the model was almost ready to call it good) while the latter represents strong disagreement. Exponential decay captures this: it is steep near the threshold and flattens out, meaning small misses are treated more leniently than large ones. This is analogous to how human annotators might disagree: if one annotator says "good" and the model says 0.79, this is a minor calibration difference; if the model says 0.2, this is a fundamental disagreement about the video's quality.

The default thresholds $\alpha = 0.4$ and $\beta = 0.8$ divide the [0, 1] score range into three regions: [0, 0.4) is "bad," [0.4, 0.8] is "ambiguous/moderate," and (0.8, 1.0] is "good." These values are not learned from data but set a priori, and Section D.5 investigates their sensitivity. The finding (Tables 9 and 10) is that while absolute performance changes with different thresholds, the relative ranking of MLLMs remains stable—a desirable property that means the benchmark's conclusions about which models are better evaluators do not depend sensitively on these hyperparameters.

Video pair comparison criterion (Apair). For video pair comparison, the evaluation is simpler because the MLLM directly outputs a choice from $\mathcal{O}$ :

$A_{\text{pair}} = \mathbb{1}(C = P)$

where $C$ is the MLLM's predicted choice and $P$ is the human preference annotation. This is straightforward four-way classification accuracy. The paper does not use a more nuanced metric (e.g., giving partial credit for confusing "V1 better" with "same good" vs. "V2 better") because the four categories are treated as mutually exclusive and equally distinct—confusing any two categories is equally wrong.

Usage in experiments. In the results tables (Tables 4 and 5), Asingle values range approximately from 50 (around random chance, since the baseline for ordinal comparison is 50%) to 80+ for the best models. Apair values range from 25% (random chance for four-way classification) to approximately 55% for the best MLLMs and 88% for humans. These different ranges reflect the different difficulty of the two tasks: single video rating with the continuous-to-discrete mapping in Asingle is inherently an easier evaluation setup because the MLLM only needs to produce relative scores, not explicitly compare, and the exponential decay functions provide partial credit for near-misses. Video pair comparison requires the MLLM to explicitly process two videos and make a discrete four-way choice, which is a harder task.

Design Choices and Their Justifications

Pairwise preference over absolute rating as the annotation format. This is the most consequential methodological decision in the benchmark construction. The justification is both psychometric (humans are more reliable at relative than absolute judgments) and practical (pairwise preferences can evaluate both single-rating and pair-comparison methods, as shown in Figure 2d). The tradeoff is that pairwise annotations provide less information per annotation than absolute ratings—rating each of 1,230 videos on a 1–5 scale would produce 1,230 scalar labels, while 4,042 pairwise annotations collectively provide information about the relative ordering of videos. However, the pairwise format enables the Asingle criterion, which evaluates single-video rating methods against the more reliable pairwise ground truth, effectively using human comparative judgment to validate absolute scoring models.

Fifteen aspects organized into four top-level categories. This taxonomy deliberately separates quality (how good is the video?) from degree (how much is happening?) from alignment (does it match the prompt?). Prior benchmarks often conflate these—treating "dynamic degree" as a quality metric, for instance, when a video can be highly dynamic and high quality, highly dynamic and low quality, static and high quality, or static and low quality. The separation into four categories makes these distinctions explicit. The choice of 15 subaspects (rather than a coarser or finer granularity) balances coverage against annotation tractability: each of the 4,042 annotations includes an aspect label, and finer granularity would either require more total annotations to maintain per-aspect sample sizes or result in insufficient per-aspect data.

Using the latest VGMs spanning a wide quality range. The decision to include eight VGMs from mid-2024 through late 2024, from Open-Sora 1.2 (relatively lower quality) to Movie Gen Video and HunyuanVideo (state-of-the-art), ensures that UVE-Bench contains both easy pairs (old vs. new model, where quality differences are obvious) and hard pairs (two modern models, where differences are subtle). This range is necessary to evaluate whether MLLM evaluators can discriminate at the fine-grained level required for comparing cutting-edge models, not just detect gross artifacts in earlier-generation outputs. Table 6 demonstrates this: Qwen2-VL-72B assigns similar scores to Movie Gen Video and HunyuanVideo (84.3 vs. 84.9 for Static Quality), but human preference rates differ (96.4% vs. 94.7%), suggesting that even advanced MLLMs struggle with the fine discriminations needed at the quality frontier.

Author-annotated with verification by external annotators. Using the paper's authors as primary annotators (Section 3.4) is a practical choice justified by three factors: the authors have domain expertise in VGMs and MLLMs, they are proficient in English (the language of all prompts and annotations), and the annotation task requires understanding of fine-grained quality distinctions that domain novices might miss. The external verification—three additional annotators on a 50-sample-per-subaspect subset achieving Fleiss' Kappa of 0.803—provides evidence that the author annotations are replicable and not idiosyncratic. This is not a perfect substitute for fully independent annotation (the authors' annotations might still reflect shared biases of the research group), but it is a reasonable tradeoff given the scale and expertise requirements of the task.

Frame sampling (16 for single rating, 12 for pair comparison). The frame counts are chosen to balance temporal coverage against MLLM context constraints. Sixteen frames over a ~5-second video corresponds to roughly 3 frames per second, which captures major motion events but misses fine-grained temporal details (e.g., rapid flickering at 10+ Hz would be undersampled). This is a known limitation: Section 4.3.4 shows that increasing frame count improves performance, especially for temporal aspects, suggesting that 16 frames is a computational compromise rather than an information-theoretic optimum. The reduction to 12 frames for pair comparison (24 total) reflects the practical context window limits of current MLLMs—the paper does not specify exact token budgets, but 24 frames at typical MLLM image tokenization rates (hundreds of tokens per frame) would consume tens of thousands of tokens, approaching the practical limits of even long-context models.

Yes/no as default scoring tokens. The choice of "yes"/"no" over alternatives ("good"/"bad," numeric generation) is evaluated empirically in Section 4.3.2. The finding (Figure 4 and related analysis) is that yes/no provides a good balance—it performs well across most aspects, and while adaptive token selection (choosing between yes/no and good/bad per aspect) provides marginal gains for some models (particularly Qwen2-VL-7B), the improvement is small or negligible for advanced models (Qwen2-VL-72B, InternVL2.5-78B). The strength of yes/no is that it maps naturally to the question format ("Is the video good in terms of [aspect]?") and the tokens "yes" and "no" are among the most well-calibrated tokens in LLM vocabularies due to their high frequency in training data. Direct numeric score generation (prompting the model to output "0–100") performs significantly worse (Figure 4, bottom row), likely because MLLMs are not well-calibrated for producing fine-grained numerical quality scores without specific fine-tuning—a finding consistent with the broader literature on using language models for numerical prediction.

The Asingle criterion design (exponential decay with thresholds). This design addresses a fundamental tension: single video rating produces continuous scores, but human pairwise preferences are discrete categories. A naive approach—binarizing scores with hard thresholds—would penalize near-threshold scores harshly and make the metric sensitive to threshold choice. The exponential decay functions $f_c$ and $f'_c$ provide a smooth transition: scores near the threshold receive near-perfect credit, while scores far from the threshold receive near-zero credit. The sensitivity analysis in Section D.5 (Tables 9 and 10) confirms that this smoothness makes the relative model rankings robust to threshold choice—a critical property for benchmark trustworthiness.

The balanced subset for video pair comparison evaluation. The original 4,042 annotations are skewed toward "V1 better" and "V2 better" (Figure 2c), which makes four-way accuracy a biased metric if computed on the full set (a model that always predicts "V1 better" would achieve above-chance accuracy). The creation of a 2,411-sample subset with more balanced category distribution directly addresses this bias, ensuring that the Apair metric in Table 5 reflects genuine discrimination ability rather than exploiting category frequency. This is standard practice in classification benchmark design but is worth noting because it represents an additional curation step beyond the raw annotation collection.

4. Key Insights and Innovations

Innovation 1: Reframing AIGV Evaluation as a Prompting Problem Rather Than a Model-Training Problem

The paper's most fundamental conceptual contribution is a reframing of what AIGV evaluation fundamentally requires. Prior to this work, the dominant assumption—implicit in both the off-the-shelf model approach (VBench, EvalCrafter) and the human-assessment-trained approach (VideoScore, AIGV-Assessor, VIDEOCON-PHYSICS)—was that evaluating a specific video quality aspect requires either a purpose-built model trained on that aspect or a serendipitous off-the-shelf model that approximately measures it. Under this assumption, adding a new evaluation aspect is a model development problem: collect data, train or fine-tune, deploy.

The paper challenges this assumption at its root by proposing that evaluation can instead be a prompting problem: given a sufficiently capable MLLM with strong joint vision-language understanding, evaluating a new aspect requires only writing a natural language description of what to evaluate, not collecting training data or training a new model. This is a category shift, not an incremental improvement. If the hypothesis holds, the marginal cost of adding an evaluation aspect drops from thousands of human annotations plus model training to the cost of writing a few sentences—a reduction of multiple orders of magnitude in both time and resources.

What makes this reframing intellectually distinctive is not the prompting mechanism itself (which is straightforward, as Section 2 demonstrates) but the underlying hypothesis it forces us to test: that MLLMs trained on real-world images and videos have internalized sufficient visual understanding to evaluate synthetic videos on arbitrary quality dimensions, purely through the lens of natural language instructions. This is a non-obvious claim. Real-world visual training data follows physical laws, exhibits predictable motion patterns, and contains subjects with coherent structure. AI-generated videos violate these properties in ways that may be outside the training distribution—a person with six fingers, a building that morphs into a different structure, motion that is locally smooth but globally impossible. Whether an MLLM's pre-trained understanding transfers to evaluating these synthetic artifacts is an empirical question that the paper poses systematically for the first time.

The paper's empirical results partially validate this reframing while also revealing its boundaries. The finding that zero-shot MLLMs "significantly surpass existing specialized evaluation methods" (Section 4.2, Tables 4 and 5) across all 15 aspects—without any AIGV-specific training—supports the core hypothesis. But the 30+ point accuracy gap to humans in pairwise comparison (Table 5: best MLLM at ~54% vs. human at 88%) and the specific weakness in temporal understanding aspects reveal that the reframing is not yet a solution but a direction: the prompting interface works in principle, but current MLLMs lack the fine-grained temporal discrimination capabilities to make it reliable.

This reframing is fundamental, not incremental, because it changes what the field optimizes for. Under the model-training paradigm, progress means building better specialized evaluators for fixed aspects. Under the prompting paradigm, progress means building better MLLMs whose general visual understanding transfers more faithfully to evaluation tasks—a very different research agenda that connects AIGV evaluation to the broader trajectory of multimodal model development.

Innovation 2: Pairwise Preference as a Unified Evaluation Substrate That Supports Both Rating and Comparison Methodologies

A second conceptual innovation is the methodological choice to structure UVE-Bench around pairwise human preferences and to design evaluation criteria that use these pairwise annotations to evaluate both single-video rating methods and video-pair comparison methods. This is not merely an annotation format choice—it is a design principle that unifies two evaluation paradigms that prior work treated as separate.

Existing AIGV datasets fall cleanly into two camps: those providing single-video ratings (FETV, T2VQA-DB, VideoFeedback, LGVQ) and those designed for video pair comparison (VBench, though VBench uses off-the-shelf models rather than human annotations for its pair-based evaluation). This division forces an artificial choice: methods designed for single-video rating cannot be fairly evaluated on pair-comparison benchmarks, and vice versa. The paper's insight is that pairwise human preferences are a more fundamental signal from which both evaluation formats can be derived. Given a human annotation that "Video A is better than Video B," one can evaluate a single-video rater by checking whether its scores correctly order A and B (S_A > S_B). Given the same annotation, one can evaluate a pair-comparison method by checking whether it correctly selects "A is better."

The Asingle metric (Equation 3) is the mathematical instantiation of this insight. It converts pairwise ground truth into a scoring function for single-video raters, using the piecewise exponential-decay functions f_c and f'_c to handle the "same good" and "same bad" cases that have no natural interpretation in pure ordinal comparison. What makes this innovative is that it extracts more information from pairwise annotations than simple accuracy metrics do. The exponential decay provides graded credit for near-misses (a single-video rater that scores two "same good" videos as 0.79 and 0.81 gets near-perfect credit, while one that scores them as 0.2 and 0.3 gets near-zero credit), capturing the intuition that not all errors are equally severe. This graded evaluation is only possible because the Asingle framework treats the MLLM's output as a continuous signal to be evaluated against a discrete preference, rather than discretizing both sides and computing simple accuracy.

The practical significance of this innovation is demonstrated by the fact that the same benchmark and the same human annotations serve both evaluation modes seamlessly (Tables 4 and 5 report results for both), and by the single-to-pairwise adaptation analysis (Section 4.3.3, Figure 6) which shows that single-video ratings can be converted to pairwise choices for models that struggle with direct pair comparison. This is an incremental methodological contribution (pairwise preference is well-established in psychophysics; the specific Asingle formulation builds on prior work in image quality assessment) but one with significant practical consequences for how future AIGV evaluation benchmarks should be designed.

Innovation 3: Difficulty-Conditioned Analysis of MLLM Evaluation Capability — Identifying Which Aspects MLLMs Can and Cannot Evaluate

While the paper does not use the term "difficulty-conditioned" in the same way as the reference example (which adapts test-time strategies per prompt difficulty), it performs an analogous diagnostic move: decomposing overall MLLM evaluation performance into per-aspect and per-category patterns to identify where MLLMs succeed and where they fail, rather than treating evaluation capability as a monolithic property.

This is a conceptual innovation in how we think about MLLM evaluators. The natural question "can MLLMs evaluate AIGVs?" has no single answer—the paper's results show that the answer depends fundamentally on what aspect is being evaluated and what evaluation mode is being used. This decomposition reveals a pattern that would be invisible in aggregate metrics:

MLLMs excel at static, appearance-based, and semantic aspects: Dynamic Degree subaspects (Subject Motion, Camera Motion, Light & Color) achieve Asingle scores of 80+ for advanced models (Table 4: Qwen2-VL-72B at 86.5, 92.6, 92.7, 86.0). Video-Text Alignment, particularly Appearance Alignment, similarly reaches 90+ (Table 4: InternVL2.5-78B at 95.5). These aspects rely on frame-level visual understanding and semantic matching—strengths that align with how MLLMs are trained (image captioning, visual question answering, image-text retrieval).
MLLMs struggle with temporal dynamics and structural understanding: Temporal Quality subaspects—particularly Motion Naturalness and Temporal Flickering—show markedly lower performance (Table 4: Qwen2-VL-72B at 48.9 for Motion Naturalness, compared to 92.6 for Subject Motion under Dynamic Degree). Structural Correctness is consistently among the lowest-scoring subaspects (Table 4: InternVL2.5-78B at 67.2, Qwen2-VL-72B at 60.2). These aspects require fine-grained temporal reasoning and understanding of physical plausibility—capabilities that current MLLMs, trained predominantly on static images or short video clips with coarse temporal resolution, have not yet developed.
The same MLLM can be simultaneously excellent and poor depending on the aspect: InternVL2.5-78B achieves 95.5 on Appearance Alignment and 61.5 on Motion Naturalness—a 34-point spread within the same model. This intra-model variability is as informative as the inter-model variability and would be invisible in any aggregate metric.

This diagnostic decomposition is not merely descriptive; it has prescriptive implications. The finding that MLLMs are weakest on temporal aspects directly motivates the paper's future work directions (Section 5): advancing video encoding techniques to "capture richer temporal information without compromising efficiency." The finding that structural correctness remains difficult even for the best models suggests that MLLMs' understanding of "normal" subject structure—learned from real-world images—does not fully transfer to detecting abnormal synthetic structures, potentially because the training objective optimizes for recognizing what things are rather than detecting what they should not be.

This contribution is incremental in form but fundamental in implication. Aspect-wise breakdown is not new—VBench already evaluates 16 aspects separately. What is new is using human-annotated pairwise preferences (rather than off-the-shelf model scores) as ground truth for this breakdown, enabling genuine measurement of MLLM evaluation capability rather than measurement of how well MLLM outputs correlate with other models' outputs. The revelation that MLLMs have a 30+ point capability spread across aspects—and the identification of temporal understanding as the primary bottleneck—reframes the research agenda from "make MLLM evaluators better overall" to "make MLLMs better at temporal reasoning specifically."

Innovation 4: Zero-Shot MLLM Evaluators Outperform Trained Specialized Evaluators — Evidence That General Visual Understanding Transfers to AIGV Judgment

The paper provides the first systematic evidence that zero-shot MLLM evaluators can outperform specialized evaluators that were explicitly trained or designed for specific AIGV evaluation aspects. This finding, while perhaps unsurprising in retrospect given the rapid advancement of MLLMs, was not obvious ex ante and has significant implications for how the field should allocate effort in evaluation method development.

Consider the specific comparisons (Table 4, "Specialized Evaluators" rows vs. "Unified Evaluators" rows):

VideoScore-v1.1 (8B parameters, fine-tuned on human ratings for 5 aspects): achieves Asingle of 43.1 on Appearance Consistency and 38.9 on Overall Alignment. The best zero-shot MLLMs achieve 70.9 (InternVL2.5-78B) and 95.0 (Qwen2-VL-72B) on the analogous aspects. The gap is not marginal—zero-shot MLLMs more than double the specialized model's performance on some aspects.
VBench (ensemble of off-the-shelf models with hand-designed rules for 16 aspects): achieves 75.6 on Aesthetic Quality and 54.4 on Overall Alignment. Zero-shot MLLMs achieve 83.4 (Qwen2-VL-72B) and 89.0 (Qwen2-VL-72B) on the analogous aspects. Again, the zero-shot approach substantially outperforms the hand-engineered ensemble.
DOVER (58M parameters, trained on real-world video quality ratings): achieves 80.3 on Aesthetic Quality. Zero-shot InternVL2.5-78B achieves 82.3, and InternVL2.5-78B reaches 88.4 on Overall Static Quality (which DOVER partially covers but does not directly evaluate).

What makes this finding intellectually significant is that it reverses the expected relationship between specialization and performance. The conventional wisdom in evaluation method development is that specialized training on in-domain data yields better performance than general-purpose models applied zero-shot. This paper demonstrates that for AIGV evaluation—at least with current-generation specialized models and current-generation MLLMs—the opposite is true: the general visual understanding acquired through massive pre-training on diverse data transfers more effectively to AIGV judgment than the narrow training of specialized evaluators on limited human-annotated datasets.

This finding has direct resource-allocation implications. If a research group needs to evaluate a new AIGV aspect, the paper's results suggest that prompting a large MLLM zero-shot is likely to outperform training a new specialized evaluator from scratch, at a fraction of the cost. The specialized evaluator approach may still be valuable in domains where MLLMs are weak (temporal quality, structural correctness), but for the aspects where MLLMs excel, further investment in specialized evaluators may be misallocated relative to improving the prompting interface or the underlying MLLM.

This is an incremental empirical finding (the superiority of large pre-trained models over task-specific trained models has been demonstrated in many domains) but one with fundamental practical consequences for the AIGV evaluation field. It shifts the burden of proof: the default approach to AIGV evaluation should now be "try zero-shot MLLM prompting first," with specialized training reserved for aspects where zero-shot performance is demonstrably insufficient.

Innovation 5: The Diagnostic Value of the Pairwise Comparison Format — Revealing That MLLMs' Relative Judgment Lags Far Behind Their Absolute Judgment

A subtle but important innovation emerges from the comparison between the single video rating results (Table 4) and the video pair comparison results (Table 5). The gap between MLLMs and humans is dramatically larger in pairwise comparison than in single-video rating. The best MLLMs achieve Asingle scores in the 75–80 range (Table 4, AVG column: InternVL2.5-78B at 78.2, Seed1.5-VL at 80.0) on a metric where random performance is approximately 50. In contrast, the best MLLMs achieve Apair accuracy of only ~54% (Table 5, AVG: Gemini2.5-Flash at 54.6, InternVL2.5-78B at 53.7) on a metric where random performance is 25%, while humans achieve 88%.

This differential gap is not an artifact of the metrics—it reveals something fundamental about the nature of MLLM evaluation capability. Single video rating (Asingle) primarily tests whether the MLLM can produce scores that are monotonically related to quality: when video A is better than video B, the MLLM's score for A should exceed its score for B. This is a relatively weak requirement; the MLLM does not need to know how much better A is than B, only to get the ordering right. The exponential decay functions in Asingle further relax this requirement by providing partial credit for near-threshold cases.

Video pair comparison (Apair) tests a much stronger capability: the MLLM must explicitly compare two videos, weigh their relative strengths and weaknesses on a specific aspect, and articulate a categorical judgment from four options. This requires simultaneously attending to both videos, performing fine-grained comparative reasoning, and handling the ambiguity of "same good" vs. "same bad" vs. directional preference.

The fact that the pair-comparison gap to humans is 30+ points while the single-rating gap is much smaller (humans are not evaluated on Asingle directly, but the best MLLMs achieve scores suggesting ~80% agreement with pairwise ground truth, while humans on a subset achieve 88% accuracy on the harder Apair task) reveals that current MLLMs are much better at producing quality-correlated scores than at performing explicit quality comparisons. They can sense quality differences (reflected in probability differences between "yes" and "no") even when they cannot reliably articulate those differences (reflected in direct pairwise choice accuracy).

This diagnostic insight has both theoretical and practical significance. Theoretically, it suggests that MLLMs' internal representations encode quality information that is partially inaccessible to their explicit reasoning/generation mechanisms—a form of capability-latency gap where the model "knows" more than it can "say." Practically, it motivates the single-to-pairwise adaptation strategy explored in Section 4.3.3 (Figure 6): for models with weak direct comparison capability, extracting single-video scores and converting them to pairwise choices via thresholding can substantially outperform direct pair comparison. This is a diagnostic innovation—it doesn't propose a new method but reveals a structural asymmetry in MLLM evaluation capability that informs how these models should be deployed.

This innovation is incremental in its immediate claim (the observation that MLLMs are better at rating than comparing is consistent with broader findings about LLM calibration and reasoning) but fundamental in its implications for evaluation system design. It suggests that future AIGV evaluation systems should not treat single-video rating and pair comparison as interchangeable modes of the same underlying capability, but rather as distinct tasks that make different demands on MLLM strengths and weaknesses—with pair comparison requiring explicit comparative reasoning that current MLLMs have not yet mastered.

5. Experimental Analysis

Evaluation Methodology

Dataset. UVE-Bench, introduced in this paper, contains 4,042 pairwise human preference annotations across 1,230 individual videos, spanning 15 evaluation aspects organized into four top-level categories (Static Quality, Temporal Quality, Dynamic Degree, Video-Text Alignment). Videos are generated by eight state-of-the-art video generative models (from Open-Sora 1.2 to HunyuanVideo) from text prompts sourced from VideoGen-Eval, Movie Gen Video Bench, and ShutterStock. Human annotations use a four-way preference format: {"V1 better", "V2 better", "same good", "same bad"}. For video pair comparison evaluation, a balanced subset of 2,411 samples is used to mitigate category imbalance bias. The human baseline is established by three annotators evaluating 50 samples per subaspect on this subset.
Base model(s). Eighteen MLLMs are evaluated: 15 open-source models (Video-LLaVA-7B, LongVA-DPO-7B, ShareGPT4Video-8B, VideoLLaMA2.1-7B, mPLUG-Owl3-7B, VideoChat2-Mistral-7B, MiniCPM-V-2.6-8B, LLaVA-OneVision-7B/72B, LLaVA-Video-7B/72B, Qwen2-VL-7B/72B, InternVL-2.5-MPO-8B/78B) and three proprietary models (GPT-4o, Seed1.5-VL, Gemini2.5-Flash). These span model sizes from 7B to 78B parameters and represent diverse architectural choices (frame-by-frame processing vs. native video encoders, different LLM backbones including Qwen2, LLaMA3, InternLM2.5, Mistral, Vicuna). The selection covers the current frontier of publicly available MLLMs with video understanding capability.
Metrics. For single video rating, Asingle (Equation 3) measures how well the MLLM's predicted scores S1, S2 for a video pair align with the human pairwise preference annotation. For "V1 better" and "V2 better" human judgments, it is binary: 1 if the MLLM's scores correctly order the videos, 0 otherwise. For "same good," it computes fc(S1|β) · fc(S2|β), a product of exponential-decay compatibility scores that reward both scores being above threshold β = 0.8. For "same bad," it computes f'c(S1|α) · fc(S2|α), requiring both scores below α = 0.4. For video pair comparison, Apair is simple four-way classification accuracy: 1 if the MLLM's predicted choice matches the human preference exactly, 0 otherwise. Results in tables are reported as percentages (0–100 scale for Asingle; 0–100% accuracy for Apair).
Baselines. Five specialized evaluation systems are compared against the zero-shot MLLMs:
- VideoScore-v1.1 (He et al., 2024): fine-tuned Mantis-Idefics2-8B trained to predict human ratings (1–4 scale) for five aspects; mapped to 8 of UVE-Bench's 15 subaspects (Table 8).
- VBench (Huang et al., 2024): ensemble of off-the-shelf models (CLIP-based ViCLIP, DINO, AMT, MUSIQ, LAION aesthetic predictor, RAFT) with hand-designed rules covering 16 aspects; mapped to 6 UVE-Bench subaspects (Table 8).
- UMTScore (Liu et al., 2023): UMT-L/16 fine-tuned on MSR-VTT for video-text retrieval; used to evaluate Overall Alignment.
- VIDEOCON-PHYSICS (Bansal et al., 2024): 7B model fine-tuned on binary human annotations for physical commonsense and text alignment; used for Motion Naturalness and Overall Alignment.
- DOVER (Wu et al., 2023): 58M-parameter model trained on DIVIDE-3k real-world video quality ratings; used for Technical Quality and Aesthetic Quality.
Additionally, a random baseline is included: 50% for Asingle (since ordinal comparison baseline is chance-level at 50%) and 25% for Apair (four-way classification).
Generation budget / compute accounting. This paper does not involve generation budget in the typical sense (no search or sampling of multiple solutions). The relevant compute consideration is the number of video frames processed, which affects both the amount of visual information available to the MLLM and the computational cost. For single video rating, 16 frames are sampled per video (except Video-LLaVA at 8 frames). For video pair comparison, 12 frames are sampled per video (24 total). Frame count ablation experiments (Section 4.3.4, Figure 5) sweep from 0 to ~15 frames per video to characterize the information-compute tradeoff.
Cross-validation / statistical protocol. No cross-validation is used for model selection since all MLLM evaluations are zero-shot—there is no training or hyperparameter tuning on UVE-Bench. The human baseline uses three annotators evaluating a subset of 50 samples per subaspect, with inter-annotator agreement measured by Fleiss' Kappa (0.803). The balanced subset for pair comparison is created by sampling from the full 4,042 annotations to achieve more uniform distribution across the four preference categories. Sensitivity analyses for the thresholds α and β (Appendix D.5) explore multiple values (β ∈ {0.7, 0.8, 0.9} with α = 0.4; α ∈ {0.3, 0.4, 0.5} with β = 0.8) to verify that relative model rankings are stable.

Main Quantitative Results

Single Video Rating: MLLMs Substantially Outperform Specialized Evaluators

The primary result for single video rating is presented in Table 4, with headline numbers in the AVG (average across all 15 subaspects) column:

Seed1.5-VL achieves the highest average Asingle of 80.0, followed by InternVL2.5-78B-MPO at 78.2, GPT-4o at 75.7, and Qwen2-VL-72B at 75.4. These four models form the top tier, with Seed1.5-VL's 80.0 representing the state of the art among evaluated systems.
In contrast, the best specialized evaluator achieves substantially lower performance on its covered aspects: VBench reaches 87.8 on Overall Dynamic Degree (but only covers 6 of 15 aspects), while DOVER achieves 80.3 on Aesthetic Quality and 69.2 on Technical Quality (covering only 2 aspects). The unified MLLMs match or exceed these numbers while covering all 15 aspects simultaneously.
The performance gap between the best (Seed1.5-VL at 80.0) and the weakest open-source 7B models is substantial but not extreme: the weakest models (Video-LLaVA-7B at 54.5, LongVA-DPO-7B at 61.6) still outperform the random baseline of ~50, confirming that even smaller MLLMs extract meaningful quality signals.
Scale matters significantly: within model families that offer both 7B and 72B variants, the larger model consistently outperforms the smaller one. LLaVA-OneVision: 73.0 → 75.0 (+2.0). LLaVA-Video: 71.0 → 74.0 (+3.0). Qwen2-VL: 70.9 → 75.4 (+4.5). InternVL-2.5-MPO: 72.6 → 78.2 (+5.6). The gap widens further when comparing 7B-class models to the 78B InternVL2.5 or the proprietary Seed1.5-VL and GPT-4o.
Per-aspect performance reveals a clear pattern (Table 4, individual subaspect columns): MLLMs excel at Dynamic Degree subaspects and Video-Text Alignment, moderate on Static Quality, and struggle with Temporal Quality. For example, Qwen2-VL-72B achieves 86.5 (Overall Dynamic), 92.6 (Subject Motion), 92.7 (Camera Motion), and 86.0 (Light & Color) for Dynamic Degree, and 81.8 (Appearance Alignment) and 95.0 (Overall Alignment) for Video-Text Alignment—all above 80. Yet on Temporal Quality subaspects, the same model scores 52.5 (Motion Naturalness), 58.0 (Temporal Flickering), and 48.9 (Appearance Consistency)—substantially lower. This 40+ point intra-model spread is the most diagnostic finding in the table.
Models with native video encoders show a distinctive strength pattern: Video-LLaVA and VideoChat2-Mistral, which use video-native encoders (LanguageBind and UMT-L/16 respectively) rather than frame-by-frame image processing, achieve surprisingly competitive performance on Motion Naturalness (59.4 and 85.6 respectively, compared to Qwen2-VL-72B's 48.9 and InternVL2.5-78B's 61.5). This suggests that architectural choices in video encoding significantly impact temporal understanding, even when aggregate metrics appear similar.
Specialized evaluators are narrowly competitive on their trained aspects but cannot generalize: VBench's 87.8 on Overall Dynamic Degree is competitive with the best MLLMs (~84–86 range), but VBench cannot evaluate Appearance Consistency, Motion Naturalness, Temporal Flickering, or any alignment subaspect. VideoScore-v1.1's performance is consistently below even the weaker 7B MLLMs across all shared aspects: 43.1 on Appearance Consistency vs. 48.1–70.9 for 7B MLLMs, 38.9 on Overall Alignment vs. 62.7–89.0 for 7B MLLMs. The fine-tuned specialized model is not merely worse on average—it is worse on every comparable aspect, despite being explicitly trained for those aspects.

Video Pair Comparison: MLLMs Lag Dramatically Behind Humans

Table 5 presents the pair comparison results, revealing a starkly different picture from single video rating:

The best MLLM achieves only 54.6% accuracy (Gemini2.5-Flash), followed by InternVL2.5-78B-MPO at 53.7%, Qwen2-VL-72B and Seed1.5-VL both at 51.6%, and GPT-4o at 50.2%. These are substantially above the random baseline of 25%, but the absolute numbers are low—the best model is correct only slightly more than half the time on a four-way decision.
Human evaluators achieve 88.0% accuracy on the same task (evaluated on a 50-sample-per-subaspect subset). The gap between the best MLLM (54.6%) and humans (88.0%) is over 33 percentage points, indicating that MLLMs are far from reliable in direct pairwise quality comparison.
The per-aspect pattern partially mirrors single video rating: Performance is relatively higher on Dynamic Degree subaspects (e.g., Qwen2-VL-72B at 71.2% on Camera Motion, InternVL2.5-78B at 61.5% on Light & Color) and on Appearance Alignment (InternVL2.5-78B at 76.6%, Qwen2-VL-72B at 78.6%). Performance is poor on Structural Correctness (Qwen2-VL-72B at only 19.4%, InternVL2.5-78B at 27.1%—barely above random at 25%) and on temporal aspects like Motion Naturalness (InternVL2.5-78B at 33.8%).
However, the ranking of models shifts between the two evaluation modes. Seed1.5-VL leads in single video rating (80.0 vs. InternVL2.5-78B's 78.2) but drops to parity in pair comparison (both at 51.6%). GPT-4o ranks third in single rating (75.7) but drops to 50.2% in pair comparison—only marginally better than some 7B models (LLaVA-Video-7B at 41.1%, Qwen2-VL-7B at 41.1%). This suggests that strong single-rating performance does not guarantee strong pair-comparison performance, reinforcing the paper's finding that these are distinct capabilities.
The human baseline exhibits a different weakness profile: humans achieve 88.0% average, with the lowest subaspect being Motion Alignment at 84.7% and Temporal Flickering at 78.7%. The human difficulty ranking (Temporal Flickering and Motion Alignment being hardest) partially aligns with MLLM difficulty (both struggle with temporal aspects), but humans maintain high accuracy even on their weakest aspects, while MLLMs collapse to near-random on their weakest aspects (Structural Correctness at 19.4% for Qwen2-VL-72B).
Model scale provides diminishing returns in pair comparison: The jump from Qwen2-VL-7B (41.1%) to 72B (51.6%) is +10.5 points. The jump from InternVL2.5-8B (41.8%) to 78B (53.7%) is +11.9 points. But the jump from 72B-class models to humans is +36.4 points—more than 3× the gain from 10× model scaling. This suggests that pair comparison is bottlenecked by a capability (explicit comparative reasoning) that does not automatically emerge from scaling current MLLM architectures.

Adapting Single Video Rating for Pairwise Comparison: A Practical Bridge for Weaker Models

Section 4.3.3 and Figure 6 examine an alternative to direct pair comparison: first independently rate each video using the single-video token-probability method, then convert the resulting scores to a four-way choice via thresholding rules (detailed in Appendix C.3):

For Qwen2-VL-7B, the single-to-pairwise adaptation substantially improves accuracy compared to direct pair comparison (Figure 6, bottom rows vs. top rows). On the AVG metric, adapted comparison achieves roughly 55–62% compared to 41.1% for direct comparison—a gain of 14–21 percentage points. This is a large practical improvement that elevates a 7B model from near-useless (41% on a four-way task with 25% random baseline) to moderately informative.
For Qwen2-VL-72B, the adaptation provides only minimal improvement (Figure 6, comparing the 72B direct and adapted bars). The model's direct comparison capability is already stronger (51.6%), and extracting scores and thresholding them does not add substantial new information. This is consistent with the hypothesis that smaller models lack explicit comparative reasoning capability and benefit from the simplification of the task to independent rating, while larger models have partially developed this capability.
For InternVL2.5-8B vs. 78B (Figure 11 in Appendix D.4), the same pattern holds: the 8B model shows substantial gains from adaptation while the 78B model does not. This cross-model-family replication strengthens the finding that the benefit of single-to-pairwise adaptation is model-size-dependent and likely reflects a fundamental capability gap in comparative reasoning between small and large MLLMs.

Evaluation of Real vs. Generated Videos: MLLM Scores Track Human Quality Judgments

Section 4.4 and Table 6 examine whether MLLM evaluations can distinguish real videos from AI-generated ones and correctly rank video generative models by quality:

Real videos receive consistently higher scores than generated ones from Qwen2-VL-72B: 85.7 for Static Quality and 78.0 for Temporal Quality, compared to 63.2–84.9 and 59.7–77.1 respectively for generated videos from four different VGMs.
The relative ranking of VGMs aligns with human perceptual judgments and model release chronology: Movie Gen Video (October 2024) and HunyuanVideo (December 2024) score 84.3–84.9 on Static Quality, substantially above CogVideoX-5B (70.0) and Open-Sora 1.2 (63.2). Human prefer rates confirm this ordering: Movie Gen Video at 96.4%, HunyuanVideo at 94.7%, CogVideoX-5B at 21.6%, Open-Sora 1.2 at 3.3% for Static Quality.
However, there is a mild disagreement between MLLM scores and human preferences at the top end: Qwen2-VL-72B scores HunyuanVideo slightly higher than Movie Gen Video on Static Quality (84.9 vs. 84.3), but humans prefer Movie Gen Video (96.4% vs. 94.7% prefer rate). This suggests that current MLLMs may not have sufficient resolution to reliably rank state-of-the-art models that are close in quality—the same limitation that the difficulty-conditioned analysis in Section 4.2 identified.

Ablation Studies and Robustness Checks

Impact of prompting strategy (Section 4.3.1, Figure 3): Using simplified prompts that remove aspect-specific descriptions generally reduces performance for InternVL2.5-78B-MPO in single video rating (Figure 3a). The degradation is most pronounced for Temporal Quality subaspects, Structural Correctness, Technical Quality, and Light & Color—aspects that likely benefit most from detailed instruction about what to look for. The AVG Asingle drops from approximately 78.2 to roughly 74–77 (estimated from Figure 3a color scale). For video pair comparison (Figure 3b), the same pattern holds with a smaller magnitude. Interestingly, removing video order indicators ("The first/second video") does not significantly impact performance for InternVL2.5-78B-MPO, contrary to findings in image quality comparison (Zhang et al., 2024), suggesting that recent MLLMs have developed robust temporal order understanding. However, Appendix D.1 (Figure 9) reveals that this prompting advantage is model-dependent: InternVL2.5-8B-MPO, Qwen2-VL-7B, and Qwen2-VL-72B show a less pronounced advantage for detailed prompts, with simplified prompts achieving better performance on several subaspects. The authors hypothesize that these three models "cannot comprehend the detailed description as effectively as InternVL2.5-78B-MPO." Nevertheless, the full prompt maintains a slight average advantage across all models, supporting the recommendation to provide detailed aspect descriptions.

Impact of scoring strategy (Section 4.3.2, Figure 4 and Appendix D.2, Figure 10): Four scoring strategies are compared for Qwen2-VL-7B (Figure 4):

Using "good"/"bad" as scoring tokens outperforms "yes"/"no" for Temporal Quality and Static Quality assessments (e.g., Temporal Quality AVG rises from roughly 50 to roughly 60 on the color scale).
Using "yes"/"no" outperforms "good"/"bad" for Dynamic Degree evaluation (e.g., Overall Dynamic Degree rises from roughly 65 to roughly 85).
An adaptive strategy that selects between the two token pairs per aspect yields the best overall results, with AVG rising to roughly 75 compared to roughly 71 for yes/no alone.
Directly prompting the MLLM to generate numerical scores as text (0–100) performs significantly worse across all aspects, with AVG dropping to roughly 45—near or below random performance on several subaspects.

However, for Qwen2-VL-72B, InternVL2.5-8B-MPO, and InternVL2.5-78B-MPO (Figure 10), the advantage of adaptive scoring is minimal or nonexistent. For InternVL2.5-78B-MPO, the AVG values for yes/no, good/bad, and adaptive are nearly indistinguishable (all clustering around 78–80 on the color scale). This suggests that larger or more advanced MLLMs are more robust to token choice. The consistent failure of direct numeric generation across all models confirms that probability-based scoring is essential—MLLMs cannot reliably produce calibrated numerical quality scores through text generation alone.

Impact of positive vs. negative prompt framing (Appendix D.3, Table 11): Qwen2-VL-7B is evaluated with both positive prompts ("Is the video good in terms of [aspect]?") and negative prompts ("Is the video bad in terms of [aspect]?"), with token assignments inverted in the latter case. For Static Quality and Temporal Quality, the performance gap between positive and negative framing is relatively small (Static Quality: 64.6 vs. 62.7; Temporal Quality: 51.1 vs. 55.0). However, for Dynamic Degree and Video-Text Alignment, positive prompts yield substantially higher performance (Dynamic Degree: 84.6 vs. 66.0; Video-Text Alignment: 85.4 vs. 16.9). The Video-Text Alignment result is dramatic—negative framing collapses performance from 85.4 to 16.9, suggesting that asking "is the video misaligned?" elicits fundamentally different (and worse-calibrated) model behavior than asking "is the video aligned?" This asymmetry may reflect MLLMs being better calibrated for affirmative evaluation than for detecting the absence of quality. The paper adopts positive prompts as the default.

Impact of frame count (Section 4.3.4, Figure 5): The number of video frames is swept from 0 to approximately 15 for four models (Qwen2-VL-7B/72B, InternVL2.5-8B/78B-MPO), evaluated on Aesthetic Quality, Light & Color, and AVG in both single rating (Figure 5a) and pair comparison (Figure 5b):

Zero-frame (text-only) performance is poor: the models have no visual information and their predictions are essentially constant, confirming that UVE-Bench cannot be solved through language shortcuts or prompt-gaming.
Increasing frames improves temporal aspects more than static aspects: For Light & Color (a temporally-dependent aspect), Asingle rises more steeply with frame count than for Aesthetic Quality (a primarily frame-level aspect). For Qwen2-VL-72B on Light & Color, performance at 15 frames is approximately 80–85 on the color scale, up from roughly 60 at 5 frames. For Aesthetic Quality, the same model rises from roughly 65 to 75 over the same frame range—a smaller absolute gain.
Larger models benefit more from additional frames: The 72B/78B models show more consistent improvement in average accuracy with increasing frame count compared to their 7B/8B counterparts. The 7B-class models' curves tend to flatten earlier, suggesting that they saturate in their ability to integrate temporal information from additional frames—a finding consistent with the broader observation that smaller MLLMs have weaker temporal processing capabilities.
For video pair comparison (Figure 5b), the same patterns hold but absolute accuracy is lower and the curves are noisier due to the smaller sample sizes and inherent difficulty of the four-way classification task.

Adapting single ratings for pairwise comparison (Section 4.3.3, Figure 6 and Appendix D.4, Figure 11): The single-to-pairwise adaptation is evaluated for Qwen2-VL-7B/72B (Figure 6) and InternVL2.5-8B/78B-MPO (Figure 11):

For Qwen2-VL-7B, adaptation improves AVG pair comparison accuracy from roughly 41% to roughly 55–62%—a substantial gain that makes the 7B model competitive with or superior to larger models' direct comparison performance on several subaspects. The improvement is particularly large on Dynamic Degree subaspects and Video-Text Alignment.
For Qwen2-VL-72B, the adaptation provides minimal overall benefit—AVG rises by only 1–2 percentage points—though some individual subaspects show moderate gains.
For InternVL2.5-8B-MPO (Figure 11), adaptation provides substantial gains similar to Qwen2-VL-7B, while InternVL2.5-78B-MPO shows minimal benefit. This replication across model families strongly supports the interpretation that the adaptation compensates for weak explicit comparative reasoning in smaller models.
The paper recommends implementing single-to-pairwise adaptation "particularly for MLLMs with limited pairwise comparison capabilities," i.e., 7B/8B-class models.

Sensitivity to α and β thresholds (Appendix D.5, Tables 9 and 10): The Asingle criterion depends on thresholds α (separating "bad" from "moderate") and β (separating "moderate" from "good"). Sensitivity is tested for four MLLMs (Qwen2-VL-2B, LongVA-DPO-7B, Qwen2-VL-7B, Qwen2-VL-72B):

Varying β from 0.7 to 0.9 (with α = 0.4): Lowering β (making the "good" criterion more lenient) increases Asingle for all models. Qwen2-VL-72B rises from 72.5 (at β = 0.9) to 77.3 (at β = 0.7). The relative ranking of the four models is preserved across all β values: Qwen2-VL-72B > Qwen2-VL-7B > LongVA-DPO-7B > Qwen2-VL-2B in all cases.
Varying α from 0.3 to 0.5 (with β = 0.8): Increasing α (making the "bad" criterion stricter) increases Asingle for all models. Qwen2-VL-72B rises from 74.7 (at α = 0.3) to 76.2 (at α = 0.5). The relative ranking is again fully preserved.
The stability of model rankings under threshold variation is the key finding: while absolute performance changes by 3–5 points, the ordering of which models perform better never changes. This supports the reliability of the benchmark's comparative conclusions even if the precise Asingle values depend on threshold choices.

Robustness across MLLM architectures and scales: The consistent evaluation of 18 MLLMs spanning multiple architectures (frame-by-frame vs. native video encoders, various LLM backbones, 7B to 78B scales) serves as an implicit robustness check. The finding that performance correlates with general multimodal capability (larger, more recent models perform better) and that the aspect-wise pattern (strong on static/alignment, weak on temporal) replicates across diverse architectures suggests these are fundamental properties of current MLLM evaluation capability rather than artifacts of specific model designs. The one interesting exception—models with native video encoders performing better on Motion Naturalness despite lower average scores—indicates that architectural choices matter for specific temporal understanding sub-capabilities, but this finding is based on only two models (Video-LLaVA and VideoChat2-Mistral) and would benefit from broader architectural diversity.

Critical Assessment

The paper's central claims, as established in the Executive Summary, are: (1) MLLMs can serve as unified AIGV evaluators, significantly outperforming specialized evaluators across diverse aspects; (2) a substantial gap remains between MLLMs and humans, particularly on temporal aspects; (3) UVE-Bench enables systematic evaluation of this capability. Each claim requires careful examination against the experimental evidence.

Claim 1: MLLMs significantly outperform existing specialized evaluation methods.

The evidence for this claim is strong for single video rating (Table 4) but requires important qualifications. The claim holds clearly when comparing zero-shot MLLMs against the specific specialized evaluators included as baselines: VideoScore-v1.1, VBench, UMTScore, VIDEOCON-PHYSICS, and DOVER. These are reasonable representatives of their respective categories, but they are not the only or necessarily the strongest specialized evaluators that could have been included. For instance, VideoScore-v1.1 was published in mid-2024 and uses an 8B backbone; a contemporary specialized evaluator trained on a larger MLLM (e.g., fine-tuning Qwen2-VL-72B on human ratings) might close or reverse the gap. The paper's comparison is between zero-shot large MLLMs and trained/fine-tuned small-to-medium models or off-the-shelf non-MLLM tools. This is a valid comparison for the paper's research question (can zero-shot MLLMs serve as evaluators without any AIGV-specific training?), but it does not demonstrate that zero-shot evaluation is superior to the best possible specialized approach—only that it outperforms the existing specialized approaches the authors tested.

Additionally, the mapping of specialized evaluators to UVE-Bench aspects (Table 8) involves judgment calls. VBench's "Overall Consistency" is mapped to UVE-Bench's "Overall Alignment," but VBench's metric (ViCLIP cosine similarity) measures a different construct than human-judged alignment to a complete text prompt. The specialized evaluators may be penalized by imprecise aspect mapping rather than by genuine inferiority. A fairer comparison would evaluate specialized evaluators only on aspects they were explicitly designed for, using their native evaluation protocols, rather than forcing them into UVE-Bench's pairwise preference framework. The paper does not report, for example, correlation between VBench's native scores and UVE-Bench human preferences on the 6 mapped aspects—a comparison that would more directly assess whether the specialized evaluator captures the same quality signal as human judgment.

Claim 2: A substantial gap remains between MLLMs and human evaluators, particularly on temporal aspects.

The evidence for this claim is clear and robust, but the magnitude and nature of the gap deserve more scrutiny. The headline 30+ point gap in pair comparison (best MLLM at ~54% vs. human at 88%) is dramatic, but the pair comparison task is inherently difficult—four-way classification with subtle distinctions between "same good" and directional preference. A more informative decomposition would be: how much of the gap is due to MLLMs failing to detect quality differences at all, versus detecting them but misclassifying the type of difference (e.g., calling "V1 better" when humans say "same good")? The paper's single-to-pairwise adaptation results (Figure 6) partially address this: the fact that 7B models' pair comparison accuracy jumps substantially when converted from single ratings suggests that much of the weakness is in the explicit comparison mechanism rather than in quality perception per se. The MLLM can sense quality differences (reflected in token probabilities) but struggles to articulate them as categorical pairwise judgments. This is a more nuanced—and more actionable—diagnosis than "MLLMs are bad at pair comparison."

The temporal quality weakness is well-documented but the diagnosis is shallow. The paper states (Section 4.2) that results "echo with previous research which reveal the limitation of MLLMs in fine-grained video temporal understanding," citing TempCompass (Liu et al., 2024) and TOMATO (Shangguan et al., 2024). This is correct but descriptive rather than explanatory. Why do MLLMs struggle with Motion Naturalness specifically? Is it because they cannot track fine-grained motion trajectories? Because they lack physical world models? Because the frame sampling rate (16 frames over 5 seconds = ~3.2 fps) is too low to capture natural motion dynamics? The paper's frame count ablation (Section 4.3.4) shows that increasing frames helps, but even at 15 frames (~3 fps for a 5-second video), temporal aspects still lag static aspects. A frame rate ablation (e.g., 16 frames at 2 fps vs. 4 fps vs. 8 fps, holding frame count constant) would help disentangle whether the bottleneck is frame count or temporal resolution. The fact that models with native video encoders perform better on Motion Naturalness (VideoChat2-Mistral at 85.6, compared to frame-based models at 48–61) hints that architecture matters—native video encoders may preserve motion information that frame-by-frame processing loses—but this finding is based on only two models and is not explored in depth.

Claim 3: UVE-Bench enables systematic evaluation of unified AIGV evaluation capability.

The benchmark design is thoughtful and addresses genuine gaps in prior work, but several aspects of its construction limit the strength of conclusions that can be drawn. First, the primary annotators are the paper's authors (Section 3.4), described as having "rich research experience in VGMs and MLLMs." While domain expertise is valuable for making fine-grained quality judgments, author-annotated benchmarks are susceptible to inadvertent bias—the authors know their research question and may unconsciously annotate in ways that favor MLLMs (or, conversely, set an unrealistically high human baseline). The external verification (three additional annotators on 50 samples per subaspect, achieving Fleiss' Kappa of 0.803) partially addresses this, but 50 samples per subaspect is a small fraction of the 4,042 total annotations, and the verification is only on agreement, not on whether the annotations are correct in any absolute sense. An ideal benchmark would have all annotations from independent raters who are blind to the research hypotheses.

Second, the benchmark's 1,230 videos from 8 VGMs represent a specific snapshot of video generation quality circa late 2024. As VGMs improve, the quality ceiling of UVE-Bench will rise, and video pairs that currently show clear differences may become indistinguishable. The paper acknowledges this implicitly by including models spanning a wide quality range, but the benchmark's lifespan as a discriminative tool is uncertain. A benchmark that includes only current SOTA models risks becoming obsolete within months; UVE-Bench mitigates this by including older models, but the "hard" pairs (two modern models) are precisely where MLLMs struggle most (Table 6), and these are the pairs that will dominate as all models improve.

Third, the 15 evaluation aspects, while comprehensive relative to prior work, are not independently validated as a taxonomy. Are the 15 subaspects genuinely separable, or do human annotators make correlated judgments across subaspects (e.g., does a video with poor Structural Correctness also tend to be rated poorly on Aesthetic Quality)? The paper reports per-subaspect results but does not analyze inter-aspect correlations in human annotations, which would reveal redundancy in the taxonomy and help prioritize which aspects are most diagnostically valuable.

Missing experiments that would strengthen the paper:

Fine-tuned MLLM baselines: Comparing zero-shot MLLMs against the same MLLMs fine-tuned on human AIGV quality judgments would directly measure how much AIGV-specific training adds beyond general visual understanding. If fine-tuning provides minimal gains, the zero-shot approach is validated. If fine-tuning dramatically improves performance, it suggests that general pre-training is insufficient and investment in AIGV-specific training data remains valuable.
Frame rate ablation: The current frame count ablation holds temporal extent (~5 seconds) constant and varies frame count, which confounds spatial information quantity with temporal sampling density. A complementary ablation that holds frame count constant (e.g., always 16 frames) and varies temporal extent (2.5s vs. 5s vs. 10s video clips, sampled uniformly) would separate whether MLLMs need more frames or more temporal context.
Per-model calibration analysis: The Asingle metric uses fixed thresholds α = 0.4, β = 0.8 for all MLLMs, but different models may have different score distributions. A model that is systematically conservative (scores clustering around 0.5) would be penalized relative to a well-calibrated model even if their relative rankings are equally accurate. Reporting per-model score distributions, or computing optimal per-model thresholds, would reveal whether some models' apparent weakness is a calibration problem rather than a discrimination problem.
Prompt sensitivity beyond the two variations tested: The paper tests only one simplified prompt variant (removing aspect-specific descriptions) and one structural variant (removing video order indicators). The finding that InternVL2.5-78B benefits from detailed descriptions while smaller models do not raises the question: what is the optimal level of prompt detail as a function of model capability? A systematic prompt complexity sweep would inform deployment decisions about how much prompt engineering effort to invest per model.
Statistical significance testing: The paper reports point estimates for Asingle and Apair without confidence intervals or statistical tests. Given the test set sizes (4,042 annotations, 2,411 in the balanced subset, further divided into 15 subaspects), some of the per-subaspect comparisons—especially for smaller models or on low-sample aspects—may not be statistically distinguishable. A difference of 2–3 points in Asingle between two models on a subaspect with ~100–200 examples may be within sampling error.

Conditions under which the claims hold:

The claim that MLLMs significantly outperform specialized evaluators holds for the specific baseline systems tested, on the UVE-Bench dataset, under the zero-shot evaluation protocol described in Section 2. It does not necessarily hold for: specialized evaluators trained on larger backbone models, aspects where the specialized evaluator was explicitly designed and the MLLM is weak (e.g., Motion Naturalness where VIDEOCON-PHYSICS at 68.7 is competitive with or exceeds several 7B MLLMs in single rating), or evaluation protocols other than token-probability scoring from a fixed prompt template.

The claim that a substantial human-MLLM gap remains holds for the pair comparison task specifically—the gap in single video rating, while not directly comparable due to different metrics, appears smaller. The claim about temporal aspects being the primary weakness holds across all models tested but might not hold for future MLLMs with improved temporal processing (e.g., models with native video encoders and higher frame rate processing).

The claim that UVE-Bench enables systematic evaluation is well-supported for the specific dimensions the benchmark covers but does not extend to evaluation aspects not represented in the 15-subaspect taxonomy (safety, ethics, bias, as the paper acknowledges in Section 5), generation paradigms beyond text-to-video (image-to-video, video-to-video), or video durations substantially beyond the ~5-second clips in the benchmark.

Overall assessment: The experimental design is competent and adequately tests the paper's core hypothesis that zero-shot MLLMs can serve as unified AIGV evaluators. The breadth of models tested (18 MLLMs) and aspects covered (15) is a genuine strength, and the consistency of the aspect-wise performance pattern across diverse architectures lends credibility to the findings. The principal weaknesses are the reliance on author annotations (partially mitigated by external verification), the absence of fine-tuned MLLM baselines (which would contextualize the zero-shot results), and the limited diagnostic depth on why MLLMs fail on temporal aspects (the paper identifies the phenomenon but does not isolate its causes). These weaknesses do not invalidate the paper's contributions but do narrow the strength of its conclusions: the paper convincingly demonstrates that zero-shot MLLM evaluation is promising and better than existing narrow evaluators, but does not establish that it is sufficient for practical deployment without further advances in MLLM temporal understanding.

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Unaccounted for and Prohibitively Expensive

The assumption or constraint. The entire compute-optimal inference framework depends on knowing each prompt's difficulty before allocating the test-time compute budget. The paper's difficulty estimation procedure—whether oracle (ground-truth pass@1 over 2,048 samples) or predicted (PRM average score over 2,048 samples)—requires generating and scoring 2,048 complete solutions per question. Section 3.2 acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency gains—4× improvement over best-of-N, matching performance with 4× fewer generations—are computed after difficulty is known, without amortizing the cost of learning it. The difficulty estimation step alone consumes 8–32× more compute than the largest test-time budgets studied (256–512 generations). In realistic deployment, total cost would be 2,048 generations (difficulty estimation) + N generations (strategy execution), and the former dominates until N becomes impractically large. This means the reported 4× efficiency figure is an upper bound on achievable gains that cannot be realized in practice without a cheaper difficulty estimation method. A practitioner deploying this system would find that the overhead of difficulty estimation erases most or all of the reported savings for any reasonable per-query budget.

What evidence exists in the paper. The difficulty estimation method is described in Section 3.2: "For each question in the test set, the authors sample 2048 complete solutions from the base model and compute the pass@1 rate." The predicted version also uses 2,048 samples, scored by the PRM. The paper explicitly notes this cost is not accounted for (quote above). No experiment measures total cost including difficulty estimation, and no ablation explores how performance degrades if difficulty is estimated from fewer samples.

Mitigation status. The paper acknowledges this as a limitation and flags it as "a key avenue for future work" (Section 3.2), suggesting that models could be trained to predict difficulty directly from question text. No such model is developed or evaluated. The exploration-exploitation framing in Section 3.2—"compute spent assessing difficulty versus compute spent solving the problem"—recognizes the structural tension but provides no resolution. A practitioner currently has no practical path to deploying the compute-optimal framework without incurring prohibitive difficulty estimation costs.

Hard Problems Remain Completely Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper's approach assumes that the base model can produce correct solutions with non-trivial probability. When the base model's pass@1 is effectively zero on a problem class, no amount of test-time compute—search, revision, or their optimal combination—can produce correct answers because there are no correct solutions in the proposal distribution to find or refine. Section 7 explicitly states this boundary condition:

"On the hardest problems (bin 5), test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time."

The consequence. For problems outside the base model's capability range (difficulty bin 5), all test-time compute strategies are equally useless. This is not a matter of diminishing returns—it is a hard failure mode. The FLOPs-matched comparison (Section 7, Figure 9) shows that on bin 5, the smaller model with compute-optimal test-time scaling achieves near-zero accuracy (~0–5%) while the 14× larger model also achieves limited but non-zero performance. In the PRM search FLOPs-matched comparison at R ≫ 1, hard questions show a −52.9% relative disadvantage compared to simply using the larger pretrained model. For production deployments where a non-trivial fraction of queries falls into this "too hard" regime, compute-optimal test-time scaling provides no benefit and may be strictly worse than routing those queries to a larger model or flagging them for human review.

What evidence exists in the paper. Figure 3 (right, bin 5): all search methods operate at 1–3% accuracy regardless of budget. Figure 7 (right, bin 5): all sequential-to-parallel ratios yield ~2–3% accuracy. Figure 9 (bin 5): the compute-optimal scaling curve is essentially flat near 0–5% and falls below the 14× larger model's performance at all R values for PRM search and at R ≫ 1 for revisions. The paper explicitly acknowledges this in the Section 7 takeaway: test-time compute "amplifies existing capability but does not create it."

Mitigation status. Fully acknowledged but not addressed. The paper positions this as a fundamental boundary condition rather than a limitation to be solved: test-time compute and pretraining compute serve complementary roles, and problems outside the base model's reach require additional pretraining. Section 8 does not propose methods to extend test-time compute to harder problems. This is a fair characterization—the limitation is inherent to the proposal distribution framework—but it means the approach offers no path forward for genuinely novel or out-of-distribution reasoning.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Making the FLOPs-Matched Comparison Asymmetric

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by 14× while holding training data fixed, following the LLaMA training paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both parameters and data are scaled equally (Hoffmann et al., 2022). The paper acknowledges this choice explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model is evaluated using only greedy decoding—no test-time compute augmentation whatsoever (no majority voting, no best-of-N, no search).

The consequence. The reported advantages of test-time compute over pretraining are measured against a suboptimal pretraining baseline. A Chinchilla-optimal model trained with 14× more total FLOPs would allocate additional compute to both parameters and training data, likely yielding higher performance than a parameter-only-scaled model. The paper's comparison therefore overstates the advantage of test-time compute relative to the best possible use of equivalent pretraining FLOPs. For the PRM search comparison specifically, the larger model's greedy decoding means it has zero test-time compute budget—a stark asymmetry given that the smaller model receives up to hundreds of generations' worth of test-time FLOPs. A fairer comparison would give the larger model at least a modest test-time compute budget (e.g., best-of-8 or best-of-16), which would consume a small fraction of the total FLOPs but could substantially improve its performance. The −52.9% relative disadvantage for test-time compute on hard problems at R ≫ 1 (Section 7) may be partially attributable to this baseline weakness rather than a fundamental inferiority of test-time compute in that regime.

What evidence exists in the paper. Section 7 describes the FLOP accounting and states the parameter-only scaling choice. The larger model is described as using "greedy decoding, no extra test-time compute." The paper does not include a baseline where the larger model receives any test-time compute budget, nor does it estimate how a compute-optimally-trained larger model would perform. The dependence of the FLOPs-matched comparison on the pretraining scaling strategy is unmeasured.

Mitigation status. The paper acknowledges the non-optimality of parameter-only scaling and defers compute-optimal pretraining comparison to future work. No sensitivity analysis is performed to estimate how results would change under Chinchilla-optimal scaling. The authors' framing of this as "representative of a canonical approach" is defensible—many production models follow the LLaMA paradigm—but the asymmetry remains unquantified. A practitioner interpreting the headline result ("test-time compute with a smaller model can outperform a 14× larger model") should understand that this has been demonstrated only against a parameter-scaled, greedy-decoding baseline, not against the best possible use of equivalent total FLOPs for pretraining.

Verifier Over-Optimization Is a Hard Ceiling That the Compute-Optimal Policy Mitigates but Does Not Solve

The assumption or constraint. All search-based test-time compute methods rely on the process reward model (PRM) to score partial solutions and guide search. The PRM is imperfect—it assigns high scores to some incorrect solutions and low scores to some correct ones. As search becomes more aggressive (higher beam width, more lookahead steps, larger generation budgets), it increasingly finds solutions that score highly under the PRM but are actually incorrect—a phenomenon the paper terms "verifier over-optimization." The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead of beam search where over-optimization is most acute), but it does not improve the PRM itself.

The consequence. Verifier over-optimization imposes a hard performance ceiling that cannot be exceeded regardless of how much additional test-time compute is allocated. On easy problems (difficulty bins 1–2), beam search performance actually degrades with increasing budget (Figure 3, right: bin 1 accuracy drops from ~78% to ~77% as budget increases from 4 to 256 generations). On medium problems (bins 3–4), beam search provides gains but plateaus well before the maximum budget is exhausted. The most powerful search method—lookahead search—paradoxically performs worst overall (Figure 3, left) because its aggressive optimization amplifies PRM errors. Qualitative examples in Appendix M show search producing degenerate outputs: repetitive low-information steps and overly short 1–2 step solutions that score highly under the PRM.

What evidence exists in the paper. Figure 3 (right): beam search degrades on bin 1 with increasing budget—the canonical over-optimization signature. Figure 3 (left): lookahead search underperforms simpler methods at equivalent budgets. Appendix M (Figures 29 and related): qualitative examples of degenerate search outputs. The paper explicitly identifies over-optimization as a central limiting factor in Section 5.3: "The degradation at high budgets is attributed to over-optimization of the PRM—search finds solutions that score highly under the PRM but are actually incorrect."

Mitigation status. The compute-optimal policy partially mitigates over-optimization by using best-of-N (weaker optimization) on easy problems where the PRM is reliable and beam search (stronger optimization) only on medium problems where the PRM signal has more room to provide genuine guidance. However, this is a routing strategy, not a solution—over-optimization still limits performance on the medium problems where beam search is deployed, and improving the PRM itself (e.g., through better training data, adversarial robustness, or ensemble methods) would likely shift the difficulty thresholds and raise the overall performance ceiling. Section 8 identifies verifier robustness as a key research direction but provides no concrete improvements. A practitioner should understand that additional test-time compute beyond a certain budget provides zero or negative marginal benefit due to this ceiling, and that the budget at which this occurs depends on the verifier's quality.

Single Benchmark, Single Model Family — No Evidence of Generalization to Other Domains, Models, or Evaluation Protocols

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021)—500 test questions from high-school competition mathematics—with PaLM 2-S* as the base model. The paper states (Section 4) that they "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is untested. No experiments are conducted on code generation, logical reasoning, scientific question answering, open-ended generation, or any domain outside competition mathematics. No models from other families (LLaMA, Qwen, DeepSeek, etc.) are evaluated.

The consequence. The difficulty-dependent patterns that form the core of the paper's contribution—beam search hurts easy problems but helps medium ones, sequential revisions dominate on easy problems but balanced parallel-sequential ratios are optimal on hard ones, the 4× efficiency gain from compute-optimal allocation—may be specific to (a) the MATH benchmark's particular distribution of problem difficulties and error types, (b) PaLM 2-S*'s specific capability profile and calibration properties, or (c) mathematical reasoning as a domain. Mathematical reasoning has characteristics that may not generalize: problems have unambiguous correct answers (enabling clean PRM training via Monte Carlo rollouts), the reasoning process is multi-step and logically structured (making step-level verification meaningful), and the difficulty distribution spans a wide range from arithmetic to competition-level proofs. For code generation, the "step" structure is different (lines of code vs. natural language reasoning steps). For factual QA, correctness depends on knowledge retrieval rather than multi-step inference. For open-ended generation, "correctness" is ill-defined and verifier training becomes fundamentally more complex.

Additionally, the PRM is trained on PaLM 2-S*'s output distribution using Monte Carlo rollouts. A different base model with different output characteristics (e.g., different typical error patterns, different step granularity, different calibration) would require a new PRM, and the PRM quality—and hence the over-optimization threshold—would change. There is no evidence that the paper's specific findings about optimal beam width, optimal sequential-to-parallel ratios, or difficulty bin boundaries transfer to other model families.

What evidence exists in the paper. All experiments in Sections 5–7 use MATH + PaLM 2-S*. Table 1 in Section 4 lists only MATH. No ablation studies test different base models or different benchmarks. The paper does not claim generalization—it presents findings as specific to the studied setting—but also does not systematically discuss which findings are likely domain-specific versus universal.

Mitigation status. The paper does not address this limitation directly. Section 8's future work mentions "extension to other domains and modalities" as a direction, acknowledging the scope limitation implicitly. No experiments are proposed or conducted to test transfer. A practitioner considering this approach for a non-math domain or a non-PaLM model family should treat the paper's specific quantitative findings (optimal beam width of M=4, 4× efficiency gain, bin boundaries) as illustrative rather than predictive and should replicate the analysis on their target domain and model. The qualitative finding—that difficulty-conditioned allocation matters—is more likely to generalize than the specific numbers, but even this depends on the existence of a reliable verifier and a reasonable difficulty estimation method, both of which are domain-specific challenges.

Sequential Revisions Introduce Latency That Is Not Accounted for in the Compute Budget Model

The assumption or constraint. The paper measures test-time compute in "generations"—the number of complete solutions sampled from the model. This is a reasonable proxy for total FLOPs but ignores wall-clock time and the serial dependencies inherent in sequential revision strategies. A sequential revision chain of length D requires D sequential forward passes through the model, each depending on the output of the previous one. Parallel best-of-N sampling of N solutions can (with sufficient hardware) execute all N forward passes simultaneously.

The consequence. The compute-optimal policy frequently favors sequential-heavy strategies, particularly for easy problems where revisions are most effective (Figure 7, right: bin 2 shows optimal performance at fully sequential or high sequential-to-parallel ratios). A strategy that allocates 128 generations as 64 sequential × 2 parallel chains takes approximately 64× longer wall-clock time than running 128 parallel samples simultaneously, even though both consume comparable total FLOPs (the sequential strategy uses fewer FLOPs due to shorter average generation length, but the latency is dominated by the chain depth). For latency-sensitive applications—interactive assistants, real-time decision-making, any deployment where users wait for responses—the sequential-heavy strategies favored by the compute-optimal policy may be practically infeasible regardless of their accuracy advantages or FLOPs efficiency.

This tradeoff is not merely an implementation detail; it is a fundamental tension between two optimization objectives: minimizing total FLOPs (where sequential revisions are efficient because later revisions are shorter and more targeted) and minimizing wall-clock latency (where parallelism is essential). The paper optimizes for the former but provides no framework for navigating the tradeoff with the latter.

What evidence exists in the paper. The revision model inference procedure is described in Section 6.1: "At test time, the revision model generates a chain of revisions: it produces an initial answer, then conditions on that answer to produce a revision, then conditions on the revision to produce another revision, and so on." The sequential dependency is explicit. The generation budget model in Section 3.1 counts each revision as one generation regardless of its position in the chain, treating all generations as equivalent units of compute. No latency measurements, wall-clock time analyses, or discussions of the throughput-latency tradeoff appear anywhere in the paper.

Mitigation status. Not acknowledged as a limitation. The paper frames compute budgets exclusively in terms of generation count, which abstracts away the serial vs. parallel distinction. There is no discussion of how a practitioner should navigate the latency implications of the recommended strategies, no suggestion of latency-aware allocation policies (e.g., capping maximum sequential chain depth for time-sensitive queries), and no proposal for architectures that could parallelize revisions (e.g., speculative revision where multiple revision branches are explored in parallel). This is a significant practical gap: a practitioner deploying the compute-optimal policy in a latency-constrained setting would need to independently determine how to trade off the FLOPs-optimal strategy against response time requirements, a problem for which the paper provides no guidance.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around MLLM evaluation from "can these models evaluate specific aspects of video quality?" to "can they serve as general-purpose, zero-shot evaluation interfaces across arbitrary quality dimensions?" This is a conceptual reframing, not a paradigm shift: the prompting mechanism is straightforward and builds on established work, but the systematic demonstration that a single MLLM, with no AIGV-specific training, can simultaneously match or exceed specialized evaluators across 15 distinct aspects fundamentally changes how the field should think about evaluation infrastructure.

The magnitude of this reframing is moderate but catalytic. It does not introduce new model architectures or training paradigms, but it establishes a new default for evaluation method development. Before this work, a researcher needing to evaluate a new AIGV quality aspect faced a choice: adapt an off-the-shelf model with heuristic rules (limited coverage, distribution shift) or collect human annotations and train a specialized evaluator (expensive, slow, non-extensible). After this work, the default first step should be "try prompting a large MLLM zero-shot." The paper's evidence—that zero-shot MLLMs outperform trained specialized evaluators like VideoScore-v1.1 and VBench across nearly all 15 aspects (Section 4.2, Tables 4 and 5)—makes this default rationally defensible rather than aspirational. Specialized evaluator training is now a fallback for aspects where zero-shot MLLM performance is demonstrably insufficient, not the starting point.

The paper also serves as a reconciliation mechanism for conflicting intuitions about MLLM evaluation capability. Prior work using MLLMs for AIGV evaluation (VideoScore, AIGV-Assessor, VIDEOCON-PHYSICS) treated MLLMs as backbones for fine-tuning, implicitly assuming that general pre-training is insufficient and that AIGV-specific human data is necessary. Other work (Q-Bench-Video, Zhang et al., 2024) showed that zero-shot MLLM prompting works for image quality assessment, suggesting the opposite. This paper's aspect-wise breakdown resolves the apparent contradiction: both intuitions are partially correct, but they apply to different evaluation aspects. MLLMs can evaluate video-text alignment and dynamic degree zero-shot with near-human-level scores (~90+ Asingle, Table 4), supporting the zero-shot hypothesis. But for temporal quality and structural correctness, even the best MLLMs achieve only modest performance (~50–65 Asingle, Table 4), suggesting that AIGV-specific training or architectural improvements remain necessary. The field no longer needs to argue about whether MLLMs "can" evaluate AIGVs in general—the answer is "it depends on the aspect"—and can instead focus on understanding what drives the capability boundary and how to push it outward.

The paper redirects research attention in several specific ways:

Toward temporal understanding as the primary bottleneck. The finding that models achieving 90+ on appearance alignment drop to ~50 on motion naturalness (InternVL2.5-78B: 95.5 vs. 61.5 in Table 4) makes temporal dynamics the single most important target for improving MLLM evaluation capability. Improvements in general multimodal reasoning or scale alone are unlikely to close this gap—the paper shows that 78B models still dramatically underperform on temporal aspects despite their advantages on static aspects. This suggests that temporal understanding requires architectural or training innovations specific to video dynamics, not just more parameters or data on the current training paradigm. The paper's finding that models with native video encoders (VideoChat2-Mistral at 85.6 on Motion Naturalness vs. ~50–65 for frame-by-frame models in single rating, Table 4) provides a concrete lead: video-native architectures may preserve motion information that frame-based processing loses, and investigating this architectural distinction should be a priority.

Toward the capability-latency gap in comparative reasoning. The dramatic disparity between single-rating and pair-comparison performance—Seed1.5-VL achieves 80.0 Asingle but only 51.6% Apair accuracy—reveals that MLLMs can sense quality differences (reflected in token probabilities) much more reliably than they can articulate them (reflected in explicit pairwise judgments). This is a form of the "competence-performance gap" known from cognitive science: the model's internal representations encode quality information that its generation mechanism cannot reliably access. This finding makes explicit comparative reasoning a distinct research target, separate from quality perception. It also suggests that the single-to-pairwise adaptation strategy (Section 4.3.3, Figure 6), while a practical workaround, is addressing a symptom rather than a cause—the real advance would be MLLMs that can directly compare videos as reliably as they can rate them individually.

Toward prompting and calibration as first-class design dimensions. The paper's ablation studies (Sections 4.3.1–4.3.3) demonstrate that seemingly minor design choices—which tokens to use for scoring, whether to include detailed aspect descriptions, whether to frame questions positively or negatively—can swing performance by 10–20 points on some aspects. The catastrophic collapse of Qwen2-VL-7B on Video-Text Alignment when using negative prompts (85.4 → 16.9, Table 11) is a striking example. This elevates prompting engineering from an implementation detail to a central research question for MLLM-based evaluation: understanding why MLLMs are asymmetric in their ability to evaluate "goodness" versus "badness," and developing robust prompting protocols that don't require per-model, per-aspect tuning, is essential for the unified evaluator vision to be practical.

Away from specialized evaluator development for well-covered aspects. The finding that zero-shot MLLMs substantially outperform VideoScore-v1.1 and DOVER on their own trained aspects (Table 4) suggests that further investment in training narrow evaluators for aspects like aesthetic quality, technical quality, or appearance alignment may be resource-misallocated. The specialized evaluator community should redirect effort toward aspects where MLLMs are weak (temporal quality, structural correctness) and toward methods that leverage MLLMs' existing capabilities rather than training separate models from scratch. The paper does not kill specialized evaluation—the 30+ point pair-comparison gap to humans makes that clear—but it narrows the domain where specialization is necessary and raises the performance bar that specialized methods must clear to justify their development cost.

Follow-Up Research This Work Enables

Fine-tuning frontier MLLMs on AIGV-specific human judgments and measuring the zero-shot-to-fine-tuned gap per aspect. The paper evaluates only zero-shot MLLM performance, leaving open the question of how much AIGV-specific training adds beyond general visual understanding. A direct follow-up would take the strongest zero-shot model (Seed1.5-VL or InternVL2.5-78B) and fine-tune it on human AIGV quality ratings for all 15 UVE-Bench aspects, then measure the per-aspect improvement over zero-shot baselines. The key question is whether fine-tuning helps uniformly or selectively: does it close the gap on temporal aspects where zero-shot performance is weak, or does it primarily improve calibration on already-strong aspects? If fine-tuning provides large gains on Motion Naturalness and Temporal Flickering but minimal gains on Appearance Alignment, this would suggest that general pre-training is sufficient for semantic alignment but insufficient for temporal dynamics, pointing toward architectural improvements rather than more training data as the solution. If fine-tuning provides minimal gains across all aspects, the zero-shot approach is validated and effort should shift entirely toward better base MLLMs rather than AIGV-specific training. UVE-Bench's pairwise annotation format supports such fine-tuning directly—the preference annotations can be converted to training signals via standard preference optimization methods—making this experiment tractable with existing infrastructure.

Training a lightweight difficulty/quality classifier from MLLM internal representations to enable cheap, real-time evaluation without full MLLM inference. The paper's evaluation protocol requires running a full MLLM forward pass for every video, which is computationally expensive (72B-parameter models for every 5-second clip) and limits deployment in high-throughput settings. A promising follow-up would investigate whether the quality-relevant information that MLLMs extract can be distilled into a lightweight model. Specifically: take the hidden state representations from a frozen MLLM (e.g., InternVL2.5-78B) at the position of the scoring token for each of the 15 UVE-Bench aspects, train a small regression model (e.g., a few million parameters) to predict the MLLM's Asingle score from these representations, and then investigate whether this distilled model can be run without the full MLLM—either by using a smaller vision encoder with the distilled head, or by training the head directly on visual features from a lightweight encoder (SigLIP, DINOv2) with the MLLM scores as targets. UVE-Bench provides the necessary training and evaluation data, with 4,042 annotated pairs covering all 15 aspects. The critical metric would be the correlation between distilled model scores and human preferences, compared against both the original MLLM and specialized evaluators, at 10–100× lower computational cost.

Frame-rate ablation to disentangle whether MLLMs' temporal quality weakness is a frame count ceiling or a temporal resolution ceiling. The paper's frame count ablation (Section 4.3.4, Figure 5) varies the number of frames while holding video duration constant at ~5 seconds, meaning frame count and temporal sampling rate are confounded. A clean follow-up would hold frame count fixed (e.g., always 16 frames for single video rating) and vary the temporal window those frames span: extract 16 frames from the first 1 second (16 fps), first 2.5 seconds (~6 fps), full 5 seconds (~3 fps), or full 10 seconds (~1.6 fps). This would separate whether MLLMs need more total frames (information quantity) or more densely sampled frames (temporal resolution) to evaluate motion naturalness and temporal flickering. If performance improves with shorter temporal windows at fixed frame count, the bottleneck is temporal resolution—MLLMs need frames close together in time to track motion. If performance is flat or improves with longer windows, the bottleneck is temporal context—MLLMs need to see the full action unfold. The answer has direct implications for video encoding design: temporal resolution bottlenecks motivate higher frame rate sampling, while context bottlenecks motivate longer video segments. UVE-Bench's motion naturalness and temporal flickering subaspects provide the evaluation signal for this experiment, and the paper already demonstrates the basic methodology (Figure 5).

Multi-benchmark replication of the aspect-wise MLLM capability profile to test domain generality. The paper's entire analysis uses UVE-Bench, which evaluates text-to-video generation from prompt-following VGMs. Whether the finding that MLLMs excel at static/alignment aspects but struggle with temporal aspects is specific to AIGV evaluation or reflects a general property of MLLM video understanding is unknown. A systematic replication would test the same 18 MLLMs (or their currently available equivalents) on: (a) real-world video quality assessment benchmarks (e.g., LSVQ, KoNViD-1k for compression artifacts, noise, flickering) to test whether the temporal weakness is specific to AI-generated artifacts; (b) general video understanding benchmarks with temporal reasoning components (TempCompass, TOMATO, MVBench) to test whether temporal evaluation weakness correlates with broader temporal reasoning deficits; (c) AIGV benchmarks from different generation paradigms (image-to-video using VBench's I2V subsets, video-to-video for editing quality). If the aspect-wise profile is consistent across all settings, it reflects a fundamental property of current MLLM architectures. If MLLMs perform well on real-world temporal quality but poorly on AIGV temporal quality, the issue is distribution shift rather than temporal processing—requiring different solutions (domain adaptation vs. architecture changes). UVE-Bench's detailed aspect taxonomy and the paper's evaluation protocol provide the template for such multi-benchmark comparisons.

Investigation of why negative prompts catastrophically fail on alignment aspects, using mechanistic interpretability. The finding that Qwen2-VL-7B's Video-Text Alignment performance collapses from 85.4 to 16.9 when framing the question negatively ("Is the video misaligned?" instead of "Is the video aligned?") is striking and unexplained (Table 11). A mechanistic follow-up would use attention head analysis or activation patching to identify where in the MLLM's processing the positive/negative asymmetry emerges. Does the model attend to different visual regions when asked about alignment vs. misalignment? Does the negative framing suppress the model's ability to retrieve the text prompt from context? Does the probability mass shift toward entirely different vocabulary items ("no" vs. "yes" base rates)? Understanding this asymmetry is practically important—if MLLMs are inherently worse at detecting the absence of quality than the presence of quality, evaluation prompts must always be framed positively, and evaluation of "undesirable" aspects (safety violations, bias, harmful content) may require fundamentally different prompting strategies. The UVE-Bench alignment subaspects provide a clean testbed because the ground truth is symmetric (a "misaligned" video is simply one that fails to align, so positive and negative framings should be informationally equivalent), making any performance gap clearly attributable to model processing rather than task structure.

Combining MLLM-based evaluation with the generative process itself to create targeted diagnostic feedback. The paper evaluates MLLMs as passive judges of completed videos, but the unified evaluation framework could be integrated into the video generation pipeline to provide actionable diagnostic feedback. A concrete experiment: for a video that an MLLM scores poorly on Appearance Consistency (e.g., "the person's shirt changes from blue to red"), can the same MLLM identify which frames exhibit the inconsistency and describe the specific artifact? This would require extending the evaluation protocol from single-score or choice output to spatial-temporal localization and natural language description. UVE-Bench's human annotations could be extended to include frame-level artifact annotations, and the MLLM's attention maps or feature attributions could be compared against these ground-truth localizations. Success would transform MLLM evaluators from quality monitors into debugging tools for VGM developers—not just "this video is bad" but "at frame 47, the subject's left hand has six fingers, and the motion between frames 30-40 violates physical plausibility." This is a natural extension of the paper's unified evaluation vision: if an MLLM can be prompted to evaluate any aspect, it can also be prompted to explain and localize its evaluation, providing the granularity needed for model improvement.

Practical Applications and Downstream Use Cases

Batch quality filtering for VGM-generated content at scale. A video generation platform or content creation tool that produces thousands of candidate videos per day needs automated quality filtering to surface the best outputs to users. The paper's results support deploying a single large MLLM (e.g., InternVL2.5-78B or GPT-4o) to score all candidates across multiple quality aspects simultaneously, replacing an ensemble of specialized models. For a platform generating 10,000 videos daily, the approach would: (1) sample 16 frames per video, (2) prompt the MLLM with 15 aspect-specific evaluation templates (or a subset covering the platform's quality criteria), (3) compute Asingle scores for each aspect via token-probability extraction, and (4) filter or rank videos based on composite quality thresholds. The paper's finding that zero-shot MLLMs outperform specialized evaluators on most aspects (Table 4, AVG 75–80 for top models vs. specialized baselines at 40–70 on their subsets) means this single-model pipeline would be simultaneously more accurate and more comprehensive than maintaining separate specialized evaluators for aesthetic quality, technical quality, and alignment. The practical engineering question—whether inference cost is acceptable for production throughput—would depend on the specific MLLM and hardware, but the paper's single-to-pairwise adaptation results (Section 4.3.3) suggest that even 7B models with this strategy may be adequate for coarse filtering, with larger models reserved for borderline cases.

Automated regression testing during VGM development. When researchers iterate on a video generative model—modifying architecture, training data, or hyperparameters—they need to know whether the change improved or degraded quality across multiple dimensions. The current workflow typically involves qualitative human inspection of a handful of cherry-picked examples, which is slow, non-systematic, and susceptible to confirmation bias. A UVE-Bench-style evaluation pipeline, using a fixed set of prompts and comparing video pairs from the old and new model checkpoints via an MLLM evaluator, would provide quantitative, multi-aspect regression testing. The paper demonstrates that MLLMs can correctly rank video generative models by quality (Table 6: MLLM scores track human preference ordering from Open-Sora 1.2 to HunyuanVideo) and can distinguish real from generated videos. For a VGM developer training a new checkpoint daily, running a fixed 500-prompt evaluation set through both old and new models and scoring outputs with InternVL2.5-78B across all 15 UVE-Bench aspects would provide a daily dashboard of per-aspect quality changes, detecting regressions in specific dimensions (e.g., "temporal flickering increased by 3 points while aesthetic quality improved by 5") that qualitative inspection would miss.

Pre-filtering for large-scale human evaluation studies. Human evaluation remains the gold standard for AIGV quality assessment, but it is expensive and slow. The paper's finding that MLLM single-video ratings correlate with human pairwise preferences (as measured by Asingle in Table 4) supports using MLLMs as a pre-filter to select video pairs for human annotation. Instead of presenting human annotators with random video pairs (many of which will be obviously different or indistinguishable), an MLLM can pre-score all videos and select pairs where: (a) the MLLM is uncertain (scores for both videos are close to 0.5, indicating ambiguity), or (b) the MLLM's ranking disagrees with expected model quality ordering (e.g., the MLLM rates an older model higher than a newer one), or (c) the MLLM's per-aspect scores reveal an interesting quality tradeoff (high static quality, low temporal quality). This targeted selection would make human evaluation budgets more informative per annotation dollar. The paper's inter-annotator agreement of 0.803 Fleiss' Kappa (Section 3.4) establishes the human baseline reliability, and the single-to-pairwise adaptation (Section 4.3.3) provides the mechanism for converting MLLM scores to the pairwise format needed for pair selection. A practical study would compare the statistical efficiency of MLLM-filtered human evaluation against unfiltered random sampling: for a fixed annotation budget, does MLLM pre-filtering yield tighter confidence intervals on model quality rankings?

When to Prefer This Method

The paper does not position MLLM-based unified evaluation against a clearly specified set of alternatives with explicit decision boundaries—it compares zero-shot MLLM prompting against existing specialized evaluators (VideoScore, VBench, DOVER, etc.) as baselines, but does not articulate a decision rule for practitioners choosing between these approaches. The paper's framing is primarily descriptive ("here is how well MLLMs perform as unified evaluators") rather than prescriptive ("under condition X, use method A; under condition Y, use method B"). The findings imply a practical preference ordering—try zero-shot MLLM prompting first, invest in specialized evaluators only for aspects where MLLMs fail—but this is not formalized in the paper as a decision framework. Including a formulaic "Prefer A when..." matrix would impose structure the paper itself does not provide, so I omit this subsection.