MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation

ArXiv: 2605.20183

🎯 Pitch

Current multi-shot audio-video generators cannot reliably perform director-level control, and our evaluation reveals that the common 'video-first, post-hoc dubbing' pipeline fundamentally fails for complex narratives. MSAVBench introduces an adaptive hybrid evaluator that correlates 91.5% with human judgment, exposing that modular open-source systems can rival closed-source models but lack unified audio-video architectures essential for fine-grained synchronization.

---

1. Executive Summary

This paper introduces MSAVBench, the first comprehensive benchmark and adaptive hybrid evaluation framework for multi-shot audio-video (MSAV) generation, spanning four key dimensions—video, audio, shot, and reference—across 286 prompts and 2,198 shots with up to 15 shots per prompt. The evaluation framework improves robustness through an adaptive self-correction mechanism for shot segmentation (iteratively merging or splitting mis-segmented boundaries via VLM-driven tool invocation) and stratified scoring paradigms that replace direct VLM scoring with instance-wise rubrics (predefined multiple-choice questions for subjective dimensions like narrative coherence) and tool-grounded evidence extraction (adaptively invoking object detectors and pose estimators to ground complex judgments like layout-text consistency). The framework achieves a Spearman rank correlation of 91.5% with human judgments, and a systematic evaluation of 19 state-of-the-art closed- and open-source models reveals that modular or agentic generation pipelines substantially narrow the gap with closed-source systems, establishing that current models remain far from reliable "director-level" generation and that the common "video-first, post-hoc dubbing" paradigm is fundamentally inadequate for complex multi-shot audio-video generation where unified audio-video architectures are essential.

2. Context and Motivation

The Core Problem: Evaluating Multi-Shot Audio-Video Generation Is Fundamentally Broken

The landscape of generative video is undergoing a rapid transformation, but the tools for measuring progress have failed to keep pace. The paper identifies a critical gap: as the field moves from simple, single-shot text-to-video (T2V) synthesis toward complex multi-shot audio-video (MSAV) narratives with synchronized audio, cinematic language, and long-range coherence, there exists no benchmark capable of evaluating these capabilities in a comprehensive or reliable way.

This is not merely an inconvenience — it is a structural barrier to progress. Without a reliable evaluation framework, model developers cannot diagnose specific weaknesses, the open-source community cannot iterate toward parity with closed-source systems, and the field lacks a shared vocabulary for discussing where models succeed and fail. The paper frames this as an urgent prerequisite for guiding the design of next-generation MSAV systems, particularly for the open-source community, which currently has no dedicated MSAV models at all.

The evaluation challenge is fundamentally harder than single-shot video assessment for three reasons that compound on each other:

1. Compositional nature. An MSAV output is not a single entity but a structured composition of multiple shots, each with its own visual content, camera parameters, and audio events, connected by transitions and governed by an overarching narrative logic. Evaluating any one dimension in isolation misses the point: a model might produce individually beautiful shots that fail to cohere into a story, or generate perfectly synchronized audio within each shot while losing speaker identity across shot boundaries.

2. Multi-modal entanglement. Unlike traditional video benchmarks that assess visual quality alone, MSAV requires evaluating the joint alignment between visual content and audio across multiple granularities — phoneme-level lip synchronization within a shot, speaker timbre consistency across shots, temporal alignment of sound effects with visual events, and narrative-level coherence between what is seen and what is heard. A model that excels at visual generation may completely collapse when audio is introduced, or vice versa.

3. Structural complexity. The number of shots, the presence of multiple subjects, the use of professional cinematic language (shot scales, camera angles, transitions, lighting), and the inclusion of non-realistic or counterfactual content all introduce evaluation dimensions that single-shot benchmarks simply do not address. A model may handle 2-shot narratives flawlessly but degrade catastrophically at 10 shots — a failure mode invisible to existing benchmarks.

Why This Problem Matters: Real-World and Research Implications

The paper's motivation is grounded in both practical deployment needs and the requirements of the research community.

On the deployment side, the push toward MSAV is driven by real-world demands that single-shot synthesis cannot satisfy. Cinematic storytelling, educational tutorials, multi-scene advertising, music videos, and dialogue-driven narratives all require the ability to compose multiple coherent shots with synchronized audio. Closed-source commercial systems — Seedance 2.0, Wan 2.7, Sora 2, Kling V3 — have already begun shipping MSAV capabilities, but without a rigorous evaluation framework, users and developers cannot meaningfully compare them, understand their failure modes, or make informed decisions about which system to deploy for which use case.

On the research side, the absence of a comprehensive benchmark creates a vicious cycle. The open-source community lacks dedicated MSAV models not because the task is technically impossible, but because there is no standard way to measure progress or diagnose weaknesses. As the paper states in Section 1:

"the open-source community currently lacks dedicated MSAV models, leaving a critical gap in the field. Therefore, establishing a comprehensive MSAV benchmark is an urgent prerequisite to providing design guidelines for the open-source community and to diagnosing model weaknesses in closed-source systems."

This is not merely aspirational — the paper's own experimental results (Section 4.2) demonstrate that when open-source pipelines are properly constructed (e.g., the modular "image + audio-video" approach of LTX-2.3 in TI2AV mode), they can approach closed-source performance. A benchmark enables the systematic experimentation needed to close that gap.

Theoretical significance comes from the fact that MSAV evaluation touches on several fundamental open problems in generative AI: long-range coherence and planning (can a model maintain narrative and visual consistency across 15 shots?), compositional generalization (can it combine realistic subjects with non-realistic scenes without mode collapse?), cross-modal binding (can it reliably associate the correct sound with the correct visual source at the correct time?), and controllable generation (can it faithfully execute specific cinematic instructions?). These are not incremental challenges — they represent some of the deepest unsolved problems in generative modeling, and a rigorous benchmark provides the measurement infrastructure to study them systematically.

Prior Approaches and Where They Fall Short

The paper positions MSAVBench against a landscape of existing benchmarks that each address fragments of the MSAV evaluation problem but none that address the whole. Table 1 in the paper provides a structured comparison that is worth examining in detail, because it reveals exactly where the gaps lie.

Single-Shot, Silent Video Benchmarks (VBench, EvalCrafter, Video-Bench)

The first generation of video generation benchmarks — VBench (Huang et al., 2024) with ~1,600 prompts, EvalCrafter (Liu et al., 2023) with 700 prompts, and Video-Bench (Han et al., 2025) with 419 prompts — all share a fundamental limitation: they assume single-shot, silent video generation. Their evaluation dimensions focus on visual quality, temporal consistency, and motion naturalness within a single continuous clip. They have no concept of shot boundaries, no mechanisms for evaluating cross-shot narrative coherence, no audio evaluation whatsoever, and no support for cinematic language or reference conditioning.

These benchmarks were appropriate when the field was focused on basic text-to-video synthesis, but they are now answering the wrong question. A model could score perfectly on VBench while producing a multi-shot narrative where each shot looks beautiful but the subject's appearance changes completely between shots, the audio is completely absent, and the camera instructions are ignored — precisely the kinds of failures that matter in real MSAV applications.

Multi-Shot, Video-Only Benchmarks (ViStoryBench, MSVBench, UniVBench)

A second generation of benchmarks recognized the need for multi-shot evaluation but remained largely video-centric. ViStoryBench (Zhuang et al., 2025) with 80 prompts averaging 16.5 shots, MSVBench (Shi et al., 2026) with only 20 prompts averaging ~14 shots, and UniVBench (Wei et al., 2026) with 200 prompts averaging 3.72 shots all extended evaluation to story structure and cross-shot consistency. However, they share a critical omission: limited or absent audio evaluation.

These benchmarks treat video generation as a purely visual task. ViStoryBench and MSVBench include no audio evaluation at all — they assess whether shots form a coherent visual story, but ignore whether the audio matches the visuals, whether lip movements synchronize with speech, or whether speaker identity remains consistent across dialogue shots. UniVBench includes some agentic scoring capabilities (a point the paper acknowledges) but still lacks comprehensive audio-visual joint evaluation.

A second limitation of these benchmarks is their data diversity and complexity. MSVBench, for example, uses only 20 prompts — a sample size so small that it cannot support statistically meaningful comparisons across the diverse genres, styles, shot counts, and subject types that real MSAV systems must handle. The paper's own benchmark, by contrast, contains 286 prompts with 2,198 shots spanning 8 video genres, 6 visual styles, 6 audio content categories, 7 emotional states, and 6 languages.

Audio-Visual Benchmarks with Limited Shot Structure (AVGen-Bench, VABench, PhyAVBench)

On the other side of the gap, a third generation of benchmarks introduced audio evaluation but remained anchored to single-shot or weakly structured settings. AVGen-Bench (Zhou et al., 2026) with 235 prompts averaging only 1.6 shots evaluates audio quality and audio-visual alignment, but its prompts lack the rich multi-shot structure, cinematic language, and challenging scenarios (counterfactual compositions, dense text rendering, multi-speaker dialogue across shots) that characterize real MSAV content.

The key pattern is that no existing benchmark simultaneously provides: (i) multi-shot structure with explicit cinematic language, (ii) comprehensive audio-visual joint evaluation, and (iii) broad coverage of challenging and diverse generation settings. Each prior benchmark addresses at most two of these three requirements, leaving the third as a blind spot.

Rigid and Static Evaluation Pipelines

Beyond the scope of their data, the paper identifies a second failure mode in existing benchmarks: their evaluation pipelines are brittle and unreliable when applied to complex MSAV outputs. The paper articulates two specific weaknesses (Section 1):

First, they struggle with shot mis-segmentation. Evaluating a multi-shot video requires first segmenting it into shots — identifying where one shot ends and another begins. This is surprisingly difficult in practice. Generated videos often exhibit variable shot counts (a model may produce 7 shots when 10 were requested), ambiguous transition boundaries (a dissolve transition may lack a clean cut point), and temporal artifacts that confuse off-the-shelf shot boundary detectors. Existing pipelines typically rely on a fixed segmenter without any self-correction mechanism. A single mis-segmentation — splitting what should be one shot into two, or merging two distinct shots into one — cascades into errors in all downstream per-shot metrics: a "cross-shot consistency" score computed on incorrectly segmented shots is meaningless, and a "camera parameter adherence" check on a merged shot that contains two different camera angles will produce nonsense results.

The paper frames this as a robustness problem: the evaluation pipeline's accuracy should not be gated by a single brittle component. Yet in existing benchmarks, it is.

Second, they employ rigid scoring paradigms for complex dimensions. For important yet challenging evaluation dimensions that lack dedicated expert models — such as narrative coherence (does the video tell a logically connected story?) or layout-text consistency (does the spatial arrangement of subjects match the prompt's description?) — existing pipelines often rely on direct VLM scoring: feed a VLM the video frames and the prompt, and ask it to produce a scalar score. Although simple to implement, this approach is plagued by two problems:

• Prompt sensitivity: the exact phrasing of the scoring prompt can dramatically change the VLM's output, making scores unstable and difficult to reproduce across different implementations.
• Hallucination: VLMs are prone to confabulating justifications for their scores, producing assessments that sound plausible but are factually incorrect — for example, claiming that a subject is holding an object in their left hand when a detection tool would reveal the opposite.

The paper quantifies this problem in Table 6: direct VLM scoring achieves a Spearman correlation with human judgments of only 0.429 on cross-shot layout consistency and 0.405 on intra-shot layout-text alignment, compared to 0.767 and 0.786 respectively for the paper's tool-grounded approach. This is not a minor calibration issue — direct VLM scoring is less than half as correlated with human judgment as the paper's proposed method on these complex dimensions.

How MSAVBench Positions Itself

The paper does not propose a single new evaluation metric or a new model architecture. Instead, it positions itself as filling the infrastructure gap — providing the benchmark and evaluation framework that the field needs to make progress on MSAV generation. This is a deliberate positioning: the authors argue that the primary bottleneck is not algorithmic but evaluative. Without a way to measure progress reliably, algorithmic innovation is unfocused and model comparisons are meaningless.

The paper's contribution is therefore threefold in its own framing (Section 1):

1. A comprehensive benchmark that spans video, audio, shot, and reference dimensions with diverse tasks and challenging generation settings — the first to combine all of these into a single evaluation suite.
2. An adaptive hybrid evaluation framework that introduces self-correction for shot segmentation, instance-wise rubrics for subjective scoring, and tool-grounded evidence extraction for complex judgments — directly addressing the brittleness and hallucination problems of prior pipelines.
3. A systematic evaluation of 19 models that reveals actionable insights about where current systems fail and what architectural directions (modular pipelines, unified audio-video architectures) are most promising.

The paper explicitly positions itself in relation to prior benchmarks through Table 1, which shows MSAVBench as the only benchmark that checks all the boxes: multi-shot structure (average 7.7 shots), counterfactual prompts, video evaluation, audio evaluation, audio-video joint evaluation, cinematic language, reference conditioning, shot segmentation correction, and agentic scoring. Prior benchmarks fill subsets of this matrix — some cover multi-shot video without audio, others cover audio-video without cinematic language, none combines robust evaluation pipelines with broad data coverage.

This positioning is important because it clarifies what the paper is not claiming: it is not claiming to have solved MSAV generation, nor is it proposing a new state-of-the-art generation method. It is providing the measurement tool that makes solving MSAV generation possible. The distinction is analogous to the role that ImageNet played for image classification or GLUE/SuperGLUE played for natural language understanding: the benchmark itself does not advance the state of the art, but it enables the community to do so by providing a shared, reliable yardstick.

3. Technical Approach

This is primarily a benchmark and evaluation framework paper whose core idea is that evaluating multi-shot audio-video (MSAV) generation requires simultaneously addressing data diversity and evaluation robustness through a benchmark spanning four dimensions (video, audio, shot, reference) and an adaptive evaluation pipeline with stratified scoring paradigms that replace brittle direct-VLM scoring with rubric-grounded and tool-grounded assessment.

3.1 Reader Orientation

The paper builds a comprehensive evaluation system for multi-shot audio-video generation models. Given a generated video with audio from any MSAV model and the prompt that produced it, the system produces a reliable multi-dimensional score that captures how well the video satisfies visual quality, audio fidelity, cross-shot consistency, cinematic adherence, and reference-conditioned fidelity requirements. The core innovation is not a single metric but rather an adaptive pipeline architecture that self-corrects shot segmentation errors and deploys different scoring strategies (expert models, rubric-based VLM scoring, tool-grounded agentic scoring) depending on what each evaluation dimension demands — replacing the fragile one-size-fits-all VLM scoring that dominates prior benchmarks.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, executed in sequence:

1. Benchmark Dataset (MSAVBench) — 286 prompts with 2,198 total shots containing structured metadata (video genre, audio content, cinematic language, reference conditions) constructed through a four-stage pipeline combining LLM generation with human expert curation. This is the input specification: it defines what gets evaluated and under what conditions.

2. Agentic Pre-Processing and Self-Correction Module — takes the generated video as input, applies TransNet V2 for initial shot boundary detection, then iteratively invokes a VLM (Qwen3.5) to inspect boundaries, diagnose mis-segmentations, and call tools to merge or split segments. This produces corrected shot boundaries that feed all downstream per-shot metrics.

3. Stratified Scoring Engine — a collection of 20 metrics organized across four hierarchical levels (global, cross-shot, intra-shot, reference), each implemented via one of three scoring paradigms chosen based on metric complexity: specialized expert models for well-defined tasks (e.g., lip-sync via StableSyncNet), instance-wise rubric-based VLM scoring for subjective dimensions (e.g., narrative coherence via predefined multiple-choice questions), and tool-grounded agentic scoring for complex compositional properties (e.g., layout-text consistency via Grounding DINO + BlazePose evidence extraction).

4. Aggregation and Scoring Module — merges overlapping fine-grained metrics into 11 non-redundant dimensions, normalizes each to $[0, 1]$, averages them, and multiplies by a shot-completion penalty coefficient (ratio of generated shots to specified shot count) to produce the final overall score.

Information flows as follows: the benchmark dataset provides structured prompts → a generation model produces a multi-shot audio-video output → the pre-processing module segments and corrects shot boundaries → the scoring engine evaluates each metric at the appropriate level → the aggregation module produces the final rankable score.

3.3 Roadmap for the Deep Dive

• First, the benchmark data design and construction pipeline, since the evaluation framework's value depends entirely on the quality, diversity, and challenge coverage of the data it evaluates against. This covers the four-stage construction process, the expert taxonomy, the LLM prompt templates, and the human curation step.
• Second, the data analysis — the quantitative distributions across video genres, audio conditions, cinematic language, reference assets, and task complexity levels — since these establish the scope and difficulty of what the benchmark measures.
• Third, the evaluation suite architecture — the 20 hierarchical metrics and how they are organized into global, cross-shot, intra-shot, and reference levels, including the overall score aggregation formula.
• Fourth, the adaptive hybrid evaluation framework in detail — the shot self-correction mechanism (how TransNet V2 outputs are iteratively refined), the three stratified scoring paradigms (expert models, rubric-based, tool-grounded), and why each was chosen for which metric types.
• Fifth, the human alignment validation methodology — how Spearman rank correlation is computed against expert annotations, the pairwise comparison protocol, and the robustness check across different VLM backbones.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Benchmark Data Design: The Four-Dimensional Taxonomy

The benchmark is organized around four orthogonal data-design dimensions — video, audio, shot, and reference — each annotated with structured sub-attributes. The core design principle is that any MSAV generation model must satisfy constraints across all four dimensions simultaneously, and the benchmark is constructed to probe failures at their intersections.

Dimension 1 — Video. Four sub-attributes define the visual specification:

• Video genre (8 categories): Action, Narrative, Tutorial, Singing & Music Performance, Multi-person Dialogue, Science/Game, Advertising, Nature. These are not merely labels — they structurally constrain what the generated video must contain. A Tutorial prompt, for example, requires procedural step-by-step visual coherence, while a Multi-person Dialogue prompt requires alternating speaker shots with lip-sync across multiple identities.
• Visual style (6 styles): photo-realistic, anime, watercolour storybook, pixel art, cyberpunk, retro film. These are explicit generation constraints that test whether models can maintain a consistent aesthetic across shots without mode collapse into photographic realism.
• Subject type (4 classes): humans, animals, inanimate objects, fictional characters. Subjects and scenes are further independently classified into realistic and non-realistic domains, enabling the benchmark to test counterfactual composition — for instance, a realistic human in a non-realistic cyberpunk scene, or a non-realistic fictional character in a realistic office environment.
• Scene type: realistic and non-realistic, where the non-realistic class encompasses both coherent fictional worlds (cyberpunk city) and counterfactual compositions (a frozen tropical desert).

Dimension 2 — Audio. Three sub-attributes define the acoustic specification:

• Audio content class (6 categories): speech, singing, instrument/machine, human-made environment (e.g., laughter, footsteps), natural ambient, mixed (foley with voice-over, music with environment). This spans the full range of audio types that an MSAV system must handle, from clean single-source speech to complex overlapping audio streams.
• Audio emotion (7 emotions): joy, fear, anger, surprise, sadness, neutrality, awe. These are annotated per-shot and test whether the audio generation captures emotional nuance consistent with the visual narrative.
• Spoken language (6 values): Chinese, English, Japanese, Korean, Spanish, French. This enables explicit multilingual evaluation of speech synthesis and lip-sync alignment.

Dimension 3 — Shot (cinematic language). Five sub-attributes are annotated per shot, introducing professional cinematographic control vocabulary that goes far beyond prior benchmarks:

• Shot scale (5 types): close-up, mid-close, mid, mid-long, long. These constrain the camera's distance from the subject and the resulting composition.
• Shot angle (5 types): eye-level, top-down, low-angle, oblique, dutch. These test whether models can execute specific camera perspectives rather than defaulting to eye-level framing.
• Camera motion (4 types): push-pull, pan-tilt, tracking, hand-held/shake. These require dynamic camera behavior within and across shots.
• Transition (4 types): hard cut, dissolve, match cut, fade. These specify how consecutive shots connect, testing whether models understand editing grammar.
• Lighting (5 types): natural, side, soft, neon, low-key. These constrain the illumination conditions and test consistency across shots.

Dimension 4 — Reference. Three sub-attributes are paired with specific prompts:

• Subject reference image (68 images): spans both realistic and anime domains, providing visual identity conditions that the generated video must preserve.
• Paired reference audio (65 audio clips): paired with subject images, providing vocal timbre conditions for voice cloning evaluation.
• Scene reference image (32 images): covers indoor and outdoor environments, testing whether models can faithfully reproduce specified background settings.

All reference assets are assigned across 96 prompts, with strict global uniqueness constraints ensuring no asset is reused across different prompts — each reference condition tests identity preservation without memorization shortcuts.

Complexity structure. Beyond diversity, the benchmark is explicitly designed around two axes of complexity:

• Reality and Non-reality: Subjects and scenes are independently categorized as realistic or non-realistic, then cross-combined to create four difficulty buckets: realistic-subject × realistic-scene (49.3% of prompts), realistic-subject × non-realistic-scene (26.2%), non-realistic-subject × realistic-scene (16.8%), and non-realistic-subject × non-realistic-scene (7.7%). This design specifically probes out-of-distribution generalization — can a model generate a photorealistic human in a watercolour storybook world without either collapsing to the dominant training mode (photorealism) or producing incoherent hybrids?
• Challenging scenarios: The benchmark includes overlapping simultaneous audio sources, complex fast-paced motions, dense on-screen text rendering (for advertising-style prompts), multilingual speech, and most importantly, shot counts extending up to 15 with an average of 7.7 shots per prompt — structural complexity absent from prior benchmarks.

3.4.2 Data Construction Pipeline: Four-Stage Expert-in-the-Loop Process

The benchmark is constructed through a four-stage pipeline that combines automated LLM generation with rigorous human expert curation, as illustrated in Figure 5 of the paper.

Stage 1: Expert-driven taxonomy and quadruple construction.

Domain experts first define the 8-category video content taxonomy described in Section 3.4.1, which is further decomposed into fine-grained themes to prevent prompt homogenization. The taxonomy contains 144 fine-grained sub-categories in total. For example, the Action genre (32 sub-categories) includes martial-arts duel, parkour, street-dance battle, basketball, boxing, skateboarding, rock climbing, BASE jumping, card shuffling, and surgical suturing. The Narrative genre (19 sub-categories) spans detective reasoning, romance, sci-fi adventure, historical legend, comedy, horror/thriller, courtroom drama, and more.

Concurrently, experts curate extensive candidate pools for subjects, scenes, and visual styles, strictly categorizing each into realistic and non-realistic domains. This stage produces a vast combinatorial pool of (theme, subject, scene, style) seed quadruples — the atomic building blocks from which prompts will be synthesized.

The experts are described in Appendix D.1 as "six domain experts, all of whom are full-time researchers in AIGC and audio-video generation," each holding a graduate degree in computer vision, multimedia, or audio signal processing. This expertise is critical because the taxonomy design determines what the benchmark can evaluate — poorly chosen categories would produce prompts that fail to stress-test model capabilities.

Stage 2: Prompt generation and rewriting.

The paper randomly samples 2,200 seed quadruples from the combinatorial pool. These are fed to GPT-5.4 with a specialized system template that converts each quadruple, plus a target shot count, into a structured multi-shot script. The initial-prompt template (provided in Appendix A.2.2) is a detailed engineering artifact specifying:

• Global attributes that are constant across the whole video: visual style, subject, colour tone (warm/cold/neutral), audio content, and spoken language.
• Per-shot attributes that may vary but must maintain continuity: scene (transitions must be narratively coherent with no abrupt jumps), lighting (1–2 dominant variants), audio emotion (1–2 dominant tones), shot scale, angle, camera motion, and transition type (must vary for cinematic richness).
• Cross-call diversity: style, subject, and scene must be varied across calls to prevent homogenization.
• Continuity: shots must follow a clear causal/temporal order, each a natural continuation of the previous.
• Per-category guidance: the template includes specific instructions for each of the 8 genres (Action, Narrative, Tutorial, etc.) to ensure genre-appropriate content.

GPT-5.4 outputs a JSON object containing the multi-shot prompt text and structured evaluation metadata — including num_persons, num_animals, num_object_subjects, and per-shot dictionaries with all cinematic language attributes. This structured metadata is essential because it enables automated evaluation: the benchmark knows what each shot should contain, allowing metrics to check whether the generated video matches the specification.

The 2,200 initial prompts are then rewritten by a Prompt Enhancement (PE) model — also GPT-5.4 but with a different system template — into a comprehensive global-to-shot script format. The PE template (Appendix A.2.2) enforces a specific structure:

1. Global framing: opens with "This video contains X shots; it is a [category] in [style], [colour tone], accompanied by [audio content] in [language]." Lighting and emotional colour are anchored to 1–2 dominant variants.
2. Per-shot detail: each shot begins with "Shot N [ts–te s]" (default total duration 15 seconds) and describes, in order: transition from previous shot; shot scale and angle (diverse across shots); composition and focus; scene; lighting; subject state and props; subject motion/physical change; audio emotional tone, language, and the precise AV interaction; camera-motion trajectory.
3. Continuity: each shot must be a natural continuation of the previous one, respecting temporal and causal order, with no abrupt jumps.
4. Naming convention: any real personal names in the input are replaced with descriptive references ("a young woman").

The PE model also performs safety filtering via an is_safe boolean flag, rejecting prompts with sexual, graphically violent, or politically sensitive content.

Why this two-stage LLM generation? The initial prompt generator focuses on content specification (what happens), while the PE model focuses on format and cinematic richness (how it's described). Separating these concerns allows each model to be optimized for its specific role, and the structured JSON intermediary ensures that metadata survives the rewriting process without degradation. The default 15-second total duration per prompt is not arbitrary — it provides a standardized temporal budget that enables fair comparison across models with different output length characteristics.

Stage 3: Expert annotation and refinement.

The 2,200 PE-rewritten prompts are not the final benchmark — they are candidates that undergo rigorous human review. Six domain experts review each prompt for diversity, structural complexity, and logical coherence. The filtering criteria are:

• Redundancy: prompts that are too similar to others are removed to prevent over-representation of certain patterns.
• Homogeneous cases: prompts that lack sufficient variation in cinematic language, subject composition, or audio conditions are filtered.
• Unnatural cross-shot transitions: narrative sequences where the shot-to-shot flow feels forced or illogical are rejected.
• LLM hallucinations: semantic deviations from the initial scripts — where the PE model introduced incorrect content or contradicted the original specification — are caught and either corrected or discarded.

Experts also manually refine ambiguous descriptions, ensuring that every prompt is "unambiguous, evaluable, and appropriately difficult" (Appendix A.2). Each prompt is reviewed by at least two experts, with disagreements escalated to a third senior expert and resolved by majority vote.

The result of this curation is striking: from 2,200 PE-rewritten prompts, only 286 prompts (13.0%) are retained in the released benchmark, comprising 2,198 individual shots. This 87% rejection rate is not a sign of poor LLM output quality — it reflects the extremely high bar for benchmark inclusion. The paper explicitly notes that "this highlights the strictness of the curation process." Every prompt that survives must simultaneously satisfy diversity (spanning all 8 genres, all 6 visual styles, all 6 audio categories, all 6 languages, and multiple shot counts), structural complexity (varying subject counts, transition types, and cinematic language), and logical coherence (a human can read the script and visualize the resulting video as a coherent narrative).

Stage 4: Reference media collection.

For the 96 prompts that require reference-conditioned generation, the paper constructs a reference asset pool. The process:

1. Sample 1,000 character image-audio pairs (spanning both realistic and anime domains) and 200 background images from established public benchmarks — specifically TalkVid (Chen et al., 2025), Anim-400k (Cai et al., 2024), UniVBench (Wei et al., 2026), and Japanese Anime Scenes (Wang, 2023).
2. Use Gemini 3.1 Pro as a VLM to categorize these assets, mapping them to the semantic conditions of the 96 reference-conditioned scripts. For example, a script requiring "a young woman with brown hair in an office setting" would be matched to an image-audio pair depicting a similar subject and a scene image of an office.
3. Enforce strict global uniqueness constraints: each reference asset is assigned to exactly one script, preventing the benchmark from inadvertently testing memorization rather than identity preservation.
4. Human experts meticulously filter out low-quality samples (blurry images, noisy audio clips, assets with artifacts) or misaligned matches (where the VLM's semantic mapping was incorrect).

The final reference subset contains 68 subject images, 65 audio clips, and 32 scene images, assigned across 96 scripts. The audio clips specifically span five age buckets (0–19, 19–30, 31–45, 46–60, 60+), multiple ethnicities, and six languages, ensuring that voice fidelity evaluation is not biased toward any particular demographic or linguistic group.

3.4.3 Data Analysis: Quantitative Distributions

The paper provides detailed distributional statistics that characterize the benchmark's coverage. These are not merely descriptive — they establish the benchmark's claim to comprehensive evaluation by demonstrating balanced representation across all design dimensions.

Video genre distribution: Action 16.4%, Tutorial 16.4%, Narrative 15.7%, Singing & Music 16.1%, Multi-person Dialogue 15.7%, Science/Game 8.4%, Advertising 8.4%, Nature 2.8%. The first five genres are roughly equally weighted, while the latter three have smaller representation reflecting their more specialized nature.

Subject distribution: humans 60.8%, animals 14.7%, inanimate objects 8.0%, fictional characters 16.4%. The dominance of humans reflects real-world MSAV use cases (cinematic storytelling, dialogue, tutorials) but the benchmark includes sufficient non-human content to test generalization.

Scene distribution: realistic 66.1%, non-realistic 33.9%. The one-third non-realistic proportion is deliberately substantial — it ensures that models cannot achieve high scores by specializing in photographic realism alone.

Visual style distribution (prompt-level): photo-realistic 54.2%, anime 13.6%, watercolour 10.8%, pixel art 10.5%, cyberpunk 10.5%, retro film 0.3%. The non-photorealistic styles collectively account for 45.8% of prompts, testing whether models can maintain style consistency across shots rather than regressing to their training distribution mode.

Audio content distribution: speech 28.7%, human-made environmental sounds 20.3%, nature ambient 14.3%, instrument/machine 10.1%, singing 10.1%, human activity sounds 16.4%. Speech is the most common but far from dominant — the benchmark intentionally includes substantial non-speech audio to evaluate general audio-visual synchronization.

Audio emotion distribution: joy 42.5%, fear/suspense 18.8%, anger/tension 11.1%, neutral 11.1%, surprise 9.6%, sadness 5.5%, others 1.4%. The distribution is intentionally non-uniform, with positive/neutral emotions overrepresented to reflect typical content distributions, but the inclusion of negative emotions tests whether audio generation captures emotional nuance.

Language distribution (by prompts): Chinese 165 prompts, English 64, Japanese 15, Korean 15, Spanish 14, French 13. The Chinese-heavy distribution reflects the development context (the authors are primarily based in Chinese institutions) but the inclusion of five additional languages with non-trivial prompt counts enables meaningful multilingual evaluation.

Cinematic language distributions (per-shot): Shot scale: close-up 26.8%, long shot 24.8%, extreme close 22.2%, mid 19.1%, mid-close+mid-long 5.2%, others 1.9%. Camera angle: eye-level 59.2%, top-down 22.6%, low-angle 15.4%, side 1.0%, others 1.8%. Camera motion: push-pull 44.6%, pan-tilt 26.5%, tracking-and-orbit 5.8%, hand-held/shake 23.1%. Transition: hard cut 66.9%, fade in/out 18.7%, none 13.0%, overlay 0.6%, others 0.6%. Lighting: 5 major types (natural, side, soft, neon, low-key) with the distribution plotted in Figure 6 but exact percentages not specified in the text.

Shot count distribution: 2–3 shots 7%, 4–5 shots 19%, 6–7 shots 23%, 8–9 shots 23%, 10–11 shots 18%, 12–13 shots 7%, 14–15 shots 3%. The mean is 7.7 shots. The distribution is concentrated in the 4–11 shot range (83% of prompts) with tails extending to 15 shots — sufficient to test both moderate and extreme multi-shot capabilities.

Subject count: 32.2% of prompts require multi-subject compositions, with over 10% demanding 5 or more simultaneous subjects. This tests spatial composition — can the model correctly place and maintain multiple subjects in a scene without merging or omitting them?

3.4.4 Evaluation Suite: Hierarchical Metric Organization

The evaluation framework comprises 20 metrics organized into four hierarchical levels, each assessing qualitatively different aspects of MSAV quality.

Global-level metrics (5 metrics). These evaluate properties that span the entire video:

1. Narrative coherence: assesses logical plot progression based on discrete events across all shots. Evaluated by a rubric-based VLM judge (Qwen3.5) over uniformly sampled frames from the full video. The VLM answers predefined binary questions about event ordering, causal validity, and completeness. The final score is the proportion of positive answers.

2. Lip synchronization: evaluates lip-speech alignment across all dialogue-bearing shots. Implemented using active-speaker localization (LR-ASD, Liao et al., 2025), speaker diarization (SortFormer, Park et al., 2024), and StableSyncNet (Li et al., 2024). Matched speaking segments are scored and averaged across the video, with raw sync confidence used directly as the score.

3. Sound attribution: measures the temporal overlap between visually active speakers and their audio. Uses visual active-speaker detection (LR-ASD) and audio diarization (SortFormer). Speakers are matched across modalities and their temporal overlap is computed. The final score is the mean overlap ratio in $[0, 1]$.

4. Audio-visual synchronization: measures the temporal offset between visual onsets and sound events at the whole-video level using the Synchformer model (Iashin et al., 2024). Synchformer predicts the global audio-video offset $\Delta t$ (in seconds). The raw offset is mapped to $[0, 1]$ by:

   $\text{score} = \max(0, 1 - |\Delta t| / 2.0)$

   where $\Delta t$ is the predicted temporal shift between the audio and video streams in seconds, and 2.0 is a threshold constant (seconds) beyond which the offset is considered severe enough to receive a score of zero.

   What it computes: a normalized synchronization score that penalizes temporal misalignment linearly up to 2.0 seconds, beyond which alignment is considered effectively absent. A perfectly synchronized video ($\Delta t = 0$) receives 1.0; a 0.5-second offset receives 0.75; a 1.0-second offset receives 0.5; and any offset of 2.0 seconds or greater receives 0.0.

   Why this form: linear decay with a hard cutoff is chosen over alternatives like Gaussian weighting because it is interpretable and does not require estimating a variance parameter. The 2.0-second threshold is a design choice reflecting that beyond this point, audio and visual events are perceptually unrelated, and further discrimination is not meaningful for human-perceived synchronization quality.

5. Visual quality: evaluates fine-grained visual fidelity — specifically, whether prompt-specified visual attributes are correctly realized. Evaluated by a rubric VLM judge (Qwen3.5) using prompt-instantiated multiple-choice questions, where each prompt slot (subject type, scene type, visual style, colour tone, lighting condition) is converted into an MCQ. The final score is the average MCQ accuracy across all tested attributes.

Cross-shot-level metrics (8 metrics). These assess consistency of visual content, audio properties, and spatial layouts across consecutive shots:

6. Cross-shot layout consistency: evaluates spatial layout coherence across shot transitions — whether the main subject maintains consistent position, orientation, scale, and prompt-specified hand relations between adjacent shots. Evaluated by a tool-grounded agentic VLM judge (Qwen3.5) that invokes Grounding DINO (Liu et al., 2024) for object detection and BlazePose (Bazarevsky et al., 2020) for pose estimation. The agent computes adjacent-shot consistency checks (e.g., "is the subject in the same relative position in shot 2 as shot 1?") and the final score is the average pass rate across all adjacent shot pairs.

7. Subject consistency: measures identity and appearance consistency of the main subject across shots. A VLM localizer (Qwen3.5) extracts subject crops, which are encoded using DINOv2 (Oquab et al., 2023) for general appearance and ArcFace (Deng et al., 2019) for facial identity. Pairwise cosine similarities are computed across all shot pairs, and the final score is:

   $\text{score} = \frac{1}{|\text{pairs}|} \sum_{(i,j)} \max(0, \cos(\mathbf{e}_i, \mathbf{e}_j))$

   where $\mathbf{e}_i$ and $\mathbf{e}_j$ are the combined DINOv2+ArcFace embedding vectors for shots $i$ and $j$, and $\cos(\cdot, \cdot)$ is the cosine similarity.

   What it computes: the mean clipped cosine similarity across all pairs of shots, where clipping at zero discards negative similarities (which would indicate anti-correlation and are not meaningful for identity comparison). High similarity means the subject looks like the same entity across shots; low similarity indicates identity drift.

   Why this form: cosine similarity is scale-invariant (unlike Euclidean distance), which matters because the same subject may appear at different sizes across shots. Clipping at zero follows standard practice in embedding-based identity verification and prevents negative values from pulling down the average.

8. Background consistency: measures background stability across shots after removing the foreground subject via a segmentation-based erasure step. Background-only image patches are encoded using CLIP (Radford et al., 2021) image embeddings, and pairwise cosine similarities are computed across shots. The final score is the mean clipped cosine similarity across all shot pairs.

9. Style consistency: measures whether the visual style (photo-realistic, anime, watercolour, etc.) remains consistent across shots. Uses CSD-ViT-L (Somepalli et al., 2024) style embeddings, which are specifically designed to capture artistic style independent of content. Pairwise cosine similarities are computed and averaged across shots.

10. Illumination consistency: measures stability of lighting, shadow, and brightness across shots. Evaluated by a rubric-based VLM judge (Qwen3.5) comparing adjacent shot pairs. The VLM answers binary questions about lighting consistency for each pair, and the final score is the average pass rate.

11. Colour consistency: measures consistency of tone, saturation, and contrast across shots. Evaluated by a rubric-based VLM judge (Qwen3.5) comparing adjacent shot pairs, analogous to illumination consistency but focused on colour properties rather than lighting geometry.

12. Music consistency: measures continuity of background music across shots for non-speech audio. Uses Demucs (Défossez et al., 2019) for source separation to isolate music, MuQ for music embedding extraction, and MIR-AIDJ All-in-one (Kim and Nam, 2023) for beat/BPM analysis. The score combines three components — music embedding similarity (cosine), BPM agreement (binary match/mismatch), and beat alignment (temporal offset) — into a weighted sum in $[0, 1]$. The exact weights are not specified in the main text but are described as a "weighted sum" in Appendix B.1.

13. Speaker timbre consistency: verifies that the distinct vocal identities of multiple speakers remain stable across different dialogue-bearing shots. Uses Silero VAD (Silero Team, 2024) for voice activity detection, Demucs for vocal isolation, and w2v-BERT-2.0 (Chung et al., 2021) for speaker embedding extraction. Per-shot speaker embeddings are extracted and pairwise cosine similarities are computed across shots. The final score is the mean clipped cosine similarity.

Intra-shot-level metrics (5 metrics). These evaluate quality and prompt adherence within individual shots:

14. Intra-shot layout-text alignment: assesses how accurately spatial layouts within a shot match the shot caption's description — for example, "gripping a phone with the left hand" requires verifying that the subject's left hand indeed holds a phone. Evaluated by a tool-grounded agentic VLM judge (Qwen3.5) that invokes Grounding DINO and BlazePose. The agent answers predefined sub-questions per shot (e.g., "is the phone in the left hand?"), and the final score is the average pass rate across all applicable sub-questions.
15. Camera parameter adherence: evaluates compliance with specified camera scale, angle, and movement. Evaluated by a rubric VLM judge (Qwen3.5) over sampled shot frames. Each specified camera attribute is checked independently — for example, if the shot specification says "close-up, low-angle, push-in," the VLM judges three binary questions and the pass rate contributes to the final score.
16. Audio quality: evaluates the acoustic and production quality of generated audio using Audiobox-Aesthetic (Tjandra et al., 2025), specifically its production-quality sub-score. The raw PQ score (on a 1–10 scale) is mapped to $[0, 1]$ by:

$\text{score} = (\text{PQ} - 1) / 9$

where $\text{PQ} \in [1, 10]$ is the production quality score from Audiobox-Aesthetic.

What it computes: a linear normalization of the 1–10 quality scale to the $[0, 1]$ unit interval. A score of 1 (lowest quality) maps to 0.0; a score of 10 (highest quality) maps to 1.0; a score of 5.5 (midpoint) maps to 0.5.

Why this form: linear normalization preserves the equidistance property of the original scale — the difference between quality scores of 3 and 4 is the same perceptual magnitude as between 8 and 9 — which a non-linear mapping (e.g., sigmoid) would distort.

17. Text rendering accuracy: measures character-level fidelity of on-screen rendered text, evaluated only on advertising-style prompts where text rendering is specified. Uses PP-OCRv5 (Cui et al., 2025; Cui et al., 2026) for optical character recognition. The recognized text is compared against the target text using character error rate (CER), and the final score is:

$\text{score} = \max(0, 1 - \text{CER})$

where $\text{CER} \in [0, \infty)$ is the character error rate (Levenshtein distance between recognized and target text, divided by target text length).

What it computes: the complement of the character error rate, clipped to $[0, 1]$. A perfect recognition (CER = 0) yields 1.0; a completely wrong recognition (CER ≥ 1) yields 0.0; a recognition with 30% character errors yields 0.7.

Why this form: CER is the standard metric for OCR evaluation because it is length-normalized and interpretable. The complement (1 - CER) converts it to an accuracy-like score where higher is better, consistent with all other metrics. Clipping at zero handles degenerate cases where CER exceeds 1 (e.g., when recognized text is much longer than target text).

18. Word error rate (WER/ASR transcription): measures speech transcription accuracy against the prompt-specified dialogue script. Uses FireRedASR2-LLM (Xu et al., 2026) or Whisper-large-v3 (Radford et al., 2023), depending on language (FireRedASR2 for Chinese, Whisper for other languages). The transcription is compared against the target script using word error rate, and the final score is:

$\text{score} = 1 - \min(\text{WER}, 1)$

where $\text{WER} \in [0, \infty)$ is the word error rate (substitutions + insertions + deletions, divided by reference word count).

What it computes: the complement of the capped word error rate. A perfect transcription (WER = 0) yields 1.0; a transcription that is entirely wrong or worse (WER ≥ 1) yields 0.0.

Why this form: min(WER, 1) caps the penalty at 1.0 because WERs above 1.0 provide no additional information for scoring purposes — a WER of 1.5 is not meaningfully "worse" than 1.0 in terms of usability. The complement converts to the standard higher-is-better convention.

Reference-level metrics (2 metrics). These assess fidelity to user-provided reference assets:

19. Subject fidelity: consistency with the reference image in appearance and identity. Uses the same embedding pipeline as cross-shot subject consistency (VLM localizer + DINOv2 + ArcFace), but compares generated subject embeddings against the reference image embedding rather than across generated shots. The final score is the mean clipped cosine similarity across all generated shots.
20. Voice fidelity: consistency with the reference audio in vocal timbre. Uses the same speaker embedding pipeline as cross-shot voice consistency (VAD + Demucs + w2v-BERT-2.0), but compares generated speech embeddings against the reference voice embedding. The final score is the mean clipped cosine similarity.

Overall score aggregation. The paper recognizes a subtle problem: several of the 20 atomic metrics reflect fine-grained sub-dimensions of the same underlying capability, and treating them as independent dimensions would overweight that capability in the final average. Specifically, five visual consistency metrics (subject, background, style, illumination, colour) all measure aspects of cross-shot visual coherence, and four dialogue-related audio metrics (voice timbre consistency, lip-sync, sound attribution, ASR transcription) all measure aspects of multi-speaker dialogue quality.

The aggregation procedure therefore first merges overlapping metrics into 11 non-redundant dimensions:

• Subject, background, style, illumination, and colour consistency are combined into a single Visual Quality dimension (replacing the global Visual Quality metric, which measures attribute adherence rather than cross-shot consistency — the paper's naming is slightly ambiguous here but the intent is clear from the metric grouping).
• Voice timbre consistency, lip-sync, sound attribution, and ASR transcription (WER) are combined into a single Multi-Speaker Dialogue Audio dimension.

The remaining metrics (narrative coherence, audio-visual synchronization, cross-shot layout consistency, music consistency, intra-shot layout-text alignment, camera parameter adherence, audio quality, text rendering accuracy, subject fidelity, voice fidelity) are kept as separate dimensions, yielding 11 total dimensions.

All 11 dimensions are normalized to $[0, 1]$ using the metric-specific mappings described above, and then averaged:

$\text{average\_score} = \frac{1}{11} \sum_{d=1}^{11} \text{score}_d$

Finally, to account for structural failures in multi-shot generation (where a model may generate fewer shots than specified, leaving part of the prompt unfulfilled), the average is multiplied by a shot-completion penalty coefficient:

$\text{overall\_score} = \text{average\_score} \times \frac{N_{\text{generated}}}{N_{\text{specified}}}$

where $N_{\text{generated}}$ is the number of valid shots actually produced by the model (after the self-correction pipeline resolves ambiguities), and $N_{\text{specified}}$ is the shot count specified in the prompt.

What it computes: the arithmetic mean of the 11 normalized dimension scores, scaled by the fraction of requested shots that were successfully generated. A model that produces perfect quality on all dimensions but only generates 5 of 10 specified shots receives at most 0.5; a model that generates all 10 shots with perfect quality receives 1.0.

Why this form: the shot-completion penalty addresses a common failure mode in MSAV generation: models that truncate long prompts, generating only the first few shots and ignoring later ones. Without this penalty, such models could achieve high per-shot quality scores while fundamentally failing at the multi-shot task. The penalty coefficient is multiplicative rather than additive to ensure that shot incompleteness scales proportionally with quality — a model that both truncates the prompt AND produces poor quality on the shots it does generate is penalized on both axes. The choice of a linear penalty (rather than, say, quadratic) reflects a value judgment that shot completeness is equally important as average per-dimension quality, not more or less so. The 11-dimension averaging with equal weights (rather than learned or domain-specific weights) represents a deliberate choice to avoid overfitting the aggregation to particular use cases — users who prioritize certain dimensions can re-weight the 11 scores accordingly.

3.4.5 Adaptive Hybrid Evaluation Framework

This is the paper's core technical contribution to evaluation methodology. The framework consists of two sequential phases: agentic pre-processing for shot self-correction, followed by stratified scoring using three different paradigms chosen per-metric.

Phase 1: Agentic pre-processing and self-correction.

The problem this phase addresses is fundamental: all per-shot and cross-shot metrics depend on correct shot segmentation. If the shot boundaries are wrong, every downstream metric becomes unreliable. Traditional approaches use a fixed shot boundary detector (typically TransNet V2) and accept whatever boundaries it produces, but this is fragile because:

• Generated videos often have variable shot counts — a model may produce 7 shots when 10 were requested, or 12 when 5 were requested (due to the model inserting its own cuts or merging specified shots).
• Transition boundaries can be ambiguous — a dissolve or fade transition lacks a clean cut point, making single-frame boundary detection inherently uncertain.
• Temporal artifacts (compression, generation glitches) can produce false boundary detections.

The paper's solution is an iterative VLM-driven correction loop, as illustrated in Figure 3:

Step 1: Initial boundary detection. The generated video is first processed by TransNet V2 (Souček and Lokoč, 2020), a deep network architecture for fast shot transition detection. TransNet V2 outputs a list of frame indices corresponding to detected shot boundaries. This provides $N_a$ actual shots (where $N_a$ may differ from $N_e$, the expected shot count specified in the prompt).

The paper explicitly notes that "direct boundary prediction by VLMs is unreliable" (Section 3.4.2), which is why TransNet V2 — a specialized, well-validated detector — is used for the initial segmentation rather than asking a VLM to identify boundaries from scratch. The VLM's role is correction, not primary detection.

Step 2: Iterative VLM inspection and tool-based correction. A VLM (Qwen3.5) is presented with the video frames and the initial shot boundaries. Its task is not to produce a scalar score but to diagnose segmentation errors and invoke corrective tools. Specifically, the VLM:

• Inspects each segment and determines whether it should be merged with an adjacent segment (splitting what is actually a single shot into two) or split into multiple segments (merging two distinct shots into one).
• Invokes splitting or merging tools that implement the VLM's recommended boundary adjustments.
• Re-evaluates the corrected segmentation, potentially identifying further errors that require additional correction.

Step 3: Iteration limit and fallback. To balance accuracy with computational cost, this correction loop is limited to a maximum of two iterations. Two iterations represents an engineering tradeoff: each iteration requires multiple VLM calls (inspecting all shot boundaries), so the cost scales linearly with iteration count, but the paper's empirical observation is that most segmentation errors are resolved within one or two corrections.

After two iterations, if the shot count $N_a$ still does not match the specified count $N_e$, the VLM performs a final shot-caption re-alignment: it discards non-aligned segments (segments that cannot be matched to any prompt-specified shot) to ensure the integrity of downstream metric computations. This means that if a model generates 12 shots for a 10-shot prompt, the two "extra" shots are discarded; if it generates 7 shots, the downstream metrics are computed only on the 7 shots that can be aligned to the prompt.

Why this design? The iterative tool-invocation approach is superior to alternatives for several reasons:

• Compared to single-pass VLM scoring of boundaries, it is more reliable because the VLM makes discrete decisions (merge/split) rather than producing continuous boundary locations, which VLMs struggle with due to their limited temporal precision.
• Compared to fully deterministic boundary detection (TransNet V2 alone), it is more robust because the VLM can use semantic understanding — recognizing that two segments with the same subject, scene, and camera angle are likely one shot, even if a transition effect confused the detector.
• Compared to human annotation, it is scalable and reproducible, though the paper acknowledges that human-annotated boundaries would be more accurate as ground truth.
• The two-iteration limit is not theoretically motivated — it is described as an efficiency constraint — and the paper does not provide an ablation on whether three or more iterations would further improve accuracy. This is a potential limitation for domains where segmentation errors are more complex.

Phase 2: Stratified scoring paradigms.

The paper argues that no single scoring approach works well for all 20 metrics. Direct VLM scoring — asking a VLM to watch the video and produce a scalar quality score — is simple but unreliable for complex or objective dimensions. The paper instead uses three paradigms, chosen based on the nature of each metric:

Paradigm 1: Specialized expert models (10 metrics).

For well-defined metrics where dedicated, validated models exist, the paper uses them directly without VLM involvement. The ten metrics using this paradigm are: audio-visual synchronization (Synchformer), lip-speech synchronization (LR-ASD + SortFormer + StableSyncNet), sound attribution (LR-ASD + SortFormer), style consistency (CSD-ViT-L), music consistency (Demucs + MuQ + All-in-one), voice timbre consistency (VAD + Demucs + w2v-BERT-2.0), audio quality (Audiobox-Aesthetic), text rendering accuracy (PP-OCRv5), ASR transcription/WER (FireRedASR2/Whisper), and voice fidelity (same pipeline as voice timbre consistency but comparing against reference audio).

Why this paradigm: these metrics have objective ground truth or well-calibrated prediction targets (was the speaker correctly identified? is the text accurately rendered? is the audio-video offset within tolerance?). Expert models trained specifically for these tasks are more reliable, more efficient, and more reproducible than VLM-based approaches for the same tasks. The choice to use specialized models rather than having the VLM do everything is a deliberate rejection of the "one model to rule them all" philosophy that some recent benchmarks adopt.

Paradigm 2: Instance-wise rubric-based scoring (5 metrics).

For subjective dimensions where no dedicated expert model exists but where evaluation requires nuanced understanding (not just objective detection), the paper replaces direct VLM scoring with a more structured approach: instance-wise rubrics.

The five metrics using this paradigm are: narrative coherence, visual quality, illumination consistency, colour consistency, and camera parameter adherence.

The mechanism works as follows (Section 3.4.2):

• For each prompt and each metric, the system pre-instantiates rubrics — specific, predefined multiple-choice questions derived from the prompt's metadata. For example, for a prompt specifying "close-up, low-angle" camera parameters on shot 3, the rubric for camera parameter adherence on shot 3 becomes: "What shot scale is used? (A) close-up (B) mid (C) long" and "What camera angle is used? (A) eye-level (B) low-angle (C) top-down."
• The VLM (Qwen3.5 for visual metrics, Gemini 3.1 Pro for audio-related metrics) is presented with the video frames and the MCQs, and selects one answer per question.
• The final metric score is the proportion of correct answers (average MCQ accuracy) across all applicable sub-questions.

Why this form? The paper quantifies the improvement in Table 6. For narrative coherence, direct VLM scoring achieves a Spearman correlation with human judgments of only 0.600, while instance-wise rubric scoring with Qwen3.5 achieves 0.850 — an improvement of 0.250. The key advantage is that MCQs:

• Reduce prompt sensitivity: the VLM is choosing from predefined options rather than producing an unconstrained scalar, making the task more like classification than open-ended evaluation.
• Reduce hallucination: the VLM cannot confabulate a justification that sounds plausible but is wrong — it must commit to a specific answer that can be checked against the ground-truth specification.
• Enable interpretability: if the VLM answers incorrectly on a specific MCQ ("the shot scale was mid, not close-up"), the failure is localized and diagnosable, unlike an opaque scalar score.
• Enable per-dimension breakdown: the MCQ structure naturally decomposes evaluation into atomic checks, allowing fine-grained analysis of which specific attributes a model succeeds or fails on.

The paper demonstrates robustness by substituting Qwen3.5 with the smaller Qwen2.5-VL-32B-Instruct backbone: rubric-based narrative coherence drops only slightly from 0.850 to 0.820, while direct VLM scoring with the same backbone change would likely degrade more severely (the paper does not report this ablation for direct scoring with Qwen2.5-VL, but the pattern of smaller models having larger gaps to human judgment is well-established).

Paradigm 3: Tool-grounded agentic scoring (5 metrics).

For complex compositional properties where pure VLM reasoning is often insufficient — properties that require precise spatial localization, object identification, or pose estimation — the paper introduces a third paradigm: tool-grounded agentic scoring.

The five metrics using this paradigm are: cross-shot layout consistency, subject consistency, background consistency, intra-shot layout-text alignment, and subject fidelity.

The mechanism (Section 3.4.2, elaborated in Appendix B.2) extends rubric-based scoring by allowing the VLM to adaptively invoke external perception tools to gather objective evidence before making its judgment. Specifically:

• The VLM (Qwen3.5) receives the video frames and the rubric questions (similar to Paradigm 2).
• However, instead of answering directly from visual inspection alone, the VLM can invoke tools: Grounding DINO (Liu et al., 2024) for open-set object detection and BlazePose (Bazarevsky et al., 2020) for body pose estimation.
• The tools return structured evidence — bounding boxes of detected objects, keypoint coordinates of detected body parts — which the VLM uses to ground its answers.

For example, for the intra-shot layout-text alignment sub-question "is the subject gripping a phone with the left hand?", the agentic flow is:

1. VLM invokes Grounding DINO to detect "phone" and "left hand" in the frame.
2. Grounding DINO returns bounding boxes (or empty results if no detection).
3. VLM invokes BlazePose to detect hand keypoints and determine which hand is "left."
4. VLM checks whether a "phone" bounding box overlaps with the "left hand" keypoint region.
5. VLM answers the MCQ based on this evidence: "Yes" if all conditions are satisfied, "No" otherwise.

Why this paradigm? The paper's quantitative justification in Table 6 is compelling. For cross-shot layout consistency:

• Direct VLM scoring achieves Spearman correlation of only 0.429 with human judgments.
• Tool-grounded agentic scoring with Qwen3.5 achieves 0.767 — an improvement of 0.338.
• With the smaller Qwen2.5-VL backbone, tool-grounded scoring still achieves 0.732.

For intra-shot layout-text alignment:

• Direct VLM scoring: 0.405.
• Tool-grounded with Qwen3.5: 0.786 — an improvement of 0.381.
• With Qwen2.5-VL: 0.741.

These are not marginal improvements — tool-grounded scoring nearly doubles the correlation with human judgment compared to direct VLM scoring on these complex dimensions. The reason is fundamental: VLMs are not reliable at fine-grained spatial reasoning from raw pixels alone. They hallucinate object locations, confuse left and right, and are insensitive to precise spatial relationships. By offloading spatial perception to specialized tools (detectors and pose estimators), the VLM's role becomes reasoning over structured evidence — a task much better aligned with its capabilities than pixel-level perception.

The paper implements all perception tools as independent FastAPI micro-services deployed on 8×A100 hosts (Appendix C.1), with tool outputs cached at the case level and reused across metrics whenever possible. This distributed architecture enables efficient evaluation — tools are called once per video and their outputs are shared across multiple metrics that need the same information (e.g., subject localization outputs are used by subject consistency, cross-shot layout consistency, and subject fidelity).

3.4.6 Human Alignment Validation Methodology

To validate that MSAVBench's automated metrics reflect human judgments, the paper conducts a human annotation study with two groups of expert annotators (Appendix D.2).

System-level evaluation (30 experts): 30 annotators compare 16 video-generation models in terms of overall quality using pairwise comparison. Each annotator labels 40 video pairs, yielding 1,200 pairwise judgments. For each pair, annotators select one of three outcomes: "A wins," "B wins," or "both good / both bad" (ties counted as 0.5 for each method when computing win rates). The resulting win-rate rankings are compared with the benchmark's overall score rankings using Spearman's rank correlation coefficient $\rho_s$.

Fine-grained evaluation (10 experts): 10 annotators evaluate three specific metrics — narrative coherence, cross-shot layout consistency, and intra-shot layout-text alignment — using the same pairwise protocol. For each metric, annotators compare 10 candidate methods and label 36 pairs each, yielding 360 judgments per metric.

Bias reduction measures: all videos are anonymized and presented in random order. Annotators follow a unified rubric for each evaluation metric. Video pairs are "uniformly sampled across genres, including realistic and stylized content, single- and multi-subject scenes, and videos with varying numbers of shots" (Appendix D.2).

Results (Table 6): The overall score achieves a Spearman correlation of 0.915 with human system-level rankings, indicating strong alignment — the benchmark's automated ranking agrees closely with how human experts would rank the same models.

Robustness check across VLM backbones: To test whether the evaluation framework's alignment with humans depends on a specific large VLM, the paper substitutes Qwen3.5 with the smaller Qwen2.5-VL-32B-Instruct for the three fine-grained metrics. The rubric- and tool-grounded designs remain highly stable:

• Narrative coherence: 0.850 (Qwen3.5) → 0.820 (Qwen2.5-VL) — drop of 0.030.
• Cross-shot layout consistency: 0.767 → 0.732 — drop of 0.035.
• Intra-shot layout-text alignment: 0.786 → 0.741 — drop of 0.045.

All three still vastly outperform direct VLM scoring (which achieved 0.600, 0.429, and 0.405 respectively with Qwen3.5), confirming that the evaluation design (rubrics and tool-grounding), not the model scale, is the primary driver of alignment quality.

The annotation interface (Appendix D.3, Figure 7) is a custom web tool that presents two candidate videos side-by-side with the corresponding prompt and metadata, ensuring annotators have full context for their judgments. The use of expert annotators (full-time AIGC researchers and aesthetic-quality annotators) rather than crowdworkers is important: evaluating multi-shot audio-video quality requires domain expertise to recognize subtle failures in cinematic language, audio-visual synchronization, and cross-shot consistency that lay annotators might miss.

4. Key Insights and Innovations

Innovation 1: Reconceptualizing MSAV Evaluation as an Infrastructure Problem, Not a Metric Problem

The paper's most fundamental intellectual move is reframing the bottleneck in multi-shot audio-video generation from algorithmic to evaluative. Prior to MSAVBench, the dominant assumption across the field was that progress in MSAV generation required better models — better architectures, better training data, better loss functions. Benchmarks existed (VBench, EvalCrafter, ViStoryBench, MSVBench, AVGen-Bench) but were treated as after-the-fact scorecards rather than as diagnostic infrastructure that shapes research direction.

This paper argues the opposite: the primary obstacle is that the field lacks a shared, reliable yardstick for measuring what models can and cannot do, and without such a yardstick, neither closed-source systems nor the open-source community can iterate effectively. The evidence for this framing is in the systematic gaps exposed by Table 1: every prior benchmark addresses at most a subset of video, audio, multi-shot structure, cinematic language, and robust evaluation pipelines, leaving blind spots where critical failure modes are invisible. A model optimized against VBench (single-shot, silent) would learn nothing about cross-shot speaker consistency; a model optimized against MSVBench (multi-shot, video-only, 20 prompts) would overfit to a tiny evaluation set while ignoring audio entirely. The paper positions MSAVBench not as "yet another benchmark with more metrics" but as the first benchmark that makes the full MSAV problem evaluable at all.

This reframing matters because it changes the research incentive structure. When evaluation is partial, the field optimizes for what it can measure, producing models that excel at visible dimensions while failing silently on unmeasured ones. The paper's experimental results (Table 2) bear this out: closed-source models like Seedance-2.0 achieve high visual quality scores (0.795) but struggle on sound attribution (0.578) and lip synchronization (1.52 — note that lower is better for sync offset, not higher, so this is a relatively weak score), failures that would be invisible on a video-only benchmark. By making these failures legible and rankable, MSAVBench transforms "audio-visual synchronization is hard" from an anecdotal observation into a quantified diagnostic that researchers can target.

This is a fundamental reframing rather than an incremental improvement: it changes what the field considers to be the research objective (from "build a better video generator" to "build a better MSAV system, evaluated holistically") and provides the measurement infrastructure to pursue that objective systematically. The paper explicitly names this as a prerequisite: "establishing a comprehensive MSAV benchmark is an urgent prerequisite to providing design guidelines for the open-source community" (Section 1).

---

Innovation 2: Difficulty-Conditioned Evaluation via Cross-Combined Reality Axes

A subtle but intellectually distinctive contribution is the paper's design of counterfactual composition difficulty as a structured evaluation dimension. Rather than treating "hard prompts" as an amorphous category, MSAVBench decomposes generation difficulty into the cross-product of two orthogonal axes: subject reality (realistic vs. non-realistic) and scene reality (realistic vs. non-realistic), yielding four difficulty tiers (real×real, real×unreal, unreal×real, unreal×unreal). This is not merely a labeling exercise — it is a diagnostic framework for probing out-of-distribution generalization that prior benchmarks ignored entirely.

The intellectual lineage here connects to work on compositional generalization in vision-language models, but the paper's innovation is applying it specifically to video generation evaluation. The question being asked is: can a model that has seen photorealistic humans and photorealistic offices separately combine them with a non-realistic element (say, a cyberpunk scene) without collapsing to the dominant training mode? The answer, from Table 4, is no: even the best closed-source model (Seedance-2.0) drops from 76.80 overall score on realistic prompts to 74.50 on non-realistic prompts, and open-source models show steeper declines (JavisDiT++ drops from 61.00 to 56.40). This quantitatively confirms what might otherwise be a vague intuition — that current models are brittle to distribution shift — and provides a precise, reproducible benchmark for measuring progress on this specific capability.

What makes this an innovation rather than an obvious categorization is that the reality axes are built into the benchmark's construction pipeline, not retroactively tagged. The seed taxonomy in Stage 1 of data construction categorizes subjects and scenes as realistic or non-realistic before prompts are generated, and the 33.9% non-realistic scene proportion and four-tier cross-combination are deliberate design choices, not emergent properties of the data. This means the benchmark is constructed to stress-test compositional generalization by design, rather than hoping that diversity in the prompt distribution happens to exercise this capability.

This is an incremental conceptual advance in the taxonomy of generation difficulty — it names and operationalizes a specific failure mode (mode collapse under counterfactual composition) that was previously implicit — but has fundamental practical significance because it provides a language and a measurement protocol for a problem that affects real deployment scenarios (generating fantasy worlds, stylized content, mixed-reality scenes).

---

Innovation 3: Stratified Scoring as a Rejection of VLM Monoculture in Evaluation

The paper makes a conceptually important architectural argument: not all evaluation dimensions should be scored by the same method, and specifically, VLM-based scoring should be reserved for tasks where it is genuinely appropriate rather than used as a universal default. This is a direct challenge to a growing trend in generative model evaluation — exemplified by benchmarks like Video-Bench, UniVBench, and aspects of VBench — where a single VLM judge is asked to produce scalar quality scores across all evaluation dimensions, under the implicit assumption that large multimodal models are sufficiently capable to serve as general-purpose evaluators.

MSAVBench's stratified scoring paradigm (Section 3.4.2) is not merely an engineering choice — it is a principled decomposition of evaluation into three categories based on what kind of judgment is required: (1) objective tasks with ground-truth targets (audio-visual offset, text rendering accuracy, speaker identity) where specialized expert models are more reliable; (2) subjective tasks requiring contextual understanding (narrative coherence, visual quality) where VLM judgment is appropriate but must be constrained via structured rubrics to prevent hallucination; and (3) spatially precise tasks (layout-text alignment, cross-shot layout consistency) where VLM reasoning is necessary but must be grounded in tool-extracted evidence because VLMs are unreliable at fine-grained spatial perception.

The quantitative evidence for this decomposition is in Table 6. Direct VLM scoring — the "monoculture" approach — achieves Spearman correlations of 0.600, 0.429, and 0.405 on narrative coherence, cross-shot layout consistency, and intra-shot layout-text alignment respectively. The paper's approach achieves 0.850, 0.767, and 0.786 — improvements of 0.250, 0.338, and 0.381. These are not marginal gains from prompt engineering; they represent a qualitative shift in evaluation reliability achieved by matching the scoring mechanism to the nature of the judgment.

The significance of this insight extends beyond MSAVBench. It implies that the field's current trajectory toward "VLM-as-universal-judge" evaluation frameworks is fundamentally misguided for certain evaluation categories, and that future benchmarks should adopt a similar stratified design: use expert models where objective accuracy matters, use structured rubrics where subjective consistency matters, and use tool-grounding where spatial precision matters. The paper does not claim that its specific decomposition is optimal — the three-paradigm taxonomy is pragmatic rather than theoretically derived — but the meta-level argument that evaluation strategy should be metric-dependent is a conceptual contribution that applies broadly.

This is an incremental methodological advance (the individual paradigms — expert models, rubrics, tool use — are not individually novel) but a fundamental rethinking of evaluation architecture (the principled combination of all three into a unified framework with per-metric selection based on judgment type is genuinely novel in the video generation evaluation literature).

---

Innovation 4: Diagnosing a Structural Barrier — The "Video-First, Post-Hoc Dubbing" Paradigm Is Inadequate

The paper's most actionable empirical finding is a negative result with architectural implications: the common strategy of generating multi-shot video first and then dubbing audio afterward using a separate model — the de facto approach for open-source MSAV pipelines — is fundamentally incapable of achieving competitive audio-visual alignment. This is not presented as a performance observation but as a structural diagnosis: the failure is inherent to the paradigm, not a matter of insufficient model scale or training data.

The evidence is in Table 2, specifically the contrast between open-source pipelines that follow this paradigm and those that do not. Native single-shot audio-video models concatenated shot-by-shot (LTX-2.3 in T2AV mode, DaVinci+MagiHuman in T2AV mode) achieve overall scores of 64.40 and 60.65 respectively. Multi-shot video models paired with post-hoc dubbing (ShotStream + HunyuanFoley, LongLive + HunyuanFoley, Helios + HunyuanFoley) achieve 58.85, 58.59, and 54.10. In contrast, the modular "image + audio-video" pipeline (LTX-2.3 in TI2AV mode) achieves 72.63 — competitive with closed-source systems like Wan2.7-T2V (72.26) and Kling-V3-T2V (72.25). The key difference is that the TI2AV pipeline generates audio and video jointly per shot, while the post-hoc dubbing pipeline generates video first and then attempts to synchronize audio afterward.

The paper traces this failure to a specific mechanism: "post-hoc dubbing lacks frame-level semantic grounding across hard camera cuts, disrupting joint cross-modal alignment" (Finding 4, Section 4.2). The word error rate (WER) column in Table 2 is particularly revealing — post-hoc dubbing pipelines show dramatically higher WER scores (LongLive + HunyuanFoley at 7.55, meaning over 700% word error rate, versus LTX-2.3 TI2AV at 0.49) — because the dubbing model has no access to the visual context that would constrain speech generation to match lip movements, speaker identity, and scene acoustics.

What makes this an innovation rather than an unsurprising observation is that the paper quantifies the failure across multiple open-source paradigms and multiple metrics, establishing that it is not a model-specific weakness but a paradigm-level limitation. Finding 4 explicitly states: "Conversely, unified architectures are essential for the MSAV task." This is a prescriptive claim — it tells the open-source community what kind of model to build — grounded in systematic empirical comparison rather than speculation.

The significance is that it provides a clear research direction for the open-source community: rather than incrementally improving post-hoc dubbing models (better vocoders, better lip-sync post-processing), the priority should be developing native joint audio-video generation architectures that model both modalities simultaneously. The paper does not provide such an architecture itself — that is beyond its scope — but the diagnosis creates the intellectual justification for that research program. This is a fundamental diagnostic contribution (it identifies a structural barrier rather than a performance gap) with practical implications (it tells the open-source community where to invest effort).

---

Innovation 5: Shot Self-Correction as a Meta-Evaluation Robustness Mechanism

The paper introduces a concept that is subtle but intellectually significant: the reliability of evaluation should not be gated by the accuracy of a single brittle preprocessing step, and the solution is to make the preprocessing step itself evaluable and correctable through iterative tool-invocation. This is a meta-evaluation insight — it's about how to build evaluation pipelines that remain trustworthy even when their input is imperfect — and it represents a departure from the standard assumption in video evaluation that shot boundary detection is a solved preprocessing problem.

The standard approach in prior multi-shot benchmarks (ViStoryBench, MSVBench, UniVBench) is to apply a fixed shot boundary detector (typically TransNet V2) and accept whatever boundaries it produces as ground truth for downstream per-shot and cross-shot metrics. The paper's key insight is that this introduces a single point of failure: if the detector incorrectly segments the video — splitting a dissolve transition into two shots, or merging a hard cut — every downstream metric that depends on shot boundaries becomes corrupted. The evaluation pipeline's overall reliability is bounded above by the detector's accuracy, which on generated videos with ambiguous transitions and variable shot counts is far from perfect.

The solution — an iterative VLM-driven correction loop with tool-based merging and splitting — is not novel in its individual components (TransNet V2 for initial detection, VLMs for visual inspection, tool invocation for action execution are all established techniques). What is novel is the architectural principle: rather than treating shot segmentation as a fixed preprocessing step and evaluation as subsequent, the framework makes segmentation itself part of the evaluation loop, subject to the same scrutiny and correction as any other metric. The VLM inspects boundaries not to produce a shot segmentation score (which would be circular) but to diagnose and repair errors before they propagate.

The paper does not provide a direct ablation quantifying how much the self-correction mechanism improves downstream metric reliability — this is a notable gap in the experimental validation. The claim of improved robustness is implicit rather than explicitly measured. However, the conceptual contribution stands independently: the paper identifies a previously unarticulated failure mode in video evaluation pipelines (shot boundary error propagation) and proposes a general architectural pattern (iterative self-correction with tool invocation) for addressing it. Future benchmarks in any domain where preprocessing errors cascade into evaluation errors can adopt this pattern regardless of the specific detectors and tools involved.

This is an incremental architectural innovation — it refines how evaluation pipelines are constructed rather than introducing a new evaluation metric or generation method — with broad applicability beyond MSAV evaluation. Any multi-step evaluation pipeline with a brittle early step (scene segmentation for activity recognition, object tracking for action evaluation, speaker diarization for conversation assessment) could potentially benefit from an analogous self-correction loop.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The benchmark itself — MSAVBench — is the evaluation dataset: 286 prompts comprising 2,198 individual shots, spanning 8 video genres, 6 visual styles, 6 audio content categories, 7 emotional states, 6 languages, and shot counts from 2 to 15 (mean 7.7). The test set is the entire benchmark. There is no separate train/validation/test split because the benchmark is used exclusively for evaluation. The data construction uses a four-stage pipeline combining LLM generation (GPT-5.4 for initial prompts, a Prompt Enhancement model for cinematic rewriting) with rigorous human expert curation, retaining only 13% (286/2200) of PE-rewritten prompts after filtering for diversity, structural complexity, and logical coherence.

• Base model(s). The paper evaluates 19 models across two families. (i) Closed-source commercial systems: Seedance-2.0, Wan2.7-T2V, Kling-V3-T2V, HappyHorse, Sora-2, plus reference-conditioned variants Wan-R2V and HappyHorse-R2V. (ii) Open-source pipelines divided into five categories: (1) single-shot audio-video models concatenated shot-by-shot — JavisDiT++, JavisGPT, MoVA (TI2AV mode), LTX-2.3, and daVinci-MagiHuman (in both T2AV and TI2AV modes); (2) multi-shot video models paired with dubbing — ShotStream + HunyuanFoley; (3) single-shot video models dubbed then concatenated — Wan2.2 + HunyuanFoley (TI2AV mode); (4) long-video models with multi-shot prompt input paired with dubbing — LongLive + HunyuanFoley, Helios + HunyuanFoley; (5) reference-conditioned open-source — DreamID-Omni. The selection spans the full spectrum of available MSAV-capable systems, chosen to represent the current state of both commercial and open-source capabilities, with the explicit note that "native open-source multi-shot audio-video models remain absent, constrained by data scarcity and prohibitive computational costs."

• Metrics. 20 metrics organized into four hierarchical levels (global, cross-shot, intra-shot, reference) and aggregated into 11 non-redundant dimensions for the overall score. The overall score is computed as: average_score = (1/11) × Σ score_d for the 11 dimensions, multiplied by a shot-completion penalty coefficient N_generated / N_specified. Each metric uses one of three scoring paradigms: specialized expert models (10 metrics, e.g., Synchformer for audio-visual sync, PP-OCRv5 for text rendering), instance-wise rubric-based VLM scoring (5 metrics, e.g., narrative coherence via predefined MCQs), or tool-grounded agentic scoring (5 metrics, e.g., layout-text alignment via Grounding DINO + BlazePose evidence extraction). All dimensions are normalized to [0, 1]. Human alignment is measured via Spearman's rank correlation ρ_s against expert pairwise judgments (described in Appendix D.2: 30 annotators for system-level evaluation producing 1,200 pairwise judgments; 10 annotators for fine-grained metrics producing 360 judgments each).

• Baselines. The evaluation framework itself has one primary baseline: direct VLM scoring (asking Qwen3.5 to produce an unconstrained scalar score for each dimension), which is compared against the paper's proposed three-paradigm approach on three metrics — narrative coherence, cross-shot layout consistency, and intra-shot layout-text alignment. The comparison is quantified in Table 6 via Spearman correlation with human judgments. The paper does not compare against other existing evaluation frameworks as baselines — the contribution is the framework itself, validated against human judgment as the ground truth. For model evaluation, baselines are the 19 evaluated systems compared against each other on the leaderboard (Table 2), with no single system designated as a reference baseline — the comparisons are relative across all models.

• Generation budget / compute accounting. For model evaluation, there is no explicit generation budget constraint — each model is run with its default inference configuration to produce outputs for all 286 prompts, and the outputs are evaluated post-hoc. The paper does not measure or control for per-model inference compute. For the evaluation framework itself, compute accounting is discussed in Appendix C.2: the framework "is designed to remain efficient in both tool usage and VLM calls" by (1) using expert models rather than VLMs for 10 metrics, (2) caching and reusing intermediate tool outputs across metrics (subject localization, embedding extraction, foreground removal, OCR, ASR are computed once and shared), and (3) using Qwen-based judges rather than requiring the largest VLMs. The self-correction loop is limited to two iterations to bound cost. Deployment details specify independent FastAPI micro-services on 8×A100 hosts (Appendix C.1), but no wall-clock time or dollar-cost measurements are reported.

• Cross-validation / statistical protocol. The human alignment study uses pairwise comparison with expert annotators, aggregated into method rankings via win rates (ties counted as 0.5 for each method). Spearman's rank correlation ρ_s is computed between automated metric rankings and human rankings. The robustness check substitutes the VLM backbone from Qwen3.5 to Qwen2.5-VL-32B-Instruct to assess sensitivity to model scale. For model evaluation (Table 2), no cross-validation or statistical significance testing is reported — scores are single-point estimates computed over the full 286-prompt benchmark. The paper does not report confidence intervals, standard deviations, or statistical tests comparing model rankings.

Main Quantitative Results

Overall Model Rankings and the Closed-Open Performance Gap

Table 2 presents the full leaderboard across all 20 atomic metrics and the Overall score. The headline finding from the Overall column is that closed-source commercial systems occupy the top five positions, with Seedance-2.0 at 75.92, Wan2.7-T2V at 72.26, Kling-V3-T2V at 72.25, HappyHorse at 71.89, and Sora-2 at 71.19. The gap between the best closed-source model (Seedance-2.0) and the best open-source pipeline (LTX-2.3 in TI2AV mode at 72.63) is 3.29 points — notable but not vast.

However, the critical result is not just the ordinal ranking but the architecture-dependent pattern within open-source systems. The open-source pipelines fall into distinct performance tiers that correlate with their generation strategy:

• Tier 1 — Modular "image + audio-video" pipelines (TI2AV): LTX-2.3 in TI2AV mode achieves 72.63, which is competitive with and slightly exceeds Wan2.7-T2V (72.26) and Kling-V3-T2V (72.25). MoVA in TI2AV mode achieves 70.32. This tier demonstrates that decomposing the MSAV problem into per-shot keyframe synthesis followed by audio-video generation can approach closed-source performance.

• Tier 2 — Native single-shot audio-video models concatenated shot-by-shot (T2AV): These fall substantially below Tier 1. LTX-2.3 in T2AV mode drops to 64.40 — a decline of 8.23 points from its TI2AV counterpart. DaVinci+MagiHuman achieves 65.01 in TI2AV and falls to 60.65 in T2AV. The performance degradation when switching from image-conditioned to text-only generation suggests that visual grounding from keyframe images provides critical structural information that pure text prompts do not convey with sufficient precision.

• Tier 3 — Multi-shot or long-video models with post-hoc dubbing: ShotStream + HunyuanFoley (58.85), LongLive + HunyuanFoley (58.59), Helios + HunyuanFoley (54.10), and Wan2.2 + HunyuanFoley in TI2AV mode (63.42 — notably higher than the other dubbing pipelines, likely because TI2AV provides visual conditions that partially compensate). These pipelines cluster in the mid-50s to low-60s, substantially below Tier 1.

• Tier 4 — Early joint audio-video models: JavisDiT++ (57.51) and JavisGPT (53.95) represent native joint generation but fall into the lower tier, suggesting that architectural integration alone is insufficient without sufficient training scale or data.

This tiered structure directly supports the paper's Finding 1: "A significant performance gap persists between closed and open-source models, but modular agentic frameworks show potential to bridge it." The evidence is nuanced — the gap is not monolithic. Open-source pipelines at Tier 1 (TI2AV) are already competitive with closed-source mid-tier (Wan2.7, Kling-V3), while Tier 3 pipelines (post-hoc dubbing) lag substantially. The "potential to bridge" is demonstrated by the fact that architectural choices within open-source systems produce a 14-point spread (from 53.95 for JavisGPT to 72.63 for LTX-2.3 TI2AV) — larger than the spread between the best and worst closed-source systems (75.92 to 71.19, a 4.73-point spread).

Director-Level Structural Control and Cinematic Language

Table 2's cross-shot and intra-shot columns provide granular evidence for Finding 2: "open-source models lag significantly behind closed systems in 'director-level' structural control and cinematic language."

Cross-shot layout consistency (C-Layout): The closed-source range is 0.632 (HappyHorse) to 0.809 (Seedance-2.0). The open-source range is 0.267 (DaVinci+MagiHuman T2AV) to 0.679 (Wan2.2 + HunyuanFoley TI2AV). The best open-source score (0.679) is below the worst closed-source score (0.632) with the exception of HappyHorse. More revealing is the collapse of certain open-source pipelines on this dimension: DaVinci+MagiHuman achieves 0.422 (TI2AV) and 0.267 (T2AV), ShotStream + HunyuanFoley achieves 0.280, and LongLive + HunyuanFoley achieves 0.589. These low scores indicate a fundamental inability to maintain spatial coherence of subjects across shot boundaries.

Camera parameter adherence (Cam.): This metric shows the most dramatic closed-open gap. Closed-source models achieve 0.617 (Wan2.7-T2V) to 0.846 (Kling-V3-T2V). However, several open-source pipelines achieve higher camera adherence than closed-source: Wan2.2 + HunyuanFoley (0.957) and LongLive + HunyuanFoley (0.956). This is a counterintuitive result — the finding that open-source pipelines can outperform on camera control while underperforming on layout consistency suggests that camera parameter execution and spatial layout control are distinct capabilities that do not covary. A possible explanation is that per-shot image generation (the approach in Wan2.2's pipeline) allows precise camera framing through the image generation model's learned priors, while spatial layout across shots requires explicit cross-shot reasoning that post-hoc dubbing pipelines lack.

Intra-shot layout-text alignment (I-Layout): Closed-source models: 0.609 (Kling-V3-T2V) to 0.822 (Seedance-2.0). Open-source: 0.151 (Helios + HunyuanFoley) to 0.765 (LTX-2.3 TI2AV). Here the open-source spread is massive — 0.614 points — reflecting that layout-text alignment is highly sensitive to pipeline architecture. DaVinci+MagiHuman (0.473 TI2AV, 0.494 T2AV), ShotStream + HunyuanFoley (0.243), and Wan2.2 + HunyuanFoley (0.430) all perform poorly, suggesting that models without native joint audio-video generation or explicit keyframe conditioning struggle to align spatial arrangements with text descriptions.

Visual consistency sub-metrics (VC): Closed-source: 0.790 (HappyHorse) to 0.856 (Kling-V3-T2V). Open-source: 0.480 (JavisDiT++) to 0.857 (LongLive + HunyuanFoley). The top open-source score (LongLive, 0.857) slightly exceeds Kling-V3-T2V (0.856), suggesting that long-video generation models can maintain visual consistency across shots even when other aspects (layout, audio) fail.

The overall pattern is that no single open-source pipeline is strong on all cinematic control dimensions simultaneously. LTX-2.3 TI2AV excels at layout-text alignment (0.765) but has only moderate camera adherence (0.814). Wan2.2 + HunyuanFoley excels at camera adherence (0.957) but has poor layout-text alignment (0.430). This suggests that architectural choices involve tradeoffs between different types of structural control, and the community lacks a unified architecture that handles all aspects of director-level generation.

Fine-Grained Audio-Visual Alignment

Four metrics in Table 2 probe joint audio-visual alignment, supporting Finding 3: "Fine-grained joint audio-visual alignment remains an unsolved challenge for both closed and open-source models."

Lip synchronization (Lip): Note that this metric uses raw sync offset — lower is better. The closed-source range is 0.73 (HappyHorse) to 1.87 (Sora-2). The open-source range: JavisGPT achieves 0.42 (best open-source lip-sync, better than all closed-source models), while DaVinci+MagiHuman T2AV achieves 4.91 (worst overall). This bimodal distribution is striking — some open-source models (JavisGPT at 0.42, JavisDiT++ at 0.59) achieve excellent lip-sync, while others (DaVinci+MagiHuman at 4.91, LTX-2.3 T2AV at 0.96, MoVA TI2AV at 1.61) range from moderate to catastrophic. The pattern correlates with audio generation strategy: JavisGPT/JavisDiT++ generate audio and video jointly, while dubbing pipelines struggle. Even the best closed-source system (Seedance-2.0 at 1.52) does not achieve the lip-sync quality of JavisGPT, suggesting that larger model scale does not automatically translate to better audio-visual synchronization.

Sound attribution (Attr.): Closed-source: 0.568 (Sora-2) to 0.661 (Wan2.7-T2V). Open-source: 0.113 (JavisGPT) to 0.654 (DaVinci+MagiHuman T2AV — notably the same model that fails catastrophically on lip-sync). The best open-source score (0.654) is competitive with closed-source (0.661). However, the worst open-source scores — JavisGPT at 0.113 and Helios + HunyuanFoley at 0.138 — indicate near-total failure to associate speech with the correct visible speaker.

Audio-visual synchronization (Sync): Lower is better (temporal offset in seconds). Closed-source: 0.14 (Seedance-2.0) to 0.50 (Sora-2). Open-source: 0.05 (DaVinci+MagiHuman T2AV) to 0.79 (Helios + HunyuanFoley). Seedance-2.0's 0.14 is the second-best overall, while DaVinci+MagiHuman T2AV's 0.05 is the best. This counterintuitive result — the model with the worst lip-sync (4.91) has the best audio-visual sync (0.05) — highlights that lip synchronization and general audio-visual synchronization are distinct capabilities that do not correlate. DaVinci+MagiHuman may produce temporally aligned audio and visual events (low Sync) while failing to match phoneme-level lip movements (high Lip). Conversely, Seedance-2.0 achieves moderate lip-sync (1.52) but excellent audio-visual sync (0.14).

Speaker timbre consistency (Spk.): Closed-source: 0.520 (Sora-2) to 0.673 (HappyHorse). Open-source: 0.097 (JavisGPT) to 0.699 (DaVinci+MagiHuman T2AV). Again, DaVinci+MagiHuman T2AV achieves the best open-source score (0.699), exceeding all closed-source models, while JavisGPT achieves near-zero (0.097). The pattern across these four audio-visual metrics is consistent: no model, closed- or open-source, excels at all four simultaneously. Seedance-2.0 is the most balanced, with moderate-to-good scores across all four. The open-source landscape is characterized by extreme specialization — each pipeline is good at some audio-visual dimensions and terrible at others — suggesting that architectural choices involve fundamental tradeoffs in audio-visual joint modeling.

The Post-Hoc Dubbing Paradigm Failure

Table 2's Overall column and WER column provide direct evidence for Finding 4: "The alternative 'video-first, post-hoc dubbing' paradigm is inadequate."

The three pipelines using HunyuanFoley for post-hoc dubbing — ShotStream (58.85), LongLive (58.59), and Helios (54.10) — all fall in the bottom tier of Overall scores. The word error rate (WER) column quantifies the specific failure mechanism claimed by the paper: "post-hoc dubbing causes severe speech distortion (high WER) and poor lip-sync." The WER metric (lower is better) shows:

• LongLive + HunyuanFoley: 7.55. This is an extreme outlier — the benchmark appears to cap WER at 1.0 in the scoring formula (score = 1 - min(WER, 1)), but the raw WER of 7.55 means that the generated speech has over 7 times more errors than words in the reference transcript. This is not a marginal degradation — it indicates near-total failure of speech generation.
• Wan2.2 + HunyuanFoley: 1.39 (over 100% error rate).
• Helios + HunyuanFoley: 1.24 (over 100% error rate).
• ShotStream + HunyuanFoley: 1.00 (exactly 100% error rate).

In contrast, native single-shot audio-video models achieve dramatically lower WER: LTX-2.3 TI2AV at 0.49, LTX-2.3 T2AV at 0.53, MoVA TI2AV at 0.66. Closed-source models: Seedance-2.0 at 0.54, Wan2.7-T2V at 0.49, HappyHorse at 0.51. The WER gap between post-hoc dubbing and native generation is roughly 4–7× in error rate.

The paper's claim that this failure is due to "lack of frame-level semantic grounding across hard camera cuts" is supported by the pattern that the TI2AV pipelines (which provide per-shot keyframe images as conditions) achieve much lower WER than pure post-hoc dubbing, even when using the same or similar audio generation backends. LTX-2.3 TI2AV (0.49) vs. LongLive + HunyuanFoley (7.55) is a difference in WER of over 15×, despite both being open-source pipelines. This suggests that the critical factor is not model scale or training data quantity, but whether audio generation has access to visual conditioning at the frame level. The post-hoc dubbing paradigm severs this connection: the audio model sees the completed video but cannot attend to frame-level visual features during the generation process in the same way that a jointly trained model can.

Complexity Scaling: Shot Count and Realism

Tables 3 and 4 decompose Overall scores by structural complexity dimensions that are invisible in the aggregate leaderboard.

Shot count scaling (Table 3): All models degrade as shot count increases, but the degradation rate differs dramatically across model families. Looking at the transition from 1–4 shots to 11–15 shots:

• Closed-source models are remarkably robust: Seedance-2.0 drops only from 77.70 to 76.00 (−1.70), Wan2.7-T2V from 73.50 to 72.10 (−1.40), Kling-V3-T2V from 73.50 to 70.00 (−3.50), HappyHorse from 74.90 to 72.80 (−2.10). The worst closed-source degradation is Kling-V3 at 3.50 points.
• Open-source TI2AV pipelines show moderate degradation: LTX-2.3 TI2AV drops from 75.10 to 72.50 (−2.60), DaVinci+MagiHuman TI2AV from 70.90 to 62.30 (−8.60). The 8.60-point drop for DaVinci is already substantial and suggests instability at higher shot counts.
• Open-source dubbing/long-video pipelines collapse catastrophically: LongLive + HunyuanFoley drops from 66.10 to 41.60 (−24.50), Wan2.2 + HunyuanFoley from 71.80 to 60.10 (−11.70). The 24.50-point drop for LongLive — from a moderate 66.10 to a failing 41.60 — indicates that the model fundamentally cannot maintain any generation quality at 11–15 shots.
• Native joint audio-video models are flat but at low baseline: JavisDiT++: 60.40 → 59.30 (−1.10), but the baseline performance is already low.

The key inference is that multi-shot consistency is the primary bottleneck, not per-shot quality. If per-shot quality were the limiting factor, the degradation would be shallow and linear — losing a small amount of quality with each additional shot. The catastrophic collapse of LongLive (−24.50) suggests a nonlinear failure mode: beyond some threshold shot count (likely around 5–7 shots), the model loses the ability to maintain any coherent structure, and the quality collapses to near-random. This is consistent with the autoregressive error accumulation hypothesis: long-video models that generate frames sequentially accumulate errors that compound nonlinearly, while closed-source systems and modular pipelines (which generate each shot independently or with explicit structural conditioning) are more robust.

Realistic vs. non-realistic (Table 4): All models degrade on non-realistic prompts relative to realistic prompts:

• Seedance-2.0: 76.80 → 74.50 (−2.30)
• Wan2.7-T2V: 73.40 → 70.50 (−2.90)
• LTX-2.3 TI2AV: 74.20 → 70.50 (−3.70)
• LongLive + HunyuanFoley: 60.30 → 56.00 (−4.30)
• JavisDiT++: 61.00 → 56.40 (−4.60)

The degradation magnitude is correlated with baseline performance — better models degrade less in absolute terms — but the relative degradation (as a proportion of realistic performance) is more uniform across models. This supports the paper's claim that "generating out-of-distribution visual contents universally compromises performance across all model families" (Section 4.3). The mechanism is likely that non-realistic content (fictional characters, counterfactual scenes, stylized visuals) lies outside the training distribution, causing the model to either mode-collapse to the nearest realistic analog or produce incoherent outputs.

Reference-Conditioned Generation

Table 5 provides results on the 96 prompts with reference conditions, revealing an asymmetric difficulty pattern: visual fidelity preservation is harder than voice preservation.

• Image fidelity (Img-DINO): Closed-source Wan-R2V achieves 0.208, HappyHorse-R2V achieves 0.259. Open-source DreamID-Omni achieves 0.119. The gap between best closed-source (0.259) and open-source (0.119) is over 2×.
• Face fidelity (Img-Face): Wan-R2V: 0.368, HappyHorse-R2V: 0.244, DreamID-Omni: 0.054. The gap is even larger — nearly 7× between Wan-R2V and DreamID-Omni — and face preservation appears to be the hardest sub-dimension.
• Voice fidelity (Voice): Wan-R2V: 0.657, HappyHorse-R2V: 0.545, DreamID-Omni: 0.535. The gap here is much narrower — only 0.122 between best closed-source and open-source, and DreamID-Omni (0.535) closely approaches HappyHorse-R2V (0.545).

The paper interprets this as evidence that "visual preservation is harder than voice cloning in joint audio-visual customization" (Section 4.3). A more precise formulation: voice cloning from reference audio is a relatively well-understood problem with mature techniques (speaker embedding extraction, voice conversion), while visual identity preservation from reference images in generated video remains substantially harder, likely because it requires maintaining consistent appearance across diverse poses, lighting conditions, and camera angles — a 3D-consistent generation problem that current video models have not fully solved.

Ablation Studies and Robustness Checks

Human alignment of the overall score (Table 6): The overall score achieves Spearman ρ_s = 0.915 against human system-level rankings. No direct ablation of the aggregation formula is provided — the paper does not test whether a different weighting of the 11 dimensions, a different shot-completion penalty function (e.g., quadratic rather than linear), or a different merging of overlapping metrics would produce higher or lower correlation. The 0.915 correlation is reported as validation, not as the result of optimization over aggregation choices.

Instance-wise rubric-based scoring vs. direct VLM scoring (Table 6): On narrative coherence, direct VLM scoring (Qwen3.5) achieves ρ_s = 0.600. Instance-wise rubric scoring with Qwen3.5 achieves 0.850 — an improvement of +0.250. With the smaller Qwen2.5-VL backbone, rubric scoring achieves 0.820, a drop of only 0.030 from the larger model. This ablation demonstrates that the improvement comes from the scoring paradigm (structured MCQs vs. unconstrained scalar) rather than from model scale. The paper does not report direct VLM scoring with Qwen2.5-VL, which would provide a complete 2×2 ablation (backbone × paradigm).

Tool-grounded agentic scoring vs. direct VLM scoring (Table 6): On cross-shot layout consistency, direct VLM scoring achieves 0.429. Tool-grounded scoring with Qwen3.5 achieves 0.767 (+0.338). With Qwen2.5-VL, tool-grounded achieves 0.732 (−0.035). On intra-shot layout-text alignment, direct VLM scoring achieves 0.405. Tool-grounded with Qwen3.5 achieves 0.786 (+0.381). With Qwen2.5-VL: 0.741 (−0.045). Two key findings: (1) the tool-grounding improvement is larger for layout-text alignment (+0.381) than for layout consistency (+0.338), suggesting that intra-shot spatial reasoning is more challenging for VLMs than cross-shot comparison; (2) the drop when switching backbones is small (0.035–0.045), confirming that tool-grounding, not model scale, drives the improvement.

VLM backbone robustness (Table 6): Across all three ablated metrics, the Qwen3.5 → Qwen2.5-VL substitution reduces Spearman correlation by at most 0.045 points (intra-shot layout-text alignment). The rubric-based and tool-grounded designs remain above 0.73 correlation with the smaller model, compared to 0.405–0.600 for direct scoring with the larger model. This is strong evidence that the evaluation framework's alignment with human judgment does not depend on access to the largest, most expensive VLM — a practically significant finding for adoption by the open-source community.

Missing ablation — shot self-correction impact: The paper does not provide a direct ablation measuring how much the self-correction mechanism improves downstream metric reliability or overall score accuracy. The claim that self-correction "mitigates shot count anomalies" (Section 3.4.2) and "eliminates cascading failures caused by shot segmentation errors" (Section 3.4.2) is stated but not quantified. An experiment comparing overall scores and per-metric scores with and without the self-correction loop would directly validate this claim. Without such an ablation, the magnitude of the improvement — and whether it justifies the computational cost of iterative VLM inspection — remains unknown.

Missing ablation — rubric instantiation quality: The paper does not ablate the effect of rubric design quality on scoring reliability. The instance-wise rubrics are instantiated from prompt metadata (Section 3.4.2, Appendix B.2), but the paper does not test whether poorly designed rubrics (ambiguous MCQ wording, missing answer options) or well-designed rubrics with adversarial distractors would affect the Spearman correlation. The reported correlations are conditional on the specific rubric instantiation procedure used, and may not generalize to benchmarks where metadata is noisier or less structured.

Missing ablation — tool availability and quality: The tool-grounded scoring paradigm depends on Grounding DINO and BlazePose providing accurate detections. The paper does not ablate how the scoring degrades when these tools fail (e.g., for heavily stylized or low-resolution video where object detection accuracy drops). Nor does it test whether alternative tools (different object detectors, different pose estimators) would produce similar or different scoring behavior. The assumption that tool outputs are reliable ground truth — rather than themselves sources of error — is untested.

Negative result — ReST^EM revision model (Appendix K, Figure 16): This is not directly part of MSAVBench evaluation but is a notable negative result in the broader experimental narrative: an attempt to optimize a revision model using ReST^EM (Singh et al., 2024) causes additional sequential revisions to substantially hurt performance. The paper hypothesizes that on-policy data collection exacerbates spurious correlations in revision data. This finding, while from a different model family than MSAVBench evaluation, serves as a cautionary example about the fragility of training pipelines for complex generation tasks.

Critical Assessment

Claim 1: MSAVBench is the first comprehensive benchmark for MSAV generation

What the experiments demonstrate: Table 1 compares MSAVBench against 8 prior benchmarks across 11 coverage dimensions. MSAVBench is the only benchmark that checks all 11 boxes. The data analysis (Figures 2, 6) provides distributional evidence that the 286-prompt/2,198-shot benchmark spans the claimed diversity dimensions (8 genres, 6 styles, 6 audio categories, 7 emotions, 6 languages, 2–15 shots, realistic/non-realistic).

What is not demonstrated: Being "the first" is a factual claim about publication order, not an empirical claim. The experiments demonstrate coverage — that the benchmark contains diverse prompts across the claimed dimensions — but do not demonstrate that this coverage is sufficient for comprehensive evaluation. There is no experiment showing that the 286 prompts are enough to reliably rank 19 models (i.e., that the ranking would be stable under resampling of prompts), nor that the 2,198 shots provide adequate per-dimension sample sizes. For example, the retro film visual style accounts for only 0.3% of prompts (approximately 1 prompt). Nature & extreme weather is 2.8% (~8 prompts). Scores on these dimensions are estimated from very few data points, and the paper does not report per-genre or per-style breakdowns of model performance that would reveal whether rankings are stable within low-sample categories.

Genuine weaknesses: (1) The benchmark construction uses GPT-5.4 for initial prompt generation — a closed-source model whose behavior may change over time, making the prompt generation process not fully reproducible. (2) The 87% rejection rate in human curation (2,200 → 286 prompts) raises the question of whether the surviving prompts are systematically biased — perhaps the prompts that survived expert review are those that happen to be easier to evaluate or more "canonical," excluding edge cases that would better stress-test models. (3) The benchmark is English+Chinese dominant (229 of 286 prompts) despite supporting 6 languages — the multilingual evaluation claims are supported by only 15 Japanese, 15 Korean, 14 Spanish, and 13 French prompts, which is insufficient for statistically reliable per-language comparisons.

Claim 2: The adaptive hybrid evaluation framework improves robustness and achieves high alignment with human judgments

What the experiments demonstrate: Table 6 provides strong evidence that rubric-based and tool-grounded scoring substantially improve Spearman correlation with human judgments compared to direct VLM scoring (improvements of +0.250 to +0.381). The overall score achieves 0.915 correlation. The backbone substitution shows that the improvement is robust to VLM scale.

What is not demonstrated: (1) The experiment compares the proposed approach against direct VLM scoring with Qwen3.5 only — there is no comparison against other VLM judges (GPT-4V, Gemini, Claude) that might perform better at direct scoring. The claim that direct VLM scoring is unreliable is demonstrated for Qwen3.5 specifically, not for VLMs in general. A stronger version of this claim would require showing that no VLM can achieve competitive correlation through direct scoring alone. (2) The human alignment study uses expert annotators (full-time AIGC researchers) — the correlation of 0.915 is with expert judgment, but the claimed benefit is for the open-source community and model developers. It is unclear whether expert judgments align with broader user satisfaction or downstream task performance. (3) The paper does not ablate the marginal contribution of each component within the framework — how much does rubric scoring alone improve over direct scoring? How much does tool-grounding alone improve? How much does the combination improve? The 2×2 ablation (rubric alone, tool alone, both) is missing, so the relative importance of each innovation is unknown.

Genuine weaknesses: (1) The correlation metric (Spearman's ρ_s) measures rank agreement, not calibration. A framework could achieve perfect rank correlation while systematically overestimating or underestimating absolute quality — models ranked correctly but scored 20 points too high or too low. The paper does not report mean absolute error or other calibration metrics. (2) The self-correction mechanism's contribution to robustness is claimed but never quantified — there is no experiment showing MSAVBench scores with vs. without self-correction, or demonstrating that self-correction actually fixes mis-segmentations in practice. (3) The human annotation sample size for fine-grained metrics (360 judgments per metric, 10 annotators) is relatively small for establishing correlation coefficients — confidence intervals on the reported ρ_s values are not provided, making it difficult to assess whether the differences between Qwen3.5 and Qwen2.5-VL (0.030–0.045) are statistically significant or within sampling error.

Claim 3: Modular and agentic generation pipelines narrow the gap between open- and closed-source models

What the experiments demonstrate: Table 2 shows LTX-2.3 in TI2AV mode achieving 72.63 Overall, which is competitive with Wan2.7-T2V (72.26) and Kling-V3-T2V (72.25). The spread within open-source pipelines (53.95 to 72.63) is larger than within closed-source (71.19 to 75.92).

What is not demonstrated: (1) This is a single point of evidence — LTX-2.3 TI2AV is one pipeline on one benchmark. The paper does not test whether other modular pipelines (different image generation models, different audio-video backbones) would achieve similar or better results, so the finding may be specific to LTX-2.3's architecture rather than generalizable to modular pipelines as a class. (2) The "agentic" characterization is somewhat aspirational — the TI2AV pipeline is "modular" (image generation → audio-video generation) but not "agentic" in the sense of dynamic decision-making or tool use. The paper uses "modular or agentic" as a compound phrase throughout, making it unclear whether the demonstrated benefit comes from modularity, from agency, or from something else entirely. (3) The comparison is between pipelines with very different total parameter counts, training data, and inference compute — these are not controlled for, so attributing the performance difference to "modularity" rather than scale or data is speculative.

Genuine weaknesses: (1) The TI2AV pipeline uses Wan2.7-Image for storyboard generation — a model from the same family as the closed-source Wan2.7-T2V. The competitive performance of LTX-2.3 TI2AV may partially reflect the quality of Wan2.7-Image's conditioning rather than LTX-2.3's capabilities. A more rigorous test would use a fully open-source image generation model for the storyboard step. (2) The claim that modular pipelines "show potential to bridge" the gap is directional but underspecified — how much of the remaining 3.29-point gap to Seedance-2.0 can modular approaches close? The paper provides no extrapolation or scaling analysis.

Claim 4: Current models remain far from reliable "director-level" generation

What the experiments demonstrate: Table 2's cross-shot and intra-shot columns show that even the best models have substantial room for improvement — Seedance-2.0 achieves cross-shot layout consistency of 0.809 and camera parameter adherence of 0.801, meaning approximately 20% of layout and camera specifications are not satisfied. Audio-visual synchronization (Sync) ranges from 0.14 (Seedance-2.0, lower is better) to 0.79 (Helios + HunyuanFoley) — meaning the worst models have nearly 0.8 seconds of audio-visual offset on average.

What is not demonstrated: "Director-level" is not operationally defined. What threshold on each metric constitutes "director-level" performance? Is 0.90 camera adherence sufficient? 0.95? The claim is qualitative and supported by the observation that scores are below 1.0, but without an operational threshold, the claim cannot be empirically falsified — a model achieving 0.999 on all metrics would still be "far from director-level" by the paper's framing, which is unfalsifiable.

Genuine weaknesses: The metrics measure prompt adherence (did the model execute the specified cinematic language?), not creative directorial quality (would a human director judge the shot composition as aesthetically effective?). A model could achieve perfect camera parameter adherence while producing boring, poorly composed shots — the benchmark penalizes deviations from the specification but does not reward excellence beyond specification. This is appropriate for a benchmark focused on controllability, but it means "director-level" conflates two distinct capabilities: instruction following and aesthetic judgment.

Claim 5: The "video-first, post-hoc dubbing" paradigm is inadequate, and unified architectures are essential

What the experiments demonstrate: The post-hoc dubbing pipelines consistently underperform, particularly on WER (7.55 for LongLive + HunyuanFoley vs. 0.49 for LTX-2.3 TI2AV) and lip synchronization. The contrast is stark and consistent across multiple dubbing pipelines.

What is not demonstrated: (1) The dubbing pipeline uses HunyuanFoley as the audio model — a single specific dubbing model. The conclusion that "post-hoc dubbing is inadequate" would be stronger if the failure replicated across multiple dubbing models with different architectures. It is possible that HunyuanFoley is a weak dubbing model and that a better dubbing model (not tested) would substantially close the gap. (2) The claim that "unified architectures are essential" is a causal claim — that the joint training of audio and video generation is the causal mechanism producing the improvement. The evidence is correlational: the models with joint generation (LTX-2.3 T2AV, JavisDiT++) perform better on WER than those with post-hoc dubbing. But the TI2AV pipeline (LTX-2.3 TI2AV) — which is not a unified architecture in the sense of joint audio-video training, but rather a modular pipeline with per-shot conditioning — achieves competitive WER (0.49), suggesting that the essential factor may be visual conditioning at generation time rather than joint training specifically. The paper does not disentangle these two hypotheses.

Missing experiment — "video-first, audio-first, joint" comparison: A direct three-way comparison between (a) video-first + dubbing, (b) audio-first + video generation conditioned on audio, and (c) joint audio-video generation, with all other factors controlled, would distinguish between the "conditioning modality matters" and "joint training matters" hypotheses. This experiment is not run.

Missing Baselines and Ablations That Would Strengthen the Paper

1. Per-genre and per-difficulty breakdown of model rankings: The paper reports overall scores and shot-count/realism breakdowns (Tables 3, 4), but does not report per-genre performance (do models rank differently on Action vs. Narrative vs. Tutorial?), per-style performance (do all models struggle equally on anime vs. photorealistic?), or per-language WER (do models perform differently on Chinese vs. Japanese?). These breakdowns would substantially increase the diagnostic value of the benchmark — the paper's stated goal — by identifying genre-specific or language-specific failure modes.

2. Statistical significance of model rankings: The overall scores are point estimates without confidence intervals. With 286 prompts, the standard error of the mean overall score is non-trivial. Two models separated by 0.01 in overall score (e.g., Wan2.7-T2V at 72.26 and Kling-V3-T2V at 72.25) are almost certainly not statistically distinguishable, but the paper presents ranked leaderboards without caveats. Bootstrap confidence intervals on the overall scores would indicate which ranking differences are reliable and which are noise.

3. Self-correction ablation: The claim that self-correction improves robustness is stated but not quantified. An experiment comparing MSAVBench scores with and without the self-correction loop, on the same set of generated videos, would directly measure the mechanism's contribution to evaluation reliability. Additionally, measuring the frequency and types of corrections made by the self-correction loop (how many merges, how many splits, in what fraction of videos) would characterize the prevalence of the segmentation failure mode the mechanism is designed to address.

4. Human-alignment comparison against other evaluation frameworks: Table 6 compares MSAVBench's proposed approach against direct VLM scoring, but does not compare against other existing evaluation frameworks applied to the same videos. Running VBench's evaluation pipeline (or AVGen-Bench's, or UniVBench's) on the MSAVBench-generated videos and measuring their correlation with human judgments would establish whether MSAVBench's improved alignment comes from the framework design or simply from having a more diverse prompt set.

5. Scalability analysis: The evaluation framework's computational cost is discussed qualitatively (Appendix C.2) but not quantified in FLOPs, GPU-hours, or dollar cost. The self-correction loop's cost (two VLM iterations over all shot boundaries) is not reported. For the benchmark to be practically adoptable by the open-source community, users need to know whether evaluating one model costs $10 or$1,000 in API credits. This is a critical missing detail for a framework positioned as infrastructure for the community.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in the Evaluation Framework

The assumption or constraint. The adaptive hybrid evaluation framework — particularly the tool-grounded agentic scoring paradigm — assumes that external perception tools (Grounding DINO, BlazePose) can be invoked at evaluation time to ground VLM judgments in objective evidence. The paper acknowledges the computational cost implicitly in Appendix C.2, stating that the framework "is designed to remain efficient in both tool usage and VLM calls" through caching and reuse, but never quantifies this cost in wall-clock time, GPU-hours, or dollar terms. The self-correction loop — which invokes a VLM (Qwen3.5) for up to two iterations of shot boundary inspection, tool-based merging/splitting, and re-evaluation — similarly has its cost acknowledged only through a design constraint ("we limit this process to a maximum of two iterations" to "balance accuracy and computational cost," Section 3.4.2) without any measurement of what that cost actually is.

The consequence. A practitioner deciding whether to adopt MSAVBench for evaluating their own models cannot make an informed cost-benefit decision. The framework deploys specialized expert models (Synchformer, LR-ASD, SortFormer, StableSyncNet, Demucs, MuQ, w2v-BERT, Audiobox-Aesthetic, PP-OCRv5, FireRedASR2/Whisper) across 10 metrics, VLM judges (Qwen3.5, Gemini 3.1 Pro) across 5 rubric-based metrics, and tool-invoking VLM agents across 5 tool-grounded metrics — all deployed as "independent FastAPI micro-services on 8×A100 hosts" (Appendix C.1). The cumulative computational cost of evaluating a single model on all 286 prompts could easily exceed the cost of generating the videos in the first place, particularly for lightweight open-source models. If evaluating one model costs $500 in compute while generating the videos costs$50, the benchmark is impractical for the resource-constrained open-source community it explicitly aims to serve. The cost problem is compounded by the self-correction loop: without knowing how frequently corrections are needed or how much each VLM inspection costs, a user cannot estimate whether the robustness improvement justifies the overhead.

What evidence exists in the paper. None. Appendix C.2 provides qualitative cost-mitigation strategies (caching, expert model preference, smaller VLM backbones) but reports no quantitative measurements — no FLOPs counts, no GPU-hours per evaluation, no API cost estimates, no wall-clock time for end-to-end evaluation of a single model. The robustness check (Table 6) shows that substituting Qwen3.5 with the smaller Qwen2.5-VL-32B-Instruct causes only small drops in Spearman correlation (0.030–0.045), which is encouraging for cost reduction, but the absolute cost difference between the two backbones is not reported, and the tool-based metrics (which invoke detectors and pose estimators) have costs independent of the VLM backbone.

Mitigation status. The paper acknowledges the cost concern implicitly but does not address it quantitatively. The caching strategy (intermediate tool outputs reused across metrics) and the smaller-backbone robustness result are partial mitigations — they suggest the cost can be reduced — but without baseline measurements, a user cannot assess whether the reduced cost is still prohibitive. The paper does not suggest specific future work on cost optimization or provide a "budget" mode that trades evaluation accuracy for reduced compute.

---

Limitation 2: No Evidence That the Self-Correction Mechanism Actually Improves Downstream Metric Accuracy

The assumption or constraint. The shot self-correction loop (Section 3.4.2, Figure 3) is presented as a key robustness innovation: "To eliminate cascading failures caused by shot segmentation errors, we introduce an agentic pre-processing phase" where a VLM iteratively inspects TransNet V2 boundaries and invokes tools to merge or split segments. The paper assumes that (a) segmentation errors are common enough and severe enough to meaningfully degrade downstream metrics, and (b) the VLM-driven correction loop successfully identifies and fixes these errors often enough to justify its computational cost.

The consequence. Without empirical validation that the self-correction mechanism actually improves shot boundary quality — and, crucially, that this improvement translates to more reliable downstream per-shot and cross-shot metric scores — the mechanism's contribution to the framework is an unverified claim. The paper's entire argument about evaluation robustness rests partially on this component: if the correction loop is ineffective (the VLM makes as many errors as it fixes), or if the base TransNet V2 segmentation is already accurate enough that correction provides negligible benefit, then the mechanism adds cost without adding value. Worse, if the VLM's boundary adjustments introduce new errors (incorrectly merging distinct shots or splitting continuous ones), the self-correction could actively degrade evaluation reliability.

What evidence exists in the paper. None. The paper provides no ablation comparing MSAVBench scores with and without the self-correction loop on the same set of generated videos. There is no measurement of: (a) how frequently TransNet V2's initial boundaries are incorrect (on generated MSAV videos with ambiguous transitions), (b) what types of errors occur (false splits at dissolve transitions? missed cuts at rapid scene changes?), (c) what fraction of these errors the VLM correction loop successfully identifies and fixes, (d) whether the VLM introduces new errors, or (e) how downstream metric scores change when computed on corrected vs. uncorrected boundaries. Figure 3 illustrates the mechanism conceptually, and Section 3.4.2 describes its operation, but the paper treats its benefit as self-evident rather than empirically demonstrated.

Mitigation status. The paper does not acknowledge this as a limitation, nor does it suggest future work to validate the mechanism. The two-iteration limit is a pragmatic cost constraint, not an accuracy-motivated choice — the paper provides no evidence that two iterations are sufficient to resolve most errors, or that additional iterations would not further improve boundary quality. The fallback behavior (when shot count remains mismatched after two iterations, "the VLM performs a final shot-caption re-alignment, discarding non-aligned segments") is described but its frequency of invocation is not reported. A user adopting MSAVBench has no way to assess whether skipping the self-correction loop — and accepting TransNet V2 boundaries directly, with the associated cost savings — would produce materially different model rankings.

---

Limitation 3: Benchmark Construction Reproducibility Depends on a Closed-Source LLM and Subjective Expert Curation

The assumption or constraint. The benchmark's 286 prompts (comprising 2,198 shots) are constructed through a four-stage pipeline (Section 3.2, Appendix A.2) whose two central generation stages rely on GPT-5.4, a closed-source model: (1) initial prompt generation from seed quadruples, and (2) Prompt Enhancement (PE) rewriting into cinematic global-to-shot format. The pipeline then applies human expert curation — six domain experts filter the 2,200 PE-rewritten prompts down to 286 (an 87% rejection rate) based on criteria including "diversity, structural complexity, and logical coherence," filtering for "redundant and homogeneous cases, unnatural cross-shot transitions, and LLM hallucinations" (Section 3.2, Stage 3). The paper states that "each prompt is reviewed by at least two experts, with disagreements escalated to a third senior expert and resolved by majority vote" (Appendix D.1).

The consequence. The benchmark is not fully reproducible by independent researchers for two distinct reasons. First, GPT-5.4's behavior may change over time (model updates, deprecation, API changes) — prompts generated by today's GPT-5.4 with the provided system templates (Appendix A.2.2) may differ in style, complexity, or error patterns from those generated during the paper's construction process. The paper releases the final 286 prompts and metadata, so the evaluation set itself is fixed and reproducible for evaluation, but the construction methodology — generating new prompts with comparable characteristics for benchmark expansion or domain adaptation — is not reproducible without access to the specific GPT-5.4 checkpoint used. Second, the expert curation step introduces subjective judgments that cannot be replicated algorithmically. The filtering criteria (what constitutes "unnatural cross-shot transitions"? at what threshold of similarity are two prompts "redundant"?) are underspecified, and different expert panels might retain different subsets of the 2,200 candidate prompts, yielding a benchmark with different difficulty characteristics and genre balance. The 87% rejection rate amplifies this concern: the retained 13% of prompts represent a highly filtered subset whose properties may reflect expert preferences as much as objective quality criteria.

What evidence exists in the paper. The paper provides the full system templates for both GPT-5.4 stages (Appendix A.2.2), the expert taxonomy (144 sub-categories, Appendix A.2.1), and the annotation protocol (two-expert review with third-expert escalation, Appendix D.1). It releases the final benchmark data. However, it provides no inter-annotator agreement metrics for the curation process — we do not know whether different expert pairs would have made the same filtering decisions, or whether the retained prompts are systematically different from the rejected ones on measurable dimensions (prompt length, shot count, vocabulary diversity, cinematic language complexity).

Mitigation status. The paper partially addresses reproducibility by releasing the benchmark prompts and evaluation code (stated in the abstract and Section 5: "We will release the benchmark data and evaluation code to facilitate future research"). This makes the benchmark usable for evaluation without requiring reproduction of the construction pipeline. However, the paper does not provide the 1,914 rejected prompts, making it impossible to analyze what was filtered out and whether the surviving prompts are representative. It also does not discuss the reproducibility of the construction methodology for researchers who want to create comparable benchmarks for other domains or languages — the templates are provided, but the GPT-5.4 dependency and subjective curation step mean the methodology is a recipe, not an algorithm.

---

Limitation 4: Multilingual and Low-Resource Genre Coverage Is Insufficient for Statistically Reliable Per-Category Evaluation

The assumption or constraint. The benchmark claims to support "multilingual evaluation of audio-visual alignment" (Section 3.3) with spoken content distributed across 6 languages — Chinese 165 prompts, English 64, Japanese 15, Korean 15, Spanish 14, French 13 — and "diverse generation categories" with 8 video genres spanning Action, Narrative, Tutorial, Singing & Music Performance, Multi-person Dialogue, Science/Game, Advertising, and Nature. However, the prompt counts per language (beyond Chinese and English) and per low-resource genre are extremely small. Japanese: 15 prompts. Korean: 15 prompts. Spanish: 14 prompts. French: 13 prompts. Nature & extreme weather: 2.8% of 286 prompts ≈ 8 prompts. Science/Game: 8.4% ≈ 24 prompts. Advertising: 8.4% ≈ 24 prompts (Section 3.3 and data analysis in Section 3.3/Appendix A.3).

The consequence. With only 13–15 prompts per non-English/non-Chinese language, the per-language WER, lip-sync, and sound attribution scores are estimated from samples too small to support statistically meaningful comparisons. If Model A achieves WER 0.5 on 13 French prompts and Model B achieves WER 0.8, the difference may be driven by one or two outlier prompts rather than systematic capability differences. The paper reports only aggregate Overall scores (Table 2) and coarse difficulty breakdowns (Tables 3, 4) — it does not report per-language or per-genre performance breakdowns. This means a user cannot assess whether a model's strong Overall score reflects genuine multilingual capability or simply strong performance on the 229 Chinese+English prompts that dominate the benchmark. A model could perform catastrophically on all 57 non-Chinese/non-English prompts and still achieve a competitive Overall score, because those prompts constitute only 19.9% of the benchmark.

Similarly, the Nature genre (8 prompts) and the retro film visual style (0.3% of prompts ≈ 1 prompt) provide essentially no statistical power for genre- or style-specific evaluation. The claim that the benchmark covers "diverse generation categories" (Figure 2A) is true at the level of inclusion but misleading at the level of evaluation reliability — the benchmark contains prompts from low-resource categories but cannot reliably evaluate performance on them. A user interested specifically in Nature documentary generation or retro-film stylization would learn essentially nothing from MSAVBench scores.

What evidence exists in the paper. The data distributions are reported transparently in Section 3.3 and Appendix A.3 — the paper does not hide the prompt counts. What is missing is any acknowledgment that these sample sizes are insufficient for reliable per-category inference, or any reporting of per-category confidence intervals that would reveal the uncertainty. The paper includes bar charts of audio category distributions (Figure 2C) and language distributions (Figure 2C inset) that visually convey diversity but not statistical power.

Mitigation status. The paper does not address this as a limitation. The transparent reporting of distributions allows users to assess statistical power for themselves, but the paper's framing — "diverse audio generation conditions" (Figure 2C caption), "multilingual evaluation" (Section 3.3) — implies evaluation capability that the sample sizes do not support for low-resource categories. The paper could mitigate this in future work by (a) expanding the benchmark with additional prompts for underrepresented languages and genres, (b) reporting confidence intervals on per-category scores to quantify uncertainty, or (c) explicitly noting which per-category comparisons are statistically reliable and which are not.

---

Limitation 5: The "Unified Architectures Are Essential" Claim Confounds Joint Training with Visual Conditioning and Is Based on a Single Dubbing Model

The assumption or constraint. Finding 4 (Section 4.2) and the associated discussion in Section 5 make a strong prescriptive claim: "the alternative 'video-first, post-hoc dubbing' paradigm is inadequate for complex multi-shot audio-video generation... Conversely, unified architectures are essential for the MSAV task." This claim is based on the empirical observation that three open-source pipelines using HunyuanFoley for post-hoc dubbing — ShotStream + HunyuanFoley (58.85 Overall, WER 1.00), LongLive + HunyuanFoley (58.59, WER 7.55), and Helios + HunyuanFoley (54.10, WER 1.24) — perform substantially worse than native joint-generation pipelines, particularly on word error rate and lip synchronization.

The consequence. The conclusion conflates two distinct hypotheses that the experimental design does not disentangle:

• Hypothesis A (joint training matters): Audio and video must be generated by a single model trained jointly on both modalities, because joint training enables cross-modal attention and fine-grained alignment that post-hoc approaches cannot achieve.

• Hypothesis B (visual conditioning at generation time matters): Audio generation must have access to visual features — frame-level conditioning, not just a completed video output — but this conditioning could be provided by a separately trained audio model that takes visual features as input, without requiring joint end-to-end training.

The evidence in Table 2 is consistent with Hypothesis B but does not distinguish it from Hypothesis A. The best-performing open-source pipeline — LTX-2.3 in TI2AV mode (72.63, WER 0.49) — generates audio and video jointly per-shot (native single-shot AV model), but it is not a unified multi-shot audio-video architecture in the sense of joint training across modalities. It is a modular pipeline: (1) Wan2.7-Image generates per-shot keyframes from the prompt, (2) LTX-2.3 generates audio and video jointly for each shot conditioned on the keyframe, (3) shots are concatenated. This pipeline achieves competitive WER and Overall scores without joint training across modalities at the architecture level — it achieves cross-modal alignment through per-shot joint generation rather than joint training.

Furthermore, all three failing dubbing pipelines use the same dubbing model (HunyuanFoley). It is possible — and the experiments do not rule out — that HunyuanFoley is simply a weak dubbing model, and that a different, more capable dubbing model (trained with more data, a better architecture, or explicit visual conditioning) would substantially close the gap. The paper's claim that the paradigm is inadequate, rather than this specific instantiation of it, is not supported by negative results from a single dubbing model.

What evidence exists in the paper. Table 2 shows the performance gap, and Section 4.2, Finding 4 attributes it to "lack of frame-level semantic grounding across hard camera cuts, disrupting joint cross-modal alignment." The TI2AV pipelines (LTX-2.3 TI2AV, MoVA TI2AV) serve as positive examples of per-shot joint generation but are not designed to test the joint-training vs. visual-conditioning hypotheses. No alternative dubbing model is tested. No ablation varying the dubbing model while holding the video generation pipeline constant is performed (which would directly test whether the failure is model-specific or paradigm-specific).

Mitigation status. The paper does not acknowledge the confound. The claim is stated as a definitive finding rather than a hypothesis requiring further validation. A fairer characterization would be: "the specific post-hoc dubbing pipeline using HunyuanFoley is inadequate, and per-shot joint audio-video generation (whether through native joint models or image-conditioned pipelines) substantially outperforms it — future work should determine whether improved dubbing models with explicit visual conditioning can close this gap, or whether joint audio-video architectures are indeed necessary." The paper's prescriptive claim ("unified architectures are essential") may be correct, but the experiments do not provide sufficient evidence to establish it as a necessity rather than a strong empirical trend with the currently available models.

---

Limitation 6: Human Alignment Validation Uses Expert Annotators and a Small Sample, with No Demonstration That Rankings Generalize to Non-Expert Users

The assumption or constraint. The human alignment study validating MSAVBench's metrics (Section 4.4, Appendix D.2) uses two groups of expert annotators: 30 full-time AIGC researchers for system-level evaluation (1,200 pairwise judgments across 16 models), and 10 full-time AIGC researchers and aesthetic-quality annotators for fine-grained evaluation (360 judgments per metric across 3 metrics). The annotators have "prior experience in aesthetic-quality annotation" and are described as holding graduate degrees in relevant fields. The paper reports a Spearman rank correlation of 0.915 between MSAVBench's Overall score and expert system-level rankings — a strong result that validates the benchmark against expert judgment.

The consequence. The Spearman correlation of 0.915 demonstrates that MSAVBench agrees well with expert judgments, but does not demonstrate agreement with the broader user population that MSAVBench is intended to serve — developers, researchers, and practitioners in the open-source community who may evaluate model outputs differently than domain experts. Expert annotators may be sensitive to technical artifacts (subtle lip-sync misalignment, pixel-level rendering errors, precise camera framing) that typical users do not notice, or may undervalue aesthetic qualities (creative shot composition, emotional expressiveness) that non-expert users prioritize. If MSAVBench's metrics are calibrated against expert preferences, the benchmark will rank models according to expert criteria — which is valuable for technical progress but may not align with what makes generated videos subjectively "good" to a broader audience. A model that scores poorly on MSAVBench due to imperfect camera parameter adherence (a technical criterion experts notice) might still produce videos that casual viewers find more engaging than a technically precise but aesthetically bland competitor.

Additionally, the sample sizes for fine-grained validation — 10 annotators producing 360 judgments per metric — are relatively small for establishing robust correlation coefficients. With only 360 pairwise judgments per metric, the standard error on the Spearman correlation could be substantial (roughly 1/√(n-1) ≈ 0.053 for n=360 under asymptotic approximation, though the exact standard error for Spearman's ρ is more complex). The reported differences between Qwen3.5 and Qwen2.5-VL backbones (0.030–0.045) and the differences between direct VLM scoring and the proposed method (0.250–0.381) are clearly outside this noise range for the larger improvements, but the backbone substitution differences (0.030–0.045) may not be statistically distinguishable from zero given the sample size. The paper does not report confidence intervals or statistical significance tests for any of the Spearman correlations.

What evidence exists in the paper. The expert demographics are reported in Appendix D.1–D.2. The annotation protocol — uniform sampling across genres, anonymized videos, random presentation order, tie handling — is well-designed for reducing bias within the expert pool. The sample sizes (1,200 system-level, 360 per fine-grained metric) are reasonable for establishing rank correlations but not for precise estimation. The paper does not compare expert rankings against non-expert rankings, nor does it discuss the generalizability of the alignment results.

Mitigation status. The paper does not acknowledge this as a limitation. Using expert annotators is explicitly framed as a strength — they have the domain knowledge to recognize subtle failure modes — and the paper does not discuss the tradeoff between expert sensitivity and general-user representativeness. The paper also does not report whether annotators agreed with each other (inter-annotator agreement, such as Krippendorff's alpha or Fleiss' kappa), which would indicate whether the "expert judgment" being correlated against is itself a stable, consensus-based construct or a noisy average of idiosyncratic preferences. Without inter-annotator agreement metrics, the Spearman correlation of 0.915 could partially reflect that MSAVBench and the expert panel share the same biases, rather than that both capture an objective quality construct. A future study with non-expert annotators, larger sample sizes, and inter-annotator agreement reporting would substantially strengthen the validation.

7. Implications and Future Directions

How This Work Changes the Landscape

This work shifts the field's conceptualization of MSAV evaluation from a post-hoc scorecard to a prerequisite diagnostic infrastructure. Prior to MSAVBench, the dominant (implicit) assumption was that evaluating multi-shot audio-video generation was "hard but solvable with existing tools" — simply combine a video quality benchmark with an audio quality benchmark, segment shots with TransNet V2, and call it done. The paper systematically dismantles this assumption by demonstrating that each component of this naïve pipeline fails in ways that compound: shot boundary detectors produce errors that corrupt all downstream per-shot metrics, direct VLM scoring produces judgments that correlate at only 0.405–0.600 with human experts on complex dimensions, and disjoint video-then-audio evaluation misses the cross-modal binding failures that are the actual bottleneck for MSAV systems.

The magnitude of this shift is a reframing, not a paradigm shift. The paper does not introduce new generation methods or new evaluation metrics in the traditional sense — it introduces a new evaluation architecture that is principled rather than ad-hoc. The key intellectual moves are: (1) decomposing evaluation into metric-specific scoring paradigms rather than applying a universal VLM judge, (2) making preprocessing (shot segmentation) itself evaluable and correctable within the evaluation loop rather than treating it as a fixed black box, and (3) designing the benchmark's data to cross-combine difficulty axes (realistic/non-realistic subjects × scenes, shot counts from 2–15, multiple cinematic language dimensions) so that failure modes are diagnosable rather than merely rankable. This reframing changes what the research community optimizes for: from "higher scores on whatever dimension the current benchmark measures" to "balanced capability across video, audio, shot structure, and reference fidelity, with explicit attention to failure modes at the intersections of these dimensions."

The work reconciles a latent contradiction in the evaluation landscape. Prior benchmarks implicitly disagreed on what matters: VBench and EvalCrafter treated video generation as a purely visual task; ViStoryBench and MSVBench added multi-shot structure but ignored audio; AVGen-Bench added audio but lost shot structure. A model developer optimizing against VBench would invest in visual fidelity and temporal consistency within single clips — capabilities that are largely irrelevant to the cross-shot consistency and audio-visual synchronization failures that MSAVBench exposes. The paper demonstrates that these are not merely different evaluation preferences; they are orthogonal capability dimensions that current models cannot satisfy simultaneously (Table 2: no model, open- or closed-source, excels at both visual consistency and audio-visual synchronization). This reconciles the apparent contradiction: prior benchmarks weren't "wrong" — they were measuring real capabilities — but they were measuring incomplete subsets of the full MSAV capability space, and models optimized against those subsets developed systematic blind spots in the dimensions they ignored.

The work redirects research attention in three specific ways. First, it makes verifier robustness (not search algorithm sophistication) the central challenge for evaluation: Table 6 shows that tool-grounding and rubric-based scoring improve Spearman correlation with humans by 0.250–0.381 over direct VLM scoring, implying that investment in better perception tools and structured evaluation protocols yields higher returns than investment in larger VLM judges. Second, it makes architectural choices in generation pipelines — specifically, the decision between per-shot joint audio-video generation versus post-hoc dubbing — a first-class research question by quantifying the performance gap (WER: 0.49 for TI2AV pipelines vs. 7.55 for post-hoc dubbing, Table 2). Third, it makes counterfactual composition and out-of-distribution generalization measurable by cross-combining realistic and non-realistic subjects and scenes as explicit difficulty axes (Table 4), shifting these from philosophical concerns to operational evaluation dimensions.

Research directions that become less attractive include: (1) developing ever-larger VLM judges for end-to-end video evaluation — the paper's quantitative evidence (Table 6) shows this approach is fundamentally limited for spatially precise and temporally structured tasks regardless of model scale; (2) building MSAV generation models evaluated only on visual quality or only on single-shot metrics — the benchmark makes such evaluation transparently incomplete; and (3) the "bigger model, same pipeline" approach to improving audio-visual synchronization — the finding that JavisGPT (a native joint model) achieves better lip-sync (0.42) than Seedance-2.0 (1.52, lower is better) despite being a much smaller open-source model suggests that architectural integration, not model scale, is the binding constraint on audio-visual alignment.

---

Follow-Up Research This Work Enables

Quantifying the cost-accuracy tradeoff in stratified evaluation. MSAVBench's evaluation framework uses three scoring paradigms with different computational costs — specialized expert models (cheapest per metric, 10 metrics), rubric-based VLM scoring (moderate cost, 5 metrics), and tool-grounded agentic scoring (most expensive, 5 metrics) — but provides no measurement of how much each paradigm costs or whether cheaper approximations could achieve comparable accuracy. A strong follow-up would: (a) measure the GPU-hours and wall-clock time for end-to-end evaluation of one model on all 286 prompts under the full framework; (b) ablate each paradigm individually — evaluate the same set of generated videos with expert models only, then add rubric scoring, then add tool-grounding — measuring both the incremental Spearman correlation improvement and the incremental computational cost; (c) test whether cheaper approximations (e.g., single-iteration instead of two-iteration self-correction, Qwen2.5-VL instead of Qwen3.5 for all VLM-based metrics, no tool invocation for "easy" prompts where direct VLM scoring may suffice) can recover most of the alignment improvement at a fraction of the cost. The paper's backbone robustness check (Table 6) suggests cost reduction is possible, but without cost baselines, the research community cannot make informed adoption decisions.

Validating the self-correction mechanism with ground-truth shot boundaries. The paper claims that iterative VLM-driven shot boundary correction "eliminates cascading failures caused by shot segmentation errors" (Section 3.4.2) but provides no evidence that the mechanism actually improves boundary accuracy or downstream metric reliability. A critical stress-test would: (a) collect human-annotated ground-truth shot boundaries for a subset of MSAVBench-generated videos (e.g., 50 videos from diverse models and shot counts); (b) measure TransNet V2's boundary detection error rate on these videos (precision, recall, F1 against human annotations); (c) measure the VLM correction loop's ability to improve these boundaries (how many errors are fixed vs. introduced?); (d) compute all 20 MSAVBench metrics using three boundary sets — TransNet V2 alone, TransNet V2 + self-correction, and human ground-truth — and measure how much the self-correction shifts metric scores toward the ground-truth values. This experiment would either validate the self-correction mechanism as a genuine robustness improvement or reveal that it is an expensive noise term (or worse, an error-introducing step) that should be removed or replaced. A negative result — that self-correction does not materially improve boundary quality — would be equally valuable, as it would redirect effort toward better base segmentation rather than iterative correction.

Disentangling joint training from visual conditioning in the post-hoc dubbing failure. The paper's Finding 4 claims that "unified architectures are essential for the MSAV task" based on the poor performance of three HunyuanFoley-based dubbing pipelines (Table 2). However, the LTX-2.3 TI2AV pipeline — which is modular, not jointly trained across audio and video — achieves competitive WER (0.49) through per-shot image conditioning. This confound makes it impossible to determine whether the dubbing failure is due to the paradigm (post-hoc audio generation) or the specific model (HunyuanFoley). A decisive experiment would: (a) test at least three different dubbing models (not just HunyuanFoley) paired with the same video generation pipeline (e.g., LongLive), measuring WER, lip-sync, and sound attribution for each; (b) test whether providing the dubbing model with explicit visual conditioning (frame-level features, per-shot keyframes) closes the gap with native joint generation — if a dubbing model with visual conditioning achieves WER comparable to LTX-2.3 TI2AV (0.49), the failure is paradigm-specific but fixable; if no dubbing model with any conditioning strategy achieves WER below, say, 2.0, the paradigm is genuinely inadequate. This experiment would replace the paper's current prescriptive claim ("unified architectures are essential") with an evidence-based characterization of when and why joint training matters — a finding with direct implications for the open-source community's architectural roadmap.

Per-difficulty, per-genre, and per-language diagnostic breakdowns of model failures. MSAVBench's overall scores (Table 2) and coarse difficulty breakdowns (Tables 3, 4) establish that models degrade on longer shot counts and non-realistic content, but the benchmark's diagnostic potential — its stated purpose — is largely untapped. A comprehensive diagnostic analysis would: (a) report per-genre overall scores for all 19 models across the 8 video genres (Action, Narrative, Tutorial, etc.), revealing whether certain genres are systematically harder (e.g., does Multi-person Dialogue degrade more severely with shot count than Nature?); (b) report per-language WER and lip-sync for the 6 languages, identifying whether models exhibit language-specific failure modes (e.g., do all models perform worse on Japanese than on English? Does the Chinese-heavy training distribution of most models create systematic disadvantages for other languages?); (c) cross-tabulate failure modes — for example, do models that fail on lip-sync for Multi-person Dialogue prompts also fail on speaker timbre consistency for the same prompts, or are these independent failure dimensions? This analysis would transform MSAVBench from a leaderboard into a diagnostic tool, providing specific guidance: "if you care about multilingual dialogue, Model X is strong on English but collapses on Korean, while Model Y is consistently mediocre across all languages."

Extending the stratified scoring paradigm to other multi-modal evaluation domains. The paper's central architectural innovation — decomposing evaluation into expert models for objective tasks, rubrics for subjective tasks, and tool-grounding for spatially precise tasks — is domain-agnostic. A natural extension would apply this paradigm to: (a) multi-scene code generation with execution traces — evaluate code correctness with unit tests (expert model), code style and readability with rubric-based VLM scoring, and variable binding across functions with tool-grounded static analysis; (b) multi-step embodied agent evaluation — evaluate task completion with environment simulators (expert model), plan coherence with rubric-based scoring, and spatial navigation accuracy with tool-grounded SLAM verification; (c) long-form document generation with figures and tables — evaluate factual accuracy with retrieval-based verification (expert model), narrative flow with rubric-based scoring, and figure-text alignment with tool-grounded layout analysis. For each domain, the key question is whether the stratified approach achieves the same magnitude of improvement over direct VLM scoring (0.250–0.381 Spearman correlation gain) that it achieved on MSAV evaluation. A domain where the improvement is negligible would clarify the boundary conditions: the stratified paradigm helps specifically when evaluation requires (a) spatial precision beyond VLM capabilities, or (b) structured decomposition of subjective judgments, and is unnecessary when VLMs are already reliable.

Stress-testing MSAVBench's coverage with adversarial prompt design. The benchmark's 286 prompts are curated for diversity and challenge, but as with any fixed evaluation set, models may eventually overfit — either through training data contamination or through architectural choices optimized specifically for MSAVBench's prompt distribution. A stress-test would: (a) use the paper's own prompt generation pipeline (GPT-5.4 + PE model, Appendix A.2.2) to generate a held-out set of, say, 500 new prompts following the same taxonomy but with different seed quadruples, without expert curation (to test whether the curation step is load-bearing or cosmetic); (b) evaluate all 19 models on both the curated 286-prompt set and the uncurated 500-prompt set; (c) measure whether model rankings are stable across the two sets — if rankings shift substantially, the expert curation step is introducing systematic bias (the curated prompts favor certain model characteristics over others); if rankings are stable, the curation step is filtering noise rather than signal, and the benchmark could be scaled up cheaply by skipping human review. Additionally, adversarial prompts designed to probe specific failure modes — prompts requiring exactly the capabilities that models are known to lack (e.g., "a conversation between four speakers in Japanese, Spanish, French, and Korean, with continuous background music that changes tempo at each shot transition") — would reveal whether current MSAVBench prompts are difficult enough to saturate model capabilities or whether substantially harder prompts would produce different model rankings.

---

Practical Applications and Downstream Use Cases

Guiding open-source MSAV architecture development with quantified capability gaps. The paper's most actionable output for open-source developers is the performance tier structure in Table 2, which provides a clear prioritization of architectural investments. Specifically: (a) the TI2AV paradigm (LTX-2.3 TI2AV at 72.63) achieves competitive performance with closed-source mid-tier models (Wan2.7-T2V at 72.26) using a modular pipeline — this tells developers that per-shot image conditioning can substitute for native multi-shot joint generation, and that effort should go into improving keyframe quality and per-shot audio-video synchronization rather than end-to-end multi-shot architectures; (b) the post-hoc dubbing failure (WER 7.55 for LongLive + HunyuanFoley vs. 0.49 for LTX-2.3 TI2AV) tells developers that investing in better dubbing models without visual conditioning is a dead end — effort should shift to either native joint generation or dubbing models with explicit frame-level visual features; (c) the camera parameter adherence anomaly — where open-source pipelines (Wan2.2 + HunyuanFoley at 0.957) can outperform closed-source (Seedance-2.0 at 0.801) on camera control while failing on layout-text alignment (0.430 vs. 0.822) — tells developers that camera execution and spatial layout are separable capabilities that require distinct architectural support. A developer building an open-source MSAV system can use these diagnostics to allocate effort: first fix layout-text alignment (the largest gap), then address cross-shot consistency, then optimize camera control (where open-source is already competitive).

Pre-deployment testing and failure mode cataloging for commercial MSAV APIs. For organizations evaluating which commercial MSAV API to integrate (Seedance-2.0, Wan2.7, Kling-V3, Sora-2, HappyHorse), MSAVBench provides a systematic pre-deployment evaluation protocol that goes far beyond marketing demos. Rather than testing a few hand-picked prompts, a team can run the full 286-prompt benchmark and obtain per-dimension scores that reveal each API's failure profile. For example, if the use case is multi-speaker dialogue videos (e.g., automated dubbing of meetings or interviews), the sound attribution (Attr.) and speaker timbre consistency (Spk.) columns in Table 2 are directly relevant — Sora-2 achieves only 0.520 on Spk. (worst among closed-source), suggesting it would be a poor choice despite its strong narrative coherence (0.852, best overall). If the use case is advertising with on-screen text, the OCR column is critical — Seedance-2.0 achieves 0.726 on text rendering accuracy, but Figure 4A qualitatively shows it still produces "unintended text" and misspellings, suggesting that no current system is reliable enough for production text-in-video applications. This diagnostic granularity — knowing not just "Model X is worse than Model Y" but "Model X fails on these three specific dimensions that matter for my use case" — is the operational value that MSAVBench provides over single-score leaderboards.

Cost-efficient data generation for MSAV model training through difficulty-stratified sampling. The benchmark's shot-count breakdown (Table 3) and realism breakdown (Table 4) reveal that model performance varies dramatically with prompt complexity — LongLive + HunyuanFoley drops from 66.10 (1–4 shots) to 41.60 (11–15 shots), a 24.5-point collapse. For a team training a new MSAV model, this suggests a curriculum learning strategy: generate training data primarily at moderate difficulty levels (4–10 shots, realistic scenes) where models are reasonably capable and can produce usable training signals, then gradually increase difficulty. Without this breakdown, a team might waste substantial compute generating 15-shot non-realistic training examples where the teacher model produces essentially random outputs (WER 7.55, near-zero quality) — data that adds noise rather than signal to the training process. The benchmark's four reality-combination tiers (real×real, real×unreal, unreal×real, unreal×unreal) provide a further stratification: if a model is being fine-tuned for stylized content generation (e.g., anime or cyberpunk), training data should emphasize the realistic-subject × non-realistic-scene combination (26.2% of the benchmark) and non-realistic × non-realistic (7.7%), since Table 4 shows these are where models degrade most severely relative to realistic baselines.

Enabling fair and reproducible comparison in MSAV research publications. A persistent challenge in generative model research is that papers report cherry-picked qualitative examples and incomparable quantitative metrics — one paper reports FVD, another reports CLIP similarity, a third reports a custom VLM score, and none are directly comparable. MSAVBench provides a standardized evaluation protocol (286 fixed prompts, 20 standardized metrics, open-source evaluation code) that could serve the same coordinating function for MSAV research that benchmarks like GLUE/SuperGLUE served for NLP or ImageNet served for computer vision: every paper reporting MSAV results would run the same evaluation and report the same metrics, making claims directly comparable and progress objectively measurable. The paper's human alignment validation (Spearman ρ_s = 0.915 for Overall score, Table 6) and backbone robustness check (minimal degradation with smaller VLM, Table 6) provide the credibility needed for community adoption — researchers can trust that MSAVBench rankings reflect genuine quality differences rather than artifacts of a specific large VLM. The release of evaluation code alongside the benchmark data (stated in the abstract) makes adoption a matter of running existing infrastructure rather than reimplementing metrics, lowering the barrier to entry.

---

When to Prefer This Method

The paper articulates clear conditions under which MSAVBench's evaluation framework should be preferred over existing evaluation approaches, grounded in the distinction between what is being evaluated and how reliably the evaluation captures human-perceived quality.

Prefer MSAVBench for evaluation when:

• Evaluating systems that produce multi-shot structured outputs with audio (2+ shots, any audio modality). Single-shot or silent-video benchmarks (VBench, EvalCrafter, Video-Bench) will miss cross-shot consistency failures, audio-visual synchronization errors, and cinematic language adherence — the capabilities that distinguish MSAV systems from basic T2V models. The paper's comparison (Table 1) shows that no prior benchmark covers multi-shot structure, audio evaluation, and audio-visual joint evaluation simultaneously.

• The evaluation requires reliable differentiation between models on the basis of fine-grained spatial, temporal, or audio-visual alignment. Direct VLM scoring — the default approach in many recent benchmarks — achieves Spearman correlations of only 0.405–0.600 with human judgment on complex dimensions (Table 6). MSAVBench's rubric-based and tool-grounded scoring achieves 0.767–0.850. When the goal is distinguishing between models that differ primarily on spatial layout, lip-sync quality, or speaker attribution rather than overall visual aesthetics, direct VLM scoring is insufficiently reliable.

• Diagnostic granularity matters more than a single aggregate score. MSAVBench reports 20 atomic metrics across four hierarchical levels, with per-dimension normalization enabling direct comparison of a model's strengths and weaknesses. When the use case requires knowing why Model A outperforms Model B — because it excels at speaker timbre consistency but fails at text rendering, or vice versa — MSAVBench's stratified design provides this breakdown, whereas single-score benchmarks obscure it.

Prefer simpler, cheaper evaluation approaches when:

• The system being evaluated generates only single-shot, silent video. In this setting, the overhead of MSAVBench's multi-shot structure, audio evaluation pipelines, and cinematic language metrics provides no benefit, and specialized single-shot benchmarks (VBench, EvalCrafter) are more cost-effective.

• Evaluation cost is severely constrained and approximate rankings suffice. The paper does not quantify MSAVBench's evaluation cost (see Limitation 6.1), but the framework invokes 10+ specialized expert models, VLM judges, and perception tools across 286 videos. If a developer needs a quick, approximate ranking of model checkpoints during training and can tolerate the 0.405–0.600 Spearman correlation of direct VLM scoring, a simpler pipeline may be pragmatically appropriate despite reduced reliability.

• The evaluation domain lacks structured metadata (per-shot cinematic language annotations, audio content categories, reference conditions) that MSAVBench's rubric-based and tool-grounded scoring requires. The framework's instance-wise rubrics are instantiated from prompt metadata — if the prompts being evaluated do not contain this metadata (e.g., user-generated prompts without per-shot specifications), the rubric-based paradigm cannot be applied, and the evaluation would need to fall back to the direct VLM scoring that the paper demonstrates is substantially less reliable.