Evaluation2026~133 minarXiv:2603.14145

MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos

score 9Arushi Goel, Sreyan Ghosh, Vatsal Agarwal +16NVIDIA↑ 14Mar 14, 2026cs.CL

🎯 Pitch

Even the best AI models score just 64% on this new benchmark for understanding real-world videos, revealing they often fail at basic tasks like spotting when events happen or connecting what they see with what they hear. The study of over 20 models shows accuracy crumbling as key information appears deeper into long, complex footage, flagging a critical blind spot in current multimodal systems.

1. Executive Summary

This paper introduces MMOU, a large-scale benchmark for systematically evaluating omni-modal understanding and reasoning in multimodal large language models (MLLMs) on long and complex real-world videos. The benchmark comprises 15,000 manually annotated multiple-choice question-answer pairs sourced from 9,038 web-collected videos averaging 711.6 seconds in duration, spanning 10 domains and 36 fine-grained subcategories, with each question engineered to require simultaneous integration of audio and visual evidence across 13 fundamental skill categories—including temporal understanding (reasoning about event ordering), needle-in-the-haystack reasoning (localizing specific moments in long videos), and audio-visual stitching (understanding narrative continuity across edited segments). Evaluating 20+ state-of-the-art open-source and proprietary models reveals that the best closed-source model achieves only 64.2% accuracy while the strongest open-source model reaches just 46.8%, establishing that current MLLMs possess substantial gaps in joint cross-modal reasoning over extended temporal horizons—even on skills considered fundamental—with performance degrading steadily as answer-relevant evidence appears later in videos.

2. Context and Motivation

The fundamental gap this paper addresses is deceptively simple to state but technically demanding to solve: we lack rigorous evaluation frameworks for assessing whether multimodal AI systems can jointly reason over audio and visual streams in long, real-world videos the way humans do. While multimodal large language models (MLLMs) have demonstrated impressive performance on isolated modality benchmarks—excelling at image understanding, audio transcription, or short video captioning—the paper argues that this progress masks a critical deficiency. Existing models process audio and visual signals largely independently, often failing to integrate them in the tightly coupled, temporally extended manner that characterizes human perception of the real world.

This gap is significant for several reasons the paper articulates throughout Sections 1–3:

Ecological validity of AI evaluation: Real-world videos—whether they are academic lectures, sports broadcasts, news segments, or DIY tutorials—inherently fuse audio and visual information. A cooking video shows and narrates the technique; a sports commentator describes action that must be matched to specific players on screen; an academic lecture presents slides synchronized with spoken explanation. An AI system that processes these modalities separately, or that succeeds only on short clips, fundamentally fails to capture the integrated understanding that real-world applications demand. The paper explicitly states in Section 1 that "most models are optimized for single-modality reasoning... and often fail to jointly perceive and reason across modalities in a manner analogous to human cognition."
Long-form understanding is qualitatively different: The paper's focus on long videos (averaging over 11 minutes) targets a capability that is not merely "short video understanding but more." Long videos introduce challenges absent from short clips: temporal dependencies spanning many minutes, the need to retain and retrieve information from earlier segments, distinguishing between multiple similar events that occur at different times, and reasoning about narrative arcs or thematic development. A model that correctly answers questions about a 30-second clip may catastrophically fail on a 10-minute video because its context retention degrades, its attention dilutes, or it cannot distinguish between repeated similar events (e.g., multiple similar actions in a sports match). The paper explicitly notes that "existing evaluations do not adequately reflect the challenges posed by long and complex real-world videos, where meaningful understanding requires tightly coupled reasoning over audio and visual streams across extended time horizons" (Section 1).
The modality gap creates brittle systems: When MLLMs are evaluated primarily on unimodal benchmarks (vision-only video QA, audio-only comprehension), developers lack signal about whether their models are learning genuine cross-modal integration or simply excelling at one modality while ignoring the other. The paper demonstrates this concretely through its modality-restricted baselines (Table 3): the vision-only Qwen3-VL-32B achieves only 44.0% on MMOU, while the audio-only Audio Flamingo 3 achieves 17.7%. These are not errors that unimodal evaluation would catch—a model might score highly on vision benchmarks and audio benchmarks separately, yet fail completely when required to integrate both for a single question. The paper argues this represents a systematic blind spot in the current evaluation landscape.

Why This Problem Matters Now

The paper's motivation is grounded in a specific historical moment in MLLM development. Three converging trends make this gap particularly urgent:

First, MLLMs are approaching deployment in high-stakes settings. As these models move from research prototypes to production systems (Section 8 discusses applications in education, media analysis, and human-computer interaction), the consequences of brittle cross-modal reasoning multiply. An educational AI that misunderstands a lecture because it failed to connect spoken explanation to on-screen diagrams; a media analysis tool that misattributes quoted speech to the wrong speaker because it processed audio and visual tracks independently; an accessibility system that describes video content incorrectly because it missed the interaction between music and visual action—these failures matter in deployment but are invisible to existing benchmarks.

Second, training data imbalances are self-reinforcing. The paper identifies a specific mechanism by which the evaluation gap perpetuates itself (Section 1): "single-modality datasets are more abundant, higher quality, and cover a wider range of tasks than their multi-modal counterparts." Because evaluation drives research investment, and because existing benchmarks overwhelmingly reward unimodal performance, model developers optimize for unimodal capabilities. This creates a cycle where training data skews toward isolated modalities, leading to models that excel at isolated modalities, which further reinforces the perceived adequacy of unimodal evaluation. Breaking this cycle requires benchmarks that explicitly demand cross-modal integration, making the capability gap visible and incentivizing its closure.

Third, video is becoming the dominant medium for information. The domains covered in MMOU—academic lectures, news, sports, DIY tutorials, travel guides, video games—reflect the reality that an increasing fraction of human knowledge is transmitted through video rather than text alone. A general-purpose AI assistant in 2026 cannot be effectively evaluated solely on text-based reasoning benchmarks; it must demonstrate the ability to learn from, reason about, and answer questions about video content as humans do, integrating everything they see and hear.

Where Existing Benchmarks Fall Short

The paper's comprehensive comparison in Table 1 and Section 3.3 systematically catalogs specific limitations of prior evaluation frameworks. Rather than merely claiming MMOU is "better," the paper identifies precise dimensions along which existing benchmarks are inadequate:

The modality problem: Most benchmarks ignore one modality entirely. Video understanding benchmarks such as Video-MME (Fu et al., 2024), VideoMMMU (Hu et al., 2025), LongVideoBench, and MVBench (Li et al., 2024b) treat video as a purely visual medium. They either strip audio from videos or ignore the audio track during question design. This means questions can be answered from visual evidence alone, making them tests of visual temporal reasoning rather than genuine audio-visual integration. The paper argues this systematically overestimates model capability: a model that succeeds on Video-MME may be completely unable to answer questions requiring it to connect spoken words to visual evidence.

The duration problem: Benchmarks focus on short clips. As Table 1 documents, the average video duration in prior benchmarks is dramatically shorter than MMOU's 711.6 seconds: OmniBench averages 9.17 seconds, AV-Odyssey 15.58 seconds, UNO-Bench 27.1 seconds, DailyOmni 43.7 seconds, and WorldSense 141.1 seconds. Even the longest prior benchmark, Video-MME, averages 1017.9 seconds but (crucially) evaluates only visual understanding. The paper's point is not simply that longer videos are harder, but that they introduce qualitatively different challenges: the need to identify which segments of a long video are relevant to a question, to track entities and events across temporal gaps, and to maintain coherent reasoning chains over extended contexts. Models that perform well on 30-second clips may rely on strategies (e.g., attending evenly to all frames) that fail catastrophically at 10 minutes.

The cross-modality rigor problem: Prior audio-visual benchmarks are insufficiently demanding. Even benchmarks that include both audio and visual inputs often fail to ensure that questions cannot be solved from a single modality. The paper cites Chen et al. (2025) in reporting that existing benchmarks fall short of requiring genuine cross-modal reasoning, and validates this claim by showing that their own random 20% audit of MMOU achieves 100% audio-visual dependency—meaning removing either modality makes the question unanswerable. This is a concrete, measurable standard that prior work has not met. The paper also provides direct evidence: modality-specific models (vision-only Qwen3-VL-32B at 44.0%, audio-only Audio Flamingo 3 at 17.7%) perform far below the best audio-visual model (Gemini 2.5 Pro at 64.2%), confirming that MMOU questions truly require both modalities.

The scale and diversity problem: Existing omni-modal benchmarks are too small. Table 1 reveals stark differences in scale: OmniBench has only 1,142 QA pairs from 1,142 videos; UNO-Bench has 1,250 QA pairs; WorldSense has 3,172; OmniVideoBench has 1,000. MMOU's 15,000 QA pairs from 9,038 videos represents an order-of-magnitude increase. This matters for statistical reliability (differences between models are more likely to be genuine when measured over thousands of questions) and for coverage of the skill taxonomy and domain distribution. Small benchmarks cannot systematically sample the space of possible audio-visual reasoning challenges; MMOU's scale enables per-skill, per-domain, and per-duration analyses that smaller benchmarks preclude.

The task diversity problem: Skill coverage is narrow. The paper's taxonomy of 13 skill categories (Tables 7–8) is deliberately broad, encompassing temporal understanding, sub-scene reasoning, holistic video comprehension, inference, needle-in-the-haystack retrieval, referential grounding, counting, comparative reasoning, object interaction understanding, audio-visual stitching, sequential event ordering, spurious correlation tracking, and context understanding. Prior benchmarks typically cover far fewer dimensions. For example, DailyOmni covers only 6 skills; OmniBench covers 8. The paper's point is that "audio-visual understanding" is not a monolithic capability but a collection of distinct cognitive skills, and a benchmark that collapses them into a single score cannot diagnose where models fail. MMOU's skill tagging (averaging 2.71 skills per question, per Table 2) enables the kind of skill-level diagnostic analysis shown in Figure 5.

The answer quality problem: Distractors in multiple-choice benchmarks are often weak. The paper devotes careful attention to distractor construction (Section 3.4, step 5), using GPT-5.2 to generate nine hard distractors per question, with half designed to be semantically plausible and grounded in video context and the other half intentionally out-of-context. This balanced design prevents models from succeeding through superficial elimination strategies. The paper further increases difficulty by using "None of the above" in 13% of questions as the correct answer and in another 13% as an incorrect distractor—following Tam et al. (2025)—which forces models to genuinely evaluate all options rather than assuming one must be correct.

How This Paper Positions Itself

The paper's positioning relative to prior work is multifaceted and explicit in Section 3.3 and Table 1:

It is not proposing a new model or training method. MMOU is purely an evaluation benchmark. This distinguishes it from papers that introduce new multimodal architectures or training procedures. The contribution is diagnostic: reveal the gap, characterize its dimensions, and provide a tool for the community to measure progress. The paper states in Section 7 that future work includes "developing more robust evaluation protocols" and "continuously expanding the benchmark"—framing itself as infrastructure for a research program rather than a one-time result.

It deliberately targets a harder evaluation regime than prior work. The paper's design choices—long videos, mandatory cross-modal dependency, hard distractors, diverse question positioning across the temporal span—all push in the direction of making the benchmark maximally challenging for current models while remaining solvable by humans (who achieve 84.3% accuracy, per Table 3). This reflects a philosophy that benchmarks should be ahead of current capability, providing headroom for years of progress, rather than saturating shortly after release.

It bridges unimodal benchmarking with genuine omni-modal evaluation. By evaluating vision-only, audio-only, and audio-visual models on the same questions (Table 3), the paper quantifies the marginal value of cross-modal integration in a way that prior benchmarks could not. The fact that Qwen3-VL-32B (vision-only) achieves 44.0% while Qwen3-Omni-30B-A3B-Instruct (audio-visual) achieves 46.0%—only slightly better—suggests that even models designed for multimodal input are not effectively integrating the additional audio information. This is a finding that only a benchmark designed with modality-controlled baselines could surface.

It connects a specific capability gap to a specific evaluation deficiency. The paper's thesis is not merely that models perform poorly on MMOU, but that the poor performance reveals capabilities that existing benchmarks systematically fail to test. The gap between 64.2% (best model) and 84.3% (human) is not just a number—it reflects specific failures in temporal reasoning across long horizons (Figure 6), in counting events that require tracking across modalities, in needle-in-the-haystack retrieval from extended videos, and in integrating audio cues with visual context. The paper's skill-level and temporal-position analyses connect low accuracy to why accuracy is low, providing actionable guidance for model developers.

It serves as a reality check on claims of multimodal capability. The paper notes that strong performance on established benchmarks (MMMU, Video-MME, MMAU) does not transfer to MMOU: "even the widely used Qwen3-Omni-30B-A3B-Thinking model reaches only 19.4% accuracy, markedly lower than its performance on existing benchmarks" (Section 3.3). This gap between benchmark-reported capability and MMOU-measured capability exposes a systematic overestimation of current model abilities by the existing evaluation ecosystem. The paper positions MMOU as a necessary corrective—a benchmark that measures what other benchmarks miss.

3. Technical Approach

3.1 Reader Orientation

This paper presents MMOU, a manually constructed evaluation benchmark—not a model or training method—designed to measure how well multimodal AI systems can jointly reason over audio and visual information in long, complex real-world videos. The core problem it solves is that existing benchmarks fail to test genuine cross-modal integration over extended temporal horizons, creating a systematic blind spot where models can score highly on unimodal or short-video tests while remaining fundamentally unable to perform the kind of integrated audio-visual understanding that humans do effortlessly. The "shape" of the solution is a carefully engineered dataset of 15,000 multiple-choice questions, each constructed through a multi-stage expert annotation pipeline to ensure mandatory cross-modal dependency, calibrated difficulty, and coverage across 13 distinct reasoning skills and 10 real-world video domains.

3.2 Big-Picture Architecture (Diagram in Words)

The MMOU benchmark is produced through a seven-stage pipeline that transforms raw web-collected videos into calibrated, quality-controlled evaluation instances. The major components and their responsibilities are:

Skill and Task Taxonomy (the evaluation ontology): A predefined set of 13 fundamental audio-visual reasoning skills—temporal understanding, sub-scene understanding, holistic video reasoning, inference, needle-in-the-haystack reasoning, referential grounding, counting, comparative reasoning, object interaction reasoning, audio-visual stitching, sequential event ordering, spurious correlation tracking, and context understanding—that defines what the benchmark measures. Every question must map to one or more of these skills, ensuring systematic diagnostic coverage.
Video Domain Selection (the content strategy): A curated taxonomy of 10 major categories and 36 fine-grained subcategories (e.g., Academic Lectures → STEM, Medicine, Law, Business; Sports → Training & Preparation, Analytics, Combat Events) that ensures broad real-world coverage and exercises different combinations of the targeted skills.
Source Video Collection (the raw material): 9,038 web-collected videos from publicly available platforms, with durations ranging from 7 seconds to 121 minutes (average 711.6 seconds), selected to align with the skill taxonomy so that each video supports construction of at least one high-quality cross-modal question.
Expert Question Generation (the core annotation): Eleven professionally trained annotators follow a standardized protocol: watch each video in full, then author open-ended question-answer pairs that require simultaneous audio and visual understanding. Each question is tagged with applicable skill categories and annotated with the earliest and latest timestamps where supporting evidence appears.
Distractor Generation (the difficulty calibration): Each open-ended question is converted to multiple-choice format by generating nine hard distractor options using GPT-5.2, with half designed to be semantically plausible and half intentionally out-of-context. Additionally, 13% of questions use "None of the above" as the correct answer, and another 13% include it as an incorrect distractor.
Quality Control and Filtering (the gate): A separate group of expert reviewers audits the questions, removing ambiguous, redundant, or trivial instances, as well as those with weak audio-visual grounding or misaligned timestamps. Only questions requiring strict joint audio-visual reasoning are retained.
MMOU Finalization (the output): The final benchmark of 15,000 curated QA pairs, ready for model evaluation.

Information flows sequentially through this pipeline: skill taxonomy → domain selection → video collection → question authoring → distractor generation → quality filtering → finalized benchmark. Each stage constrains the next, ensuring that every question in the final dataset inherits the design properties of mandatory cross-modality, skill coverage, domain diversity, and calibrated difficulty.

3.3 Roadmap for the Deep Dive

I will explain the technical approach in the following order, which mirrors the construction pipeline and builds understanding from design principles to concrete implementation:

First, the skill taxonomy (Section 3.4.1): because it is the conceptual foundation that defines what MMOU measures. Understanding the 13 skill categories—their definitions, their differences, and why they were chosen—is prerequisite to understanding why the rest of the pipeline is structured as it is.
Second, video domain selection and collection (Section 3.4.2): because the choice of domains directly operationalizes the skill taxonomy. I will explain the 10 major categories and 36 subcategories, the selection criteria, and the properties of the collected videos (durations, resolution, source).
Third, the expert annotation protocol (Section 3.4.3): the human-in-the-loop process that converts raw videos into question-answer pairs. This is the most resource-intensive and quality-critical stage. I will cover annotator qualifications, the step-by-step annotation workflow, the mandatory cross-modality constraint, timestamp annotation, and skill tagging.
Fourth, distractor generation and difficulty calibration (Section 3.4.4): the automated stage that converts open-ended questions to multiple-choice format. I will explain the GPT-5.2 prompting strategy, the balanced hard-distractor design, the "None of the above" manipulation, and the rationale for each choice.
Fifth, quality control and filtering (Section 3.4.5): the verification stage that ensures the benchmark meets its design specifications. I will cover the audit axes, the rejection criteria, and the final retention process.
Sixth, dataset statistics and properties (Section 3.4.6): the quantitative summary of the resulting benchmark—video durations, question lengths, answer option distributions, temporal answer positions, skill co-occurrence patterns—and why these statistics matter for evaluation validity.

This order is chosen because each stage's design choices are constrained by the outputs of prior stages. Starting with the skill taxonomy establishes what is being measured; the domain selection shows where it is being measured; the annotation protocol shows how questions are authored to measure it; the distractor generation shows how difficulty is calibrated; quality control shows how validity is enforced; and the statistics show what the final measurement instrument looks like.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset and benchmark construction paper whose core idea is that evaluating omni-modal understanding in long real-world videos requires a carefully engineered evaluation instrument with mandatory cross-modal dependency, systematic skill coverage, calibrated difficulty, and rigorous quality control—all of which are instantiated through a multi-stage expert-in-the-loop annotation pipeline.

3.4.1 Skill and Task Taxonomy: The Evaluation Ontology

The foundation of MMOU is a taxonomy of 13 fundamental audio-visual reasoning skills, defined in Tables 7 and 8 of the paper. These skills are not arbitrary categories; they are designed to capture the distinct cognitive operations that joint audio-visual understanding demands, and they serve as the annotation ontology that guides every subsequent stage of the pipeline.

Why a taxonomy rather than a single "understanding" score? The paper's design philosophy, articulated implicitly throughout Section 3 and explicitly in the skill-wise analysis (Section 6, Figure 5), is that "audio-visual understanding" is not monolithic. A model might excel at identifying objects interacting in a scene while failing completely at counting how many times a specific audio-visual event occurs. A single accuracy number collapses these distinct capabilities, making it impossible to diagnose where models fail. By tagging each question with the specific skills it requires (averaging 2.71 skills per question, per Table 2), MMOU enables the kind of per-skill performance breakdown shown in Figure 5, where it becomes immediately visible that all models—including the best closed-source system—struggle with counting and temporal relationship identification.

The 13 skill definitions. Each skill is defined with sufficient precision that annotators can reliably assign skill tags and that question authors can deliberately target specific skills. The skills, drawn from Tables 7–8, are:

Temporal Understanding: Questions about the order of particular audio-visual events. This requires the model to track event sequences across time, not merely recognize individual events. Example from Table 8: "What does the man in the blue shirt and orange pants do right after a person on the radio is heard saying, 'so, my wife and I...'?" The model must locate the audio event ("so, my wife and I..."), identify the immediately subsequent visual event, and connect them.
Sequential Event Ordering: A more structured variant of temporal understanding that asks the model to order multiple key events. The example in Table 8 requires ordering four cricket commentary and play events in correct sequence. This tests whether the model can maintain a temporal ordering of multiple events rather than just identifying a single before/after relationship.
Sub-scene Understanding: Questions about a semantically important segment of a long video that requires understanding the surrounding context. The paper's example (Table 8) asks about a strange scene where potato chips float through the air into someone's mouth—a moment that makes sense only in the context of the preceding narration ("They begin to employ their abilities to do something edgy").
Holistic Video Reasoning: Questions requiring integration of information across the entire video timeline to understand the main activity, objective, or theme. The example from Table 7 asks about the purpose of overlaying eerie music over movie clips—requiring the model to synthesize audio mood and visual content across multiple segments to infer the intended effect (building tension for a horror-mystery film).
Inference and Context Understanding: Questions requiring deduction of unstated intentions, causes, or situational context from multiple audio-visual cues. Table 8's example asks why a blonde man is shocked when saying "it's undone now"—the answer requires inferring that he believes a ghost wrote a message, which is never explicitly stated but must be deduced from audio-visual context.
Needle-in-the-Haystack Reasoning: Questions about a specific moment in a long video. This tests the model's ability to localize and retrieve information from extended temporal contexts—a capability that Figure 6 shows degrades sharply as evidence appears later in videos. The example from Table 7 asks about how a character sounds when reading jokes from a phone.
Referential Grounding: Questions requiring linking between audio references and visual entities, or vice versa. The example from Table 8 asks about the purpose of a man in a white shirt and gold tie who appears in some shots but not others—the answer requires connecting his visual presence to his role as a translator for non-English speakers.
Counting and Comparative Reasoning: Quantitative reasoning over repeated audio-visual events. The example from Table 7 asks how many shadows are visible on a skate rink when a horn starts blaring—requiring the model to count visual entities at a specific audio-cued moment.
Object Interaction Reasoning: Questions about actions performed on objects and their resulting transformations. The example from Table 7 asks how and why the pitch of a running saw changes—requiring understanding that the visual action (shaving wood) causes the audio change (pitch shift).
Audio-Visual Stitching: Questions about how separate clips or segments are combined to create narrative continuity. The example from Table 7 asks how splicing a clip of a ride cart going through water relates to a voiceover—requiring understanding of editing intent across discontinuous segments.
Tracking Spurious Correlations: Questions about surprising or unintuitive details that cannot be inferred from language priors alone. The example from Table 7 asks about an unexpected event when "Jump Around" begins playing—a woman colliding with a ladder on the beat. This tests whether models rely on statistical language patterns rather than actual audio-visual perception, since a text-only model would never predict this specific coincidence.
Comparative Reasoning: Questions about key differences or similarities between two distinct audio-visual segments. The example from Table 7 asks how a character's motion changes when he speaks a specific line—requiring comparison of pre-utterance and post-utterance visual states.
Context Understanding: Questions about broader setting, background elements, or situational context emerging only from integrated audio-visual perception. The example from Table 8 asks what is heard in the background as a woman smiles before credits roll, and why—requiring the model to recognize audience applause and connect it to her completed concert performance.

The mandatory audio-visual understanding tag. In addition to the 13 skill tags, every question is tagged with an overarching "audio-visual understanding" label (Section 3.4, step 1). This is not an additional skill but a constraint: it means the question cannot be answered from either modality alone. The paper enforces this by explicitly excluding questions solvable from a single modality during the filtering stage (Section 3.4, step 6) and validates it through manual audit (20% sample achieving 100% audio-visual dependency, per Section 3.3).

Skill co-occurrence. The paper reports that questions average 2.71 skill tags (Table 2), and Figure 3(b) shows the co-occurrence matrix of skill pairs. This reveals that real-world audio-visual reasoning is inherently multi-skill: a question about a specific moment in a long video (needle-in-the-haystack) may simultaneously require temporal understanding (locating the moment in sequence), referential grounding (connecting spoken references to visual entities), and inference (deducing unstated implications). The co-occurrence matrix enables analysis of which skill combinations are most challenging and whether models exhibit correlated failures across related skills.

Design rationale summary. The taxonomy serves four functions simultaneously: (1) it guides annotators in question construction by specifying what types of reasoning to target; (2) it enables diagnostic evaluation by allowing per-skill accuracy breakdowns; (3) it ensures coverage by requiring that the benchmark spans all 13 categories; and (4) it prevents trivial questions by defining skills that inherently demand cross-modal integration. The alternative—an unstructured collection of "video questions" without skill tagging—would produce a benchmark that could measure overall performance but could not explain why performance was low or which capabilities needed improvement.

3.4.2 Video Domain Selection and Collection

With the skill taxonomy established, the next stage selects the actual video content that will serve as the raw material for question construction. This stage must satisfy two competing constraints: the videos must be diverse enough to exercise all 13 skills across varied contexts, yet coherent enough that annotators can consistently produce high-quality questions.

The domain taxonomy: 10 categories, 36 subcategories. The paper defines the following structure, detailed in Table 5:

Academic Lectures: STEM (701 questions), Medicine (133), Law (60), Business (50). These videos are chosen because they feature tightly synchronized audio-visual content—a lecturer speaking while presenting slides, demonstrating experiments, or annotating diagrams. The separation between spoken explanation and visual illustration makes them ideal for testing referential grounding (does the model connect what is said to what is shown?) and holistic reasoning (does the model integrate information across the lecture?).
Sports: Training & Preparation (2,770 questions—the single largest subcategory), Analytics (288), Combat Events (239), Racket Sports (99), Press Conference (88). Sports videos are chosen because they feature rapid, repeated events with strong audio-visual coupling (a commentator describes an action as it occurs on screen). This makes them natural tests for temporal understanding (ordering events), counting (how many times did X happen?), and needle-in-the-haystack reasoning (finding a specific play in a long match).
News: World News (4,221 questions—the largest category overall). News videos combine spoken reporting with visual footage, often with complex relationships between audio narration and visual content (the audio may describe events not shown on screen, or the visual may contradict or contextualize the audio). This tests inference and context understanding.
Daily Life: DIY (1,941 questions), Crafts (1,011), Gaming (233), Home Repair (52). These procedural videos require tracking object interactions and transformations over time, making them natural tests for object interaction reasoning and sequential event ordering.
Music: Live Performance (620), Themed Narrative & Parody (10). Music videos require integrating auditory content (music, lyrics) with visual performance, testing audio-visual stitching and holistic reasoning about artistic intent.
Film: Plot Recaps (249), Character & Theme Focus (126), Trailers & Promos (90). Film content tests narrative understanding across edited sequences, inference about character motivation, and audio-visual stitching (why was this clip placed after that one?).
Pranks: Physical Mishap (214), Stunt Pranks (152), Reveal Pranks (68), Animal Pranks (46), Jump Scare (38). Prank videos are deliberately chosen because they create spurious correlations—the audio and visual setup leads to an unexpected outcome that a text-only model could not predict. This directly tests the spurious correlations skill.
Travel: Destination Guides (212), Vlogs (195), Travel Tips & Etiquette (186), Cultural Insights (108). Travel content tests contextual understanding (what cultural norms are being demonstrated?) and sub-scene reasoning (what is significant about this particular location?).
Animation: Comedy & Kids (139), Action & Fantasy (45). Animated content tests whether models can reason about non-photorealistic visuals while integrating audio cues, testing generalization beyond natural video distributions.
Video Games: Esports Tournaments (255), Competitive Match Commentary (118), Production & Overlays (101), Battle & Mechanics (87), Player Reactions (55). Gaming content features complex visual overlays (health bars, minimaps) alongside rapid audio commentary, testing whether models can track multiple information streams simultaneously.

Why this taxonomy structure? The two-level hierarchy (10 major categories → 36 subcategories) enables analyses at multiple granularities. Category-level results (Table 3) show broad patterns: models perform best on Academic Lectures (71.4% for Gemini 2.5 Pro) and Daily Life (68.1%), worst on Animation (58.2%) and Music (59.7%). Subcategory-level distributions (Table 5) reveal more: certain subcategories dominate their parent categories (STEM lectures vs. Law or Business; DIY vs. Home Repair; World News vs. everything else). This imbalance is deliberate—the paper prioritizes content richness (long videos with rich audio-visual coupling) over uniform distribution, and certain domains (news, sports, DIY) naturally produce more question-viable content than others (music parody, animation action).

Video properties. The collected videos have an average duration of 711.6 seconds, with a minimum of 7.0 seconds and a maximum of 7,255.0 seconds (approximately 121 minutes), all sampled at 720p resolution (Section 3.2). The duration distribution (Figure 3e) shows a long tail: while many videos cluster in the 5–20 minute range, a substantial fraction extend beyond 30 minutes, and some exceed 60 minutes. This distribution is not uniform because the paper deliberately includes very long videos to test the limits of context retention—the temporal position sensitivity analysis in Figure 6 requires answer evidence that appears deep into extended videos, which would be impossible if all videos were short.

Collection source and criteria. Videos are sourced from publicly available online platforms (e.g., YouTube), with priority given to "naturally occurring content over scripted or synthetic data" (Section 3.4, step 3). This choice reflects the paper's commitment to ecological validity: real-world videos feature realistic audio conditions (background noise, overlapping speech, varying recording quality), diverse visual scenes, and authentic temporal structure. The alternative—scripted or synthetic videos—would provide cleaner signals but would not reflect the conditions under which deployed MLLMs actually operate.

Alignment with skill taxonomy during collection. The paper states that videos are "selected to align with the curated skill taxonomy, ensuring that each video supports the construction of at least one high-quality question" (Section 3.4, step 3). This means the collection process is not random web scraping; it is a targeted curation where domain experts identify videos likely to support questions targeting specific skills. A cricket match is collected because it will support temporal ordering and counting questions; a prank video is collected because it will support spurious correlation tracking; an academic lecture is collected because it will support referential grounding and holistic reasoning.

3.4.3 Expert Question Generation: The Annotation Protocol

This is the most labor-intensive and quality-critical stage of the pipeline. Eleven expert annotators convert raw videos into open-ended question-answer pairs following a standardized protocol, with the outputs then converted to multiple-choice format through automated distractor generation.

Annotator qualifications. The paper provides detailed annotator background information in Appendix B. The eleven annotators were recruited for "strong backgrounds in creative and technical writing, linguistics, journalism, and analytically rigorous STEM disciplines." Their educational backgrounds span bachelor's and master's degrees in English, English Literature, Creative Writing (including MFA training), Linguistics and Communication, Audio Engineering and Acoustics, Applied and Computational Mathematics, Biochemistry with Computer Science, and Computational Applied Mathematics. This deliberate mixture of humanities and STEM expertise ensures that questions are both linguistically precise (well-formed, unambiguous) and analytically rigorous (logically sound, factually correct). The paper explicitly states that annotators were selected for "demonstrated critical thinking, creative problem-solving ability, and exceptional attention to detail."

The annotation workflow (Appendix C). The protocol consists of five explicit steps:

Step 1: Watch the video in full length. This is not optional or compressible. The annotator must experience the entire video before writing any questions, ensuring that questions can reference events across the full temporal span and that the annotator understands the overall context, theme, and structure. This requirement is particularly important for long videos where relevant information may be distributed across multiple segments.

Step 2: Create a Q&A pair about the video. Two mandatory constraints govern question construction:

All questions should be open-ended (no multiple choice or yes/no questions). The paper explicitly precludes yes/no questions because they reduce reasoning to binary classification and enable guessing strategies. Open-ended questions force the annotator to articulate a specific, substantive answer that the model must generate or select from among carefully constructed options.
All questions should assess both video and audio understanding simultaneously. This is the mandatory cross-modality constraint. The paper states that annotators must "explicitly avoid yes/no questions or questions answerable from text alone" (Section 3.4, step 4). A question like "What color is the car?" (visual only) or "What word does the speaker emphasize?" (audio only) would be rejected. A valid question might be: "What does the man in the blue shirt do immediately after the radio announcer says 'so, my wife and I...'?"—this requires locating an audio event, identifying the subsequent visual action by a specific visually-identified person, and connecting them temporally.

The annotators are encouraged to generate multiple diverse questions per video. The paper provides guidance on quantity: 2–3 questions for short videos (< 5 minutes), 3–5 for medium videos (5–10 minutes), and more than 5 for long videos (> 10 minutes). They are further "encouraged to generate multiple diverse questions per video, which are then filtered" (Section 3.4, step 4).

Step 3: Annotate the timestamps of the video segment where the answer can be located. For each question, the annotator records the earliest and latest timestamps at which supporting evidence appears. If the answer evidence appears in multiple locations, only the first occurrence is annotated. This temporal annotation serves multiple purposes: (a) it enables the temporal position sensitivity analysis in Figure 6, which shows that model accuracy degrades as answer evidence appears later in videos; (b) it provides a ground-truth temporal localization that could support future work on video grounding evaluation; and (c) it enables quality control by allowing reviewers to verify that the answer evidence actually exists at the claimed timestamps.

The paper reports that the average answer position is 302.28 seconds (Table 2), with the distribution illustrated in Figure 3(c)—answers are "spread across the entire video timeline." The paper explicitly states that annotators "are encouraged to diversify the same" (Section 3.4, step 4), referring to the temporal positions of answer evidence. This deliberate diversification ensures that the benchmark tests both early-video and late-video reasoning, rather than concentrating answers in the first few minutes where models would have an easier time.

Step 4: Select the task type of the question as listed in the reference table. Select all that apply. The annotator tags each question with one or more of the 13 skill categories from the predefined taxonomy. The "select all that apply" instruction is important: a single question may test multiple skills simultaneously, and the tagging should reflect this reality rather than forcing annotators to choose a single "best" category. The average of 2.71 skills per question (Table 2) confirms that annotators routinely assign multiple tags.

Step 5: Repeat steps 1–4 if additional questions can be generated. The paper provides general recommendations for question quantity per video length, as noted above, and states: "We encourage diverse and creative questions."

Post-annotation audit. After the first round of question-answer annotations, a separate group of 10 annotators audits 20% of the QA pairs (Appendix C). The audit evaluates four quality axes:

Whether the question is relevant to the video.
Whether the question is grammatically correct.
Whether the assigned task type (skill tag) is accurate.
Whether the provided answer is correct.

This audit serves as a quality gate: questions that fail any of these checks would be flagged for revision or removal. However, the paper does not report the pass/fail rate of this audit—an omission that leaves the reader uncertain about the rejection rate and the difficulty of producing questions that meet all criteria.

Why this annotation protocol design? Several design choices deserve explicit justification:

Why open-ended first, then convert to multiple-choice? The paper constructs questions as open-ended first to ensure that answers are substantive and well-defined before distractors are generated. If annotators wrote multiple-choice questions directly, they might unconsciously write questions that are answerable through elimination strategies or that have obviously wrong distractors. The open-ended-first approach forces the question and correct answer to stand on their own before distractors are introduced.
Why human annotators rather than LLM-generated questions? The paper uses expert human annotators for question generation (Section 3.4, step 4) while using GPT-5.2 only for distractor generation (Section 3.4, step 5). This division reflects a judgment about where human expertise is most valuable: crafting questions that genuinely require cross-modal reasoning and that have unambiguous correct answers is a creative, context-sensitive task that current LLMs would struggle to perform reliably, especially for long videos. Distractor generation, by contrast, is a more mechanical task of producing plausible-but-incorrect options given a known correct answer and video metadata, which LLMs can perform competently.
Why explicit timestamp annotation? The temporal annotation requirement serves both immediate evaluation purposes (enabling the temporal position analysis) and future research purposes (providing ground-truth temporal localization data that could support video grounding tasks). It also serves as a quality control mechanism: if an annotator cannot identify specific timestamps where the answer evidence appears, the question is likely too vague or the answer insufficiently grounded in the video.
Why 11 annotators? This is a practical choice reflecting the scale of the task (15,000 questions across 9,038 videos). The paper does not report inter-annotator agreement metrics, which would be valuable for assessing the consistency of skill tagging and answer construction across annotators—this is a limitation of the current documentation.

3.4.4 Distractor Generation and Difficulty Calibration

Once open-ended questions are authored and audited, they are converted to multiple-choice format through an automated distractor generation process followed by strategic manipulation of the answer options.

The distractor generation prompt. GPT-5.2 is used to generate nine incorrect answer options per question. The model is conditioned on "the question and additional video-level metadata" (Section 3.4, step 5), and the full prompt is provided in Figure 8 of the paper. The prompt includes the question text, the correct answer, and video-level context (presumably including the video domain, a summary or caption, and possibly the skill tags), enabling the LLM to generate distractors that are contextually relevant rather than generic.

The balanced distractor strategy. A critical design choice is that "half of the distractors are designed to be semantically plausible and grounded in the video context, while the remaining half are intentionally out-of-context" (Section 3.4, step 5). This balanced design serves a specific purpose: it prevents models from succeeding through superficial elimination strategies.

If all distractors were plausible and context-grounded, the task would require deep reasoning but might be solvable through elimination (identifying which options cannot be true based on video evidence). If all distractors were obviously wrong, the task would be trivial. By mixing plausible and out-of-context distractors, the benchmark forces models to genuinely identify the correct answer rather than merely eliminating obviously wrong ones—an out-of-context distractor is easy to eliminate, but a plausible distractor requires careful comparison against the video evidence.

What makes a distractor "plausible"? The prompt conditions on video-level metadata, so GPT-5.2 can generate options that reference entities, events, or concepts that actually appear in the video, but in incorrect relationships or at wrong times. For example, if a video shows a character wearing a red shirt and a blue shirt at different times, a plausible distractor might reference the red shirt in a context where the blue shirt was actually worn—requiring the model to track temporal changes in visual attributes to distinguish the correct answer from the distractor.

The "None of the above" manipulation. The paper introduces two variants of the "None of the above" (NOTA) option, following prior work by Tam et al. (2025):

In 13% of the questions (2,000 out of 15,000), the correct answer is replaced with "None of the above." This means that for these questions, none of the nine generated distractors are correct—the model must recognize that all presented options are wrong and select NOTA as the answer. This prevents models from adopting a strategy of always selecting one of the nine specific options, which would fail on NOTA-correct questions.
In another 13% of questions (2,000), one of the incorrect options is randomly replaced with "None of the above." In these questions, NOTA is a distractor—it is incorrect, and one of the other nine options is the correct answer. This prevents models from adopting the opposite strategy of always selecting NOTA when it appears, which would fail on NOTA-as-distractor questions.

The two manipulations together create a game-theoretic balance: if a model always picks a specific option (ignoring NOTA), it fails on the 13% of questions where NOTA is correct. If a model always picks NOTA when it appears, it fails on the 13% of questions where NOTA is a distractor. If a model randomly guesses, it achieves chance-level accuracy. The only viable strategy is to genuinely evaluate all ten options against the video evidence.

Why 10 options? The paper constructs 10 answer options per question (1 correct + 9 distractors, or when NOTA is used, the correct answer + 8 distractors + NOTA, or 9 distractors including NOTA as distractor + 1 correct option). Ten options reduces the random-guessing baseline to 10%, compared to 25% for standard 4-option multiple choice. This is important because with 15,000 questions, even a model achieving 30% accuracy (triple the random baseline) is performing substantially above chance—but not by enough to claim genuine understanding. The paper's reported model accuracies (best: 64.2%) are well above the 10% random baseline, confirming that models are not merely guessing.

Answer option distribution. Table 6 reports the distribution of correct answers across the ten option letters (A through J). The distribution ranges from 8.37% (option F) to 9.27% (option I), with the exception of option J which accounts for 19.60% of correct answers. The paper states that "the correct answer is distributed uniformly among the 10 option categories" (Appendix A), but option J's elevated proportion complicates this claim. This may be an artifact of how the NOTA manipulation interacts with option positioning—if NOTA is consistently placed in position J, questions where NOTA is correct would inflate J's count. The paper does not explicitly address this, leaving some ambiguity about whether the distributional anomaly introduces an exploitable bias.

Evaluation-time option randomization. To reduce option-order bias—where models may favor earlier or later positions regardless of content—the paper randomizes the option order five times per question during evaluation and takes the majority-selected answer (Section 4.2). If a model is biased toward position A, it would give different answers across the five randomizations, and the majority vote would not consistently select any particular position. This protocol ensures that the reported accuracy reflects content-based reasoning rather than position-based heuristics.

Design rationale summary. The distractor generation and difficulty calibration stage serves to convert open-ended questions into a multiple-choice format that is (a) resistant to superficial elimination strategies (through balanced plausible/implausible distractors), (b) resistant to option-position bias (through five-fold randomization), (c) resistant to NOTA gaming strategies (through balanced NOTA-as-correct and NOTA-as-distractor), and (d) statistically reliable (through 10 options reducing the random baseline to 10%). The overall effect is that the multiple-choice format captures the same reasoning demands as the original open-ended questions while enabling automated, scalable evaluation.

3.4.5 Quality Control and Filtering

The final construction stage is a rigorous quality control process that filters the annotated questions to ensure they meet MMOU's design specifications.

The review process. A separate group of expert reviewers (distinct from the original annotators) conducts the quality control. The paper states that these reviewers "remove ambiguous, redundant, or overly trivial questions, as well as instances with misaligned timestamps or weak audio-visual grounding" (Section 3.4, step 6). The key criterion for retention is: "Only questions that strictly require joint audio-visual reasoning and adhere to the annotation guidelines are retained."

What gets removed? The paper identifies several rejection categories:

Ambiguous questions: Questions where the correct answer is unclear, where multiple answers could be reasonably defended, or where the question wording is confusing. These would produce unreliable evaluation results because different models (or different human annotators) might legitimately disagree on the answer.
Redundant questions: Questions that are too similar to other questions on the same video, testing the same capability in essentially the same way. The paper states that annotators are encouraged to generate multiple diverse questions which are "then filtered" (Section 3.4, step 4)—the filtering removes near-duplicates to ensure that each question contributes independent information to the evaluation.
Overly trivial questions: Questions that are too easy (answerable from superficial cues, answerable from a single modality despite the annotation guidelines, or requiring only basic recognition rather than reasoning). These would not discriminate between models of different capability levels and would inflate accuracy scores without testing genuine understanding.
Misaligned timestamps: Questions where the annotated timestamps do not actually contain the answer evidence. This would compromise the temporal position analysis and the benchmark's validity as a temporally-grounded evaluation.
Weak audio-visual grounding: Questions that technically require both modalities but where one modality provides only minimal or redundant information. The paper's standard is that removing either modality should make the question unanswerable—questions that fail this standard, even if they nominally involve both modalities, are rejected.

The cross-modal dependency audit. The paper validates the filtering process by conducting a manual audit on a random 20% sample of MMOU (Section 3.3). The audit finds that "this subset satisfies 100% answer correctness and 100% strict audio-visual dependency." This is a strong claim: every audited question had both a verifiably correct answer and mandatory cross-modal dependency. The paper explicitly compares this to prior benchmarks, stating that MMOU "substantially exceeds the cross-modal rigor of existing benchmarks reported in Chen et al. (2025)."

From initial annotations to 15,000 final QAs. The paper does not report the number of questions initially generated by annotators versus the number retained after filtering. This is a missing piece of information—knowing the rejection rate would help readers assess the difficulty of meeting MMOU's quality standards and the efficiency of the annotation process. However, the final output is clearly stated: "The final MMOU benchmark consists of 15,000 carefully curated and reviewed QA instances" (Section 3.4, step 7).

Why this quality control design? The quality control stage is not optional or cosmetic—it is the mechanism that enforces the benchmark's defining properties. Without rigorous filtering, the benchmark would drift toward questions that are easier to write (single-modality, trivial, temporally concentrated) rather than questions that test the intended capabilities. The separation of annotators and reviewers (different groups) prevents the conflict of interest where annotators might be reluctant to reject their own work. The explicit rejection criteria make the filtering process transparent and reproducible, even though the exact rejection rate is not reported.

3.4.6 Dataset Statistics and Properties: The Resulting Measurement Instrument

The output of the construction pipeline is a benchmark whose quantitative properties determine what it can measure and how reliably. Table 2 summarizes the key statistics; Figures 3(a–e) visualize the distributions.

Video statistics. The benchmark contains 9,038 videos spanning 10 major categories and 36 fine-grained subcategories. The average duration is 711.6 seconds (approximately 11.9 minutes), with a minimum of 7.0 seconds and a maximum of 7,255.0 seconds (approximately 121 minutes). The duration distribution (Figure 3e) shows that while many videos cluster in shorter ranges, a substantial fraction extend into the 20–60+ minute range. All videos are sampled at 720p resolution (Section 3.2).

The key property of this distribution is that it is not uniform or symmetric. The long tail toward extended durations is deliberate: the benchmark must include very long videos to test context retention and temporal reasoning at scale. A model might perform perfectly on 5-minute videos and catastrophically on 60-minute videos, and a benchmark with only short videos would never reveal this distinction. The presence of videos spanning from 7 seconds to 121 minutes means that MMOU can measure the relationship between video length and model performance—though the paper's temporal position analysis (Figure 6) studies this question through answer position within videos rather than video duration per se.

Question statistics. The benchmark contains 15,000 multiple-choice QA pairs, each with 10 answer options. The average question length is 26.22 words, and the average answer length is 25.53 words (Table 2). These are relatively substantial texts—questions and answers are complete sentences or multi-sentence descriptions, not single-word labels. This reflects the complexity of the reasoning being tested: a question about audio-visual stitching might need to describe multiple video segments and their relationships, and the correct answer might need to articulate a narrative or causal explanation.

Skill distribution. Figure 3(d) shows the distribution of QA instances across the 13 skill types. The distribution is not uniform—some skills (e.g., temporal understanding, sub-scene understanding) are more heavily represented than others (e.g., audio-visual stitching, spurious correlations). This imbalance reflects two factors: (a) some skills are more naturally testable across a wide range of video domains, and (b) the annotation process generates more questions targeting certain skills because they are more common in the selected videos. The paper does not claim or aim for uniform skill distribution; instead, it aims for sufficient representation of each skill to enable meaningful per-skill analysis (Figure 5), which the distribution supports.

The average of 2.71 skills per question (Table 2) means that most questions test multiple skills simultaneously. The co-occurrence matrix (Figure 3b) visualizes which skill pairs frequently appear together. Strong co-occurrence between, for example, temporal understanding and sequential event ordering would be expected (both involve reasoning about event order); co-occurrence between temporal understanding and counting would indicate questions that require counting events across time. These patterns reflect the natural structure of real-world audio-visual reasoning—skills are not deployed in isolation.

Temporal answer position. The average answer position is 302.28 seconds (approximately 5 minutes) into the video (Table 2). Combined with the average video duration of 711.6 seconds, this means that on average, answer evidence appears roughly 42% of the way through the video. The distribution (Figure 3c) shows that answer positions are "spread across the entire video timeline" (Section 3.2), with representation in early, middle, and late temporal regions. This deliberate diversification (annotators were "encouraged to diversify" temporal positions) enables the key analysis in Figure 6: plotting model accuracy as a function of answer evidence position.

Answer option distribution. Table 6 reports the per-option distribution of correct answers:

Option	Count	Percentage
A	1,325	8.83%
B	1,329	8.86%
C	1,380	9.20%
D	1,308	8.72%
E	1,337	8.91%
F	1,255	8.37%
G	1,384	9.23%
H	1,352	9.01%
I	1,390	9.27%
J	2,940	19.60%

Options A–I range from 8.37% to 9.27%—approximately uniform, as the paper claims ("we ensure that the correct answer is distributed uniformly among the 10 option categories," Appendix A). Option J's 19.60% is a clear outlier. The most likely explanation is that the NOTA manipulations are consistently placed in position J: the 2,000 questions where NOTA is the correct answer and/or the 2,000 questions where NOTA appears as a distractor may be assigned to position J disproportionately. If all NOTA-correct answers occupy position J, that would account for 2,000 of the 2,940 J-position correct answers, leaving 940 for other questions—roughly in line with the ~1,300 per other position. The paper does not explicitly confirm this, but the numbers are consistent with this interpretation. The practical implication is that a model that learns a "pick J when unsure" heuristic would benefit disproportionately—the five-fold option randomization during evaluation (Section 4.2) mitigates this by shuffling option positions, but the paper does not report whether position-J bias persists after randomization.

Domain distribution. Table 5 provides the full breakdown. The largest categories by question count are News (4,221), Sports (3,484), Daily Life (3,237), and Academic Lectures (944). The smallest are Animation (184), Music (630), and Pranks (518). This imbalance reflects the availability of question-viable videos: news broadcasts, sports matches, and DIY tutorials are abundant sources of long, content-rich videos with strong audio-visual coupling, while music parody or jump-scare prank videos are less common and less suitable for generating many diverse questions.

Human performance baseline. Table 3 reports human accuracy at 84.3% overall, with per-domain scores ranging from 82.0% (Animation) to 90.0% (News). Five graduate students, none of whom are authors, answered the benchmark questions with the ability to pause and rewind videos as needed (Appendix D). The 84.3% figure serves as the ceiling: it represents the performance of educated humans with unlimited time access to the video content. The gap between 84.3% (human) and 64.2% (best model, Gemini 2.5 Pro) of 20.1 percentage points is the paper's primary quantitative claim about the state of the field.

Why these statistics matter for evaluation validity. A benchmark's statistical properties determine what conclusions can be reliably drawn from it:

Scale (15,000 questions): Provides statistical power for subgroup analyses. A per-skill accuracy difference of 5 percentage points is more likely to be genuine when measured over 500+ questions per skill than over 50.
Domain diversity (10 categories, 36 subcategories): Prevents overfitting to a single video type. A model that excels on sports videos but fails on academic lectures might achieve a misleadingly high aggregate score on a sports-heavy benchmark; MMOU's domain spread forces broad capability.
Temporal answer diversity: Enables the temporal position analysis (Figure 6) that would be impossible if all answers clustered at the beginning of videos.
Skill co-occurrence (2.71 skills/question): Reflects real-world reasoning complexity but complicates skill-level attribution. If a model fails on questions tagged with both "counting" and "temporal understanding," is the failure due to counting difficulty or temporal difficulty? The co-occurrence matrix partially disentangles this.
Option count (10 options): Reduces random baseline to 10%, making even modest accuracy scores meaningful, but also increases the cognitive load on models (10 options require more careful discrimination than 4).

4. Key Insights and Innovations

The field's dominant assumption before MMOU was that evaluating models on vision benchmarks and audio benchmarks separately would reveal whether they possess integrated audio-visual understanding. The reasoning was additive: strong vision performance plus strong audio performance ought to imply strong audio-visual performance. MMOU demonstrates that this assumption is false, and it does so not through an argument but through a benchmark design choice that makes the failure visible.

What's distinctive at the idea level. MMOU enforces a constraint that no prior omni-modal benchmark had systematically applied: every question must be unanswerable from either modality alone. This is not a statistical aspiration or a rough guideline; it is a hard design rule verified through manual audit. The paper reports that a random 20% sample achieves 100% strict audio-visual dependency (Section 3.3), and that modality-specific models—the vision-only Qwen3-VL-32B at 44.0% accuracy and the audio-only Audio Flamingo 3 at 17.7%—perform far below the best audio-visual model at 64.2% (Table 3). These numbers are not merely lower; they confirm that removing either modality makes the benchmark essentially unsolvable at human-competitive levels.

Comparison to prior work. Prior audio-visual benchmarks such as OmniBench (Li et al., 2024d), AV-Odyssey (Gong et al., 2024), and WorldSense (Hong et al., 2025) include both modalities but do not enforce this constraint as a design requirement. The paper explicitly cites Chen et al. (2025) in reporting that existing benchmarks "often limit questions to a single dominant modality" and "fall short of requiring genuine cross-modal reasoning" (Section 3.3). The consequence, visible across the literature, is that models could achieve strong scores on these benchmarks by relying primarily on one modality while treating the other as auxiliary or ignoring it entirely. MMOU's mandatory cross-modal dependency closes this loophole: there is no shortcut via unimodal reasoning.

Significance beyond raw performance. This is a diagnostic reframing, not merely a harder benchmark. The paper is arguing, in effect, that the evaluation community has been measuring the wrong thing—or more precisely, measuring something that is correlated with but not identical to the capability of interest. Strong vision-only performance on video benchmarks like Video-MME (Fu et al., 2024) and strong audio-only performance on benchmarks like MMAU (Sakshi et al., 2024a) may both be true of a model that completely lacks the ability to integrate modalities. By making cross-modal dependency a necessary condition for benchmark inclusion, MMOU transforms "audio-visual understanding" from a property that can be approximated by summing unimodal scores into a property that must be directly demonstrated. This is a conceptual contribution to how we define and measure multimodal intelligence, independent of the specific accuracy numbers.

Evidence anchor. The modality-restricted baseline results in Table 3 are the operational proof. Qwen3-VL-32B (vision-only) achieves 44.0% overall; Qwen3-Omni (audio-only) achieves 35.6%. These are not random baselines but state-of-the-art unimodal systems. The gap between these numbers and the best audio-visual model (64.2%) quantifies the marginal value of cross-modal integration—a quantity that no prior benchmark could reliably estimate because none controlled for the possibility that questions could be answered unimodally.

Innovation 2: "Audio-Visual Understanding" Is Not a Monolithic Capability — It Decomposes Into 13 Distinct Skills That Current Models Fail on Differentially

The field has largely treated multimodal video understanding as a single, undifferentiated capability measured by aggregate accuracy on QA pairs. MMOU challenges this with a skill taxonomy that reveals which aspects of audio-visual reasoning are hard, and the finding is not simply that models are uniformly bad—it is that they exhibit starkly differential failure patterns that a single aggregate score would mask.

What's distinctive at the idea level. The paper constructs a taxonomy of 13 fundamental audio-visual reasoning skills, defined with operational precision (Tables 7–8), and tags every question with the specific skills it requires (averaging 2.71 skills per question, per Table 2). This enables a per-skill accuracy breakdown (Figure 5) that reveals a pattern invisible in the aggregate 64.2% number: even the best closed-source model (Gemini 2.5 Pro) struggles disproportionately with counting and temporal understanding, skills that humans consider basic. The paper's framing is that "audio-visual understanding" is not one thing that models are partially good at—it is a collection of distinct cognitive operations, and current models are good at some (e.g., holistic reasoning, inference) and bad at others (e.g., counting, needle-in-the-haystack retrieval), with the pattern consistent across open and closed models.

Comparison to prior work. Prior benchmarks either do not decompose performance by skill at all (OmniBench, AV-Odyssey, DailyOmni) or decompose along dimensions that are not specifically audio-visual reasoning skills—Video-MME's 30 task types, for example, include categories like "OCR" and "object recognition" that are primarily visual. MMOU's skill taxonomy is deliberately cross-modal: skills like "audio-visual stitching" (understanding narrative continuity across edited segments) and "tracking spurious correlations" (identifying surprising audio-visual coincidences that language priors cannot predict) have no analog in vision-only or audio-only benchmarks because they exist only at the intersection of modalities. The taxonomy is therefore not just a finer-grained version of existing categorizations; it is a new categorization space that prior benchmarks never entered.

Significance beyond raw performance. This is a diagnostic infrastructure contribution. The skill taxonomy and per-skill analysis framework give model developers actionable information about where to improve, rather than simply that improvement is needed. If a team knows their model scores 46.8% overall (best open-source, MiniCPM-o 4.5, Table 3) but also knows it achieves near-ceiling on holistic reasoning and near-floor on counting (Figure 5), they can prioritize training data, architecture changes, or inference strategies that specifically target quantitative reasoning over repeated audio-visual events. Without the skill decomposition, all failures look the same and all improvements are guesswork. The co-occurrence matrix (Figure 3b) further enables analysis of correlated failures—if counting and temporal understanding failures are correlated, the underlying deficit may be a shared mechanism (e.g., tracking discrete events across time) rather than two independent weaknesses.

Evidence anchor. Figure 5 shows that for Gemini 2.5 Pro (the best model), Counting accuracy is dramatically lower than Holistic Reasoning or Context Understanding accuracy. The bar plot shows Counting near the bottom of the skill ranking for all three plotted models (Gemini 2.5 Pro, Qwen3-Omni-Think, OmniVinci), with a spread of roughly 20–40 percentage points between the easiest and hardest skills. This differential is the empirical basis for the claim that aggregate accuracy masks skill-level heterogeneity.

Innovation 3: Video Duration Alone Is Not the Problem — Temporal Position of Answer Evidence Within the Video Is What Causes Performance Degradation

Prior video understanding benchmarks have focused on video duration as the primary difficulty axis: longer videos are assumed to be harder. MMOU introduces a more precise diagnostic: it annotates the temporal position of answer evidence within each video (as earliest and latest timestamps) and uses this to measure whether models degrade because they cannot process long contexts or because they cannot retrieve information from specific temporal locations.

What's distinctive at the idea level. The paper shows (Figure 6) that model accuracy degrades steadily as answer-relevant evidence appears later in the video, with a sharp drop for evidence toward the end of long sequences. This is not the same as saying "longer videos are harder." A video could be 60 minutes long but the answer evidence could appear at minute 2—and the model might perform well. Conversely, answer evidence at minute 25 of a 30-minute video might produce worse performance. The distinction matters because it points to a specific mechanism—retention and retrieval of information from earlier parts of long contexts—rather than a vague notion of "complexity." The paper explicitly frames this as revealing "a fundamental limitation in long-horizon temporal reasoning and context retention" (Section 6).

Comparison to prior work. Prior benchmarks report performance by video duration bucket (including MMOU itself in Table 3, which breaks accuracy by <5, 5–10, 10–20, 20–30, and >30 minute bins). But duration bucketing conflates two variables: how long the video is, and where the answer evidence is located. A model that appears to degrade on >30-minute videos might be failing because it cannot handle 30+ minutes of context, or because the answer evidence in those videos happens to be positioned later. The temporal position analysis disentangles these: holding video duration approximately constant (by analyzing within-duration bins, though the paper does not explicitly perform this conditioning), accuracy decreases with answer position. This is a more targeted diagnostic than duration-based analysis.

Significance beyond raw performance. This is a diagnostic refinement with direct architectural implications. If models fail primarily because of temporal position rather than total duration, the solution is not necessarily longer context windows—it may be better temporal attention mechanisms, improved retrieval of earlier frames, or strategies for identifying which segments of a long video are relevant to a query. The finding also suggests that evaluation protocols should control for temporal position when comparing models: two models might achieve identical aggregate accuracy, but one might maintain performance at late temporal positions while the other collapses—a difference invisible in the aggregate number. The annotation of explicit answer timestamps (Section 3.4, step 3) is the infrastructure that enables this analysis, and the paper's advocacy for temporal position as an evaluation axis is a methodological contribution beyond the specific numbers.

Evidence anchor. Figure 6 plots accuracy as a function of answer evidence start time, showing a clear downward trend for all three plotted models (Gemini 2.5 Pro, OmniVinci, Qwen3-Omni). The curve is not flat—it slopes downward, with the steepest decline in the later time ranges. This is the direct empirical evidence that temporal position, not just video duration, drives performance degradation.

Innovation 4: Multiple-Choice Evaluation Overestimates True Understanding — Open-Ended Assessment Reveals a Capability Gap That MCQ Masks

The paper contributes a specific empirical finding about evaluation methodology itself: models that achieve respectable multiple-choice scores on MMOU perform substantially worse when required to generate open-ended answers without option scaffolding, revealing that MCQ performance partially reflects recognition and elimination strategies rather than genuine understanding.

What's distinctive at the idea level. The paper does not simply report open-ended scores alongside MCQ scores; it quantifies the conditional relationship between them. For questions where Gemini 2.5 Pro scored poorly on open-ended correctness (<2 out of 5), it still answered 21.1% of those same questions correctly in MCQ format (Section 6). The implication is precise and falsifiable: approximately one-fifth of the model's correct MCQ answers are questions it cannot answer when required to generate the answer from scratch. This proportion varies by skill—Subscene shows the highest MCQ-correct rate among open-ended failures (29.1%), while Holistic Reasoning shows the lowest (10.5%)—suggesting that some skills are more susceptible to MCQ scaffolding than others.

Comparison to prior work. Most multimodal benchmarks report only MCQ accuracy (MMMU, Video-MME, AV-Odyssey, OmniBench, DailyOmni, WorldSense). A smaller number include open-ended evaluation (MMVU, MMAU-Pro, UNO-Bench), but these typically report open-ended and MCQ scores as parallel metrics rather than analyzing the relationship between them. The paper's innovation is the conditional analysis: given that a model fails open-ended, how often does MCQ format rescue it? This transforms open-ended evaluation from a separate score into a diagnostic for whether MCQ scores overestimate competence, and by how much.

Significance beyond raw performance. This is both a methodological caution and a measurement innovation. The methodological caution is that MCQ benchmarks may systematically overstate model capability, and the degree of overstatement may vary by skill category—meaning cross-model or cross-skill comparisons based solely on MCQ accuracy could be misleading. The measurement innovation is the rubric-based open-ended scoring (four criteria: Correctness, Completeness, Faithfulness, Clarity, with a weighted overall formula in Equation 1) which provides a richer signal than binary MCQ correctness. The paper's finding that Clarity scores are consistently high (4.50–4.70, Figure 7) while Correctness and Completeness vary widely suggests that models are fluent and articulate even when factually wrong—a pattern that binary correctness would not capture but that has significant implications for deployment safety (users may trust fluent-but-wrong answers).

Evidence anchor. Table 4 reports open-ended scores across eight models, with the best (Gemini 2.5 Pro) achieving only 3.90/5.00 overall. The conditional analysis reports: among questions with poor open-ended correctness, Gemini 2.5 Pro answered 21.1% correctly in MCQ; Qwen3-Omni-Think 13.5%; and Omnivinci 12.9%. Figure 7 shows the per-skill breakdown of Gemini 2.5 Pro's open-ended scores, revealing that Counting is the hardest skill across all four rubric dimensions—a finding consistent with the MCQ skill analysis in Figure 5 but richer because it decomposes the failure into correctness (3.01), completeness (3.27), and faithfulness (3.50) components.

The paper finds that while closed-source models achieve higher absolute accuracy than open-source models on MMOU (64.2% vs. 46.8% for the best in each category), the pattern of failures—which skills are hard, how performance degrades with temporal position, where open-ended generation falls short—is remarkably consistent across model families, scales, and access levels.

What's distinctive at the idea level. The paper does not merely report that larger or proprietary models perform better—that would be unsurprising. The contribution is the observation that the structure of difficulty is invariant to model scale and training methodology. Figure 5 shows that Gemini 2.5 Pro (the strongest closed-source model), Qwen3-Omni-Think, and OmniVinci all struggle with the same skills (Counting, Needle-in-the-Haystack, Temporal Understanding) and excel at the same skills (Context Understanding, Holistic Reasoning, Sub-scene Understanding). The rank ordering of skill difficulty is largely preserved across models, even though the absolute accuracy levels differ. Figure 6 shows a similar pattern for temporal position degradation: all models exhibit the downward slope, with the closed-source model maintaining a higher intercept but the same qualitative trend.

Comparison to prior work. Scaling laws research in LLMs (Kaplan et al., 2020; Hoffmann et al., 2022) has established that larger models generally perform better across most tasks, but also that some capabilities fail to emerge even at scale. The parallel in MMOU is that cross-modal integration quality does not appear to be purely a function of parameter count or training compute—the relative difficulty of different audio-visual reasoning skills is stable across model families, suggesting there is something fundamental about these skills that resists scaling-based solutions. This finding is reminiscent of results in mathematical reasoning (e.g., from the scaling-test-time-compute literature) where certain problem difficulties remain insurmountable regardless of model size because they require capabilities the base model simply does not possess.

Significance beyond raw performance. This is a negative result with positive implications for research prioritization. If the failure patterns were idiosyncratic to specific models, the solution would be model-specific fixes (better training data for model X, architectural changes for model Y). The uniformity of failure patterns suggests instead that the deficits are task-inherent: counting audio-visual events across long videos is hard for the same fundamental reasons regardless of whether the model is Gemini 2.5 Pro or OmniVinci. This means research investment should focus on developing new training paradigms, data, or architectures specifically targeting these skills, rather than on incremental scaling of existing approaches. The finding also implies that open-source models can serve as valid proxies for studying these failure modes—the qualitative patterns are the same, even if the absolute scores differ.

Evidence anchor. Figure 5 shows three models (Gemini 2.5 Pro, Qwen3-Omni-Think, OmniVinci) with accuracy bars plotted side-by-side for each skill. The bar heights differ (Gemini is consistently higher), but the low-high pattern across skills is nearly identical: all three show Counting as the lowest or near-lowest bar, all three show Holistic Reasoning and Context Understanding among the highest. Figure 6 shows three accuracy-vs-temporal-position curves with similar downward slopes but different intercepts. Table 3 shows that the performance ranking of models is largely consistent across video domains and duration buckets—no model shows a dramatic domain-specific reversal where it excels in a domain others find hard.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MMOU benchmark itself, consisting of 15,000 multiple-choice QA pairs sourced from 9,038 web-collected videos averaging 711.6 seconds in duration, spanning 10 major categories and 36 fine-grained subcategories (Section 3.2, Table 2). The dataset is used in its entirety for evaluation; there is no separate training/validation/test split because MMOU is a zero-shot evaluation benchmark—models are evaluated as-is with no fine-tuning on MMOU data.
Base model(s). The paper evaluates 20+ models spanning five architectural families: closed-source audio-visual MLLMs (Gemini 2.5 Pro, Gemini 2.5 Flash), open-source audio-visual MLLMs (Qwen2.5-Omni-7B, Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Thinking, Phi-4 Multimodal, Gemma 3n, MiniCPM-o 4.5, Video-LLaMA 2, OmniVinci, Baichuan-Omni-1.5), vision-only LVLMs (Qwen3-VL-32B-Instruct, Qwen3-VL-8B-Instruct, Qwen2.5-VL-7B-Instruct), audio-only LALMs (Audio Flamingo 3, Qwen3-Omni-30B-A3B in audio-only mode), and text-only LLMs (Qwen3-235B, GPT-5.2, GPT-4.1-mini). These models are chosen to span the full range of current capability—from proprietary frontier systems to open-source alternatives to modality-restricted baselines—enabling the paper to measure both the absolute state of the art and the marginal value of cross-modal integration (Section 4.1, Appendix E).
Metrics. The primary metric is micro-averaged accuracy (% of questions answered correctly). For each question, models are shown a set of 10 answer options (including "None of the above" variants as described in Section 3.4.4) and instructed to select exactly one. The predicted option is extracted via robust regular-expression-based parsing and matched against the ground-truth option via string comparison. To reduce option-order bias, the option order is randomized five times per question, and the majority-selected answer across the five randomizations is used as the final prediction. The paper further reports that each model is evaluated under multiple prompt variants (the specific prompts are not enumerated), with the best-performing prompt configuration reported for all MLLMs (Section 4.2).
Baselines. The paper organizes baselines into five tiers, each serving a distinct diagnostic purpose:
- Proprietary audio-visual MLLMs: Gemini 2.5 Pro and Gemini 2.5 Flash (Comanici et al., 2025), representing the current performance frontier with access to both modalities.
- Open-source audio-visual MLLMs: Qwen2.5-Omni-7B (Xu et al., 2025a), Qwen3-Omni-30B-A3B-Instruct and Qwen3-Omni-30B-A3B-Thinking (Xu et al., 2025b), Phi-4 Multimodal (Abouelenin et al., 2025), Gemma 3n (Team et al., 2025), MiniCPM-o 4.5 (OpenBMB, 2025), Video-LLaMA 2 (Cheng et al., 2024), OmniVinci (Ye et al., 2025), and Baichuan-Omni-1.5 (Li et al., 2025b), representing the best publicly available models with joint audio-visual input.
- Vision-only models: Qwen3-VL-32B-Instruct, Qwen3-VL-8B-Instruct (Bai et al., 2025a), and Qwen2.5-VL-7B-Instruct (Bai et al., 2025b), which receive the video track but no audio. These quantify the upper bound of what can be achieved from visual information alone and, by comparison with audio-visual models, measure the incremental value of audio integration.
- Audio-only models: Audio Flamingo 3 (Goel et al., 2025) and Qwen3-Omni-30B-A3B operated in audio-only mode, which receive the audio track but no visual input. These serve the symmetric purpose of quantifying audio-only capability.
- Text-only LLMs: Qwen3-235B (Yang et al., 2025), GPT-5.2 (OpenAI, 2025), and GPT-4.1-mini, which receive only the question and answer options with no access to video or audio. These measure whether language priors alone can answer MMOU questions—a crucial control for verifying that the benchmark genuinely requires perception rather than textual reasoning. Cascaded models (Qwen3-VL + Qwen3-Omni-Audio captions fed to Qwen3-235B or GPT-5.2) are an additional baseline that tests whether text descriptions of audio and visual content are sufficient for solving MMOU without end-to-end multimodal perception (Section 4.1).
Generation budget / compute accounting. MMOU is a zero-shot evaluation benchmark; models are evaluated once on each question without any test-time compute scaling, search, or generation budget variation. There is no "compute budget" to account for because all models are evaluated under identical conditions: they receive the video, audio, question, and answer options, and produce a single answer. The fairness of comparison is maintained by (a) using the same questions for all models, (b) randomizing option order five times to neutralize position bias, and (c) evaluating each model under multiple prompt variants and reporting the best-performing variant. For the cascaded baselines, the additional compute of generating separate audio and visual captions is noted but not quantified in FLOPs—the paper simply reports whether this approach works at all.
Cross-validation / statistical protocol. The paper does not employ cross-validation (there is no training or hyperparameter tuning on MMOU data). The primary statistical protocol is the five-fold option-order randomization described above, which ensures that reported accuracy reflects content-based reasoning rather than position-based heuristics. For the open-ended evaluation (Section 6), a separate protocol is used: models generate free-form answers without access to options, and responses are scored by GPT-5 as an LLM judge using a four-dimensional rubric (Correctness, Completeness, Faithfulness, Clarity) on a 1–5 scale, with a weighted overall score computed as 0.5 × Correctness + (0.5/3) × (Completeness + Faithfulness + Clarity) (Appendix G, Equation 1). The judge receives the question, ground-truth answer, model response, and a detailed audio-visual caption for context. Human evaluation (84.3% accuracy, Table 3) is conducted by five graduate students who are not authors, with scores averaged across annotators (Appendix D). The paper does not report confidence intervals or statistical significance tests for any model comparisons—a limitation discussed in the Critical Assessment below.

Main Quantitative Results

Overall Model Performance on MMOU

Headline result. The best closed-source model, Gemini 2.5 Pro, achieves 64.2% overall accuracy on MMOU, while the strongest open-source model, MiniCPM-o 4.5, reaches 46.8% (Table 3). Human performance is 84.3%, establishing a 20.1-percentage-point gap between the best AI system and educated humans with unlimited video access. The random baseline (10 options) is 10.0%.

Proprietary vs. open-source gap. Gemini 2.5 Pro (64.2%) outperforms the best open-source audio-visual model by 17.4 percentage points. This gap is not uniform across all proprietary models: Gemini 2.5 Flash (the lightweight proprietary variant) achieves 55.8%, only 9.0 points above MiniCPM-o 4.5, suggesting that the gap is driven by the largest proprietary model rather than proprietary status per se. The second-best open-source model, Qwen3-Omni-30B-A3B-Instruct, achieves 46.0%, essentially tied with MiniCPM-o 4.5 at 46.8%.

Open-source model spread. Open-source audio-visual models span a wide range: from 19.4% (Qwen3-Omni-30B-A3B-Thinking, surprisingly the worst open-source omni-modal model despite being a reasoning-enhanced variant) to 46.8% (MiniCPM-o 4.5). The thinking variant's poor performance (19.4%) versus its non-thinking counterpart (46.0%) is a striking negative result: enhanced reasoning capabilities, as implemented in this model, actively harm performance on MMOU, perhaps because the thinking process overfits to unimodal reasoning patterns or because extended generation introduces errors. The paper does not investigate this discrepancy in detail. Other open-source models cluster in the 23–31% range: Baichuan-Omni-1.5 (23.2%), OmniVinci (24.7%), Video-LLaMA 2 (24.8%), Gemma 3n (30.7%), Qwen2.5-Omni-7B (31.3%), Phi-4 Multimodal (31.4%).

The cross-modal integration gap. The incremental value of audio-visual integration over vision-only reasoning is modest for most models. Qwen3-VL-32B-Instruct (vision-only) achieves 44.0%, while Qwen3-Omni-30B-A3B-Instruct (audio-visual) achieves 46.0%—only 2.0 percentage points higher. This suggests that even models designed for multimodal input are extracting limited additional value from the audio stream on MMOU. Similarly, Qwen3-VL-8B-Instruct (vision-only, 36.1%) and Qwen2.5-VL-7B-Instruct (vision-only, 30.2%) are not dramatically lower than similarly-sized audio-visual models. The exception is Gemini 2.5 Pro, whose 64.2% substantially exceeds any vision-only model, indicating that its audio-visual integration is genuinely effective in a way that open-source models have not replicated.

Audio-only and text-only baselines confirm benchmark validity. Audio-only models perform poorly: Audio Flamingo 3 achieves 17.7%, and Qwen3-Omni in audio-only mode achieves 35.6%. Text-only LLMs achieve non-trivial but sub-frontier accuracy: GPT-5.2 at 40.7%, Qwen3-235B at 37.5%, GPT-4.1-mini at 33.9%. These numbers are well above the 10.0% random baseline, indicating some textual priors in the questions and answer options, but well below Gemini 2.5 Pro's 64.2%, confirming that genuine audio-visual perception provides substantial additional information. Cascaded models (caption-then-answer) perform worse than end-to-end multimodal models: Qwen3-(VL+O-A) + Qwen3-235B achieves 33.1%, and the same pipeline with GPT-5.2 achieves only 28.1%. This is an important negative result: providing textual descriptions of both modalities to a strong text-only LLM is substantially worse than end-to-end multimodal processing, indicating that captions lose information critical for answering MMOU questions—likely fine-grained temporal alignment, subtle cross-modal interactions, and precise visual details that captioning models omit.

Per-domain performance (Table 3, columns 2–11). Gemini 2.5 Pro's accuracy varies substantially across the 10 video domains:

Domain	Gemini 2.5 Pro Accuracy
Academic Lectures	71.4%
Pranks	71.0%
Daily Life	68.1%
Travel	67.3%
Film	66.5%
News	61.8%
Sports	61.2%
Video Games	60.9%
Music	59.7%
Animation	58.2%

The spread of 13.2 percentage points between the easiest (Academic Lectures, 71.4%) and hardest (Animation, 58.2%) domains is substantial. The pattern is not trivially explainable by a single factor: Academic Lectures, which feature tightly synchronized speech and visual aids, are among the easiest, while Animation and Music—which might seem simpler due to their entertainment nature—are among the hardest. This may reflect that animated content requires generalization beyond photorealistic training distributions, while music videos require integrating abstract visual narratives with auditory content. The best open-source model, MiniCPM-o 4.5, shows a similar but noisier domain pattern, with a spread from 50.8% (Daily Life) to 29.2% (Music) and 30.4% (Video Games).

Per-duration performance (Table 3, columns 12–16). Accuracy generally degrades with video duration, but the pattern is not monotonic for all models:

Gemini 2.5 Pro: 62.2% (<5 min), 66.2% (5–10 min), 66.2% (10–20 min), 59.0% (20–30 min), 58.5% (>30 min). The surprising increase from <5 to 5–10 minutes may reflect that very short videos lack sufficient context for some questions, or that the distribution of question difficulty differs across duration buckets. The clear drop at 20+ minutes is consistent with long-context degradation.
Open-source models: Show similar late-duration degradation but with lower absolute numbers. MiniCPM-o 4.5 drops sharply from 48.1% (<5 min) to 9.1% (>30 min)—a catastrophic 39-point collapse. Qwen3-Omni-30B-A3B-Thinking drops to 18.2% at >30 min.
Vision-only models: Qwen3-VL-32B shows remarkable stability across durations: 44.5% (<5 min), 45.3% (5–10), 43.3% (10–20), 40.4% (20–30), 44.1% (>30). This near-flat profile suggests that vision-only processing may be less susceptible to temporal degradation than audio-visual processing—or that vision-only models are failing for reasons unrelated to duration, such as missing the audio information needed to answer any question correctly regardless of temporal position.

The >30-minute bucket is particularly revealing: even Gemini 2.5 Pro drops to 58.5%, and all open-source models except Qwen3-Omni-30B-A3B-Instruct (43.6%) fall below 30%. This confirms that extended temporal contexts remain a significant challenge even for frontier systems.

Skill-Wise Performance Analysis

Headline result. All models exhibit substantial weaknesses in what the paper terms "basic and essential skills such as temporal understanding, counting, and needle-in-the-haystack reasoning" (Section 6, Figure 5). Even Gemini 2.5 Pro shows dramatic performance variation across skills, with Counting being consistently among the worst-performing categories.

Figure 5 interpretation. The bar chart in Figure 5 plots accuracy for three models—Gemini 2.5 Pro, Qwen3-Omni-Think, and OmniVinci—across the 13 skill categories. Several patterns are evident:

Skill difficulty ranking is preserved across models. Counting, Temporal Understanding, and Needle-in-the-Haystack Reasoning appear among the lowest bars for all three models. Holistic Reasoning (labeled "Hol. Reason." in the figure), Context Understanding, and Subscene Understanding appear among the highest. This invariant ranking suggests that these skills have inherent difficulty properties that are not overcome by scaling or architectural differences—a finding consistent with the paper's claim that failure patterns are systematic rather than idiosyncratic.
Counting is the hardest skill. For all three plotted models, Counting shows the lowest or near-lowest accuracy. This is a specific, interpretable finding: current MLLMs cannot reliably count instances of audio-visual events across extended videos, even when they can understand and describe those events individually. The paper's example of a counting question (Table 7) asks how many shadows are visible on a skate rink when a horn starts blaring—this requires (a) locating the specific audio event (the horn), (b) identifying all shadow instances at that exact moment, and (c) enumerating them correctly. Each sub-task is individually within model capabilities (event localization, object detection, counting), but the combination across modalities and precise temporal alignment creates a failure mode.
Temporal Understanding and Needle-in-the-Haystack are problematic even for the best model. These skills require tracking event sequences across time and localizing specific moments in long videos, respectively. Their low accuracy even at 64.2% overall performance confirms that temporal reasoning—not just perception—is a core weakness. This connects directly to the temporal position analysis (Figure 6), which shows that accuracy degrades as relevant evidence appears later in videos.
The open-source/closed-source gap is skill-dependent. For some skills (Context Understanding, Holistic Reasoning, Subscene Understanding), Gemini 2.5 Pro dramatically outperforms open-source models. For others (Counting, Needle-in-the-Haystack), the gap narrows because all models perform poorly—the floor effect compresses the difference. This means the aggregate 17.4-point gap between Gemini 2.5 Pro and the best open-source model is driven disproportionately by skills where Gemini excels, and underestimates the parity (or shared inadequacy) on the hardest skills.

Quantitative skill-level numbers are not reported. A significant limitation is that Figure 5 is a bar chart without numerical labels in the paper text. The paper describes the pattern qualitatively ("struggle with basic skills like counting and finding temporal relationships") but does not provide per-skill accuracy numbers that would enable precise comparisons or statistical tests. This makes the skill-wise analysis suggestive rather than definitive—readers can see the relative patterns but cannot quote specific accuracy values for individual skills.

Temporal Position Sensitivity Analysis

Headline result. Model accuracy degrades steadily as the answer-relevant evidence appears later in the video, with a sharp drop for evidence toward the end of long sequences (Section 6, Figure 6). This trend is consistent across all plotted models (Gemini 2.5 Pro, OmniVinci, Qwen3-Omni).

Figure 6 interpretation. The plot shows accuracy (y-axis, 10–60% range) as a function of answer evidence start time (x-axis, 0 to >50 minutes). Three curves are shown:

Gemini 2.5 Pro maintains the highest accuracy across all temporal positions, starting around 55–60% for evidence in the first 10 minutes and declining to approximately 35–40% for evidence beyond 50 minutes.
OmniVinci starts around 35–40% for early evidence and declines to approximately 15–20% at late positions.
Qwen3-Omni starts around 25–30% and declines to approximately 10–15%.

All three curves slope downward, but the shape is not linear—the decline appears steeper in the later time ranges (beyond approximately 30 minutes), consistent with the paper's description of a "sharp drop for evidence located toward the end of long sequences."

What this means mechanistically. The paper frames this as revealing "a fundamental limitation in long-horizon temporal reasoning and context retention" (Section 6). The mechanism is likely twofold: (a) as more video content accumulates between the start and the answer evidence, the model's attention is diluted across a larger context, making it harder to attend precisely to the relevant evidence; (b) the model must retain information from earlier parts of the video (context that makes the answer evidence interpretable) across a longer temporal gap. A question about an event at minute 40 of a 50-minute video may require integrating information from minute 5 (who the characters are), minute 20 (what they're trying to accomplish), and minute 40 (the specific event), and the model may lose or confuse information from minutes 5 and 20 by the time it reaches minute 40.

Interaction with video duration. Figure 6 plots against answer position, not video duration, but the two are correlated (answer evidence in very long temporal positions can only occur in sufficiently long videos). The paper does not disentangle the effect of absolute temporal position from the effect of video duration by, for example, plotting accuracy for answers at minute 10 in 15-minute videos vs. minute 10 in 60-minute videos. Such an analysis would reveal whether the degradation is due to the amount of preceding content (absolute temporal position) or the total context length (video duration). The current analysis cannot distinguish these.

Interaction with skill type. The paper does not break down temporal position sensitivity by skill type. It is plausible—and would strengthen the analysis considerably—that some skills are more sensitive to temporal position than others. Needle-in-the-haystack questions, which explicitly require locating specific moments, may show the steepest degradation, while holistic reasoning questions (which integrate information across the entire video) may be less affected because they do not depend on a single temporal location. The paper's infrastructure (timestamp-annotated answers, skill tags) supports this analysis, but it is not reported.

Open-Ended Evaluation Results

Headline result. Open-ended evaluation reveals a substantial capability gap beyond what MCQ accuracy suggests. The best model, Gemini 2.5 Pro, achieves an overall weighted score of 3.90 out of 5.00, with Correctness at 3.71 and Completeness at 3.86—well below ceiling (Section 6, Table 4).

Table 4: Open-ended scores across eight models.

Model	Correctness	Completeness	Faithfulness	Clarity	Overall
Gemini 2.5 Pro	3.71	3.86	3.80	4.62	3.90
Qwen3-Omni-30B-Instruct	2.27	2.34	3.36	4.62	2.86
Qwen3-Omni-30B-Think	2.31	2.55	2.45	4.05	2.66
OmniVinci	2.06	2.17	3.06	4.40	2.64
Audio Flamingo 3	1.77	1.86	2.99	4.03	2.37
Qwen2.5-VL-7B-Instruct	1.53	1.64	2.63	3.83	2.12
Gemma 3n	1.71	1.92	2.48	4.15	2.28
Qwen3-VL-8B-Instruct	1.30	1.41	2.06	3.18	1.76

Several patterns are immediately visible:

Clarity is universally high, Correctness and Completeness are not. Every model scores substantially higher on Clarity (3.18–4.62) than on Correctness (1.30–3.71). This means models produce well-articulated, fluent, and seemingly confident answers even when those answers are factually wrong or incomplete. This is a specific, quantitative demonstration of the "fluent but wrong" failure mode that has been qualitatively observed in LLMs—MMOU's rubric-based scoring makes it measurable and comparable across models.
The gap between Gemini 2.5 Pro and open-source models is larger in open-ended than in MCQ. Gemini 2.5 Pro's overall open-ended score of 3.90 is 1.04 points above Qwen3-Omni-30B-Instruct (2.86). In MCQ, the gap is 64.2% vs. 46.0% = 18.2 percentage points. Normalizing these to comparable scales is not straightforward, but the open-ended evaluation clearly separates models more dramatically: open-source models score in the 2.1–2.9 range (out of 5.0), while Gemini scores near 4.0, suggesting that MCQ format compresses the capability range by providing option scaffolding that helps weaker models more than stronger ones.
Faithfulness varies but is not the primary failure mode. Faithfulness scores range from 2.06 (Qwen3-VL-8B-Instruct) to 3.80 (Gemini 2.5 Pro). While hallucinations exist (scores below 5.0 indicate some unsupported claims), the dominant source of low overall scores is low Correctness and Completeness—models simply cannot produce the right answer or cover all key points, even when they avoid fabricating information.
The thinking variant underperforms in open-ended too. Qwen3-Omni-30B-Think (2.66) scores below Qwen3-Omni-30B-Instruct (2.86), consistent with its MCQ underperformance (19.4% vs. 46.0%). This reinforces that the thinking variant's degradation is not an artifact of MCQ format but reflects genuine capability regression on MMOU tasks.

Skill-wise open-ended analysis (Figure 7). For Gemini 2.5 Pro, the paper reports per-skill breakdown of the four rubric dimensions. Key findings:

Counting is the hardest skill across all dimensions. Correctness: 3.01, Completeness: 3.27, Faithfulness: 3.50—all the lowest or near-lowest across the 13 skills. This is consistent with the MCQ skill analysis (Figure 5) and confirms that counting difficulty is not specific to multiple-choice format.
Holistic Reasoning is the easiest skill. Correctness: 4.24, Completeness: 4.34, Faithfulness: 4.18—the highest across skills. This is the skill where Gemini 2.5 Pro most closely approaches ceiling performance.
Clarity is consistently high across all skills (4.50–4.70), indicating that response quality does not degrade for harder skills—models remain articulate even when they are wrong.

MCQ-to-open-ended transfer analysis. The paper reports a specific conditional analysis: among questions where Gemini 2.5 Pro scored poorly on open-ended correctness (<2 out of 5), it answered 21.1% of those same questions correctly in MCQ format (Section 6). For Qwen3-Omni-Think, this figure is 13.5%; for OmniVinci, 12.9%. This quantifies the degree to which MCQ format "rescues" performance: approximately one-fifth of Gemini 2.5 Pro's correct MCQ answers are questions it cannot answer in open-ended format. The paper further reports that this rescue rate varies by skill: for Gemini 2.5 Pro, Subscene shows the highest rescue rate (29.1%), while Holistic Reasoning shows the lowest (10.5%). This means MCQ format inflates apparent Subscene understanding more than it inflates Holistic Reasoning scores—a finding with implications for benchmark design, as it suggests some skills are more susceptible to MCQ overestimation than others.

Cascaded and Text-Only Baselines

Headline result. Providing detailed audio and visual captions to a strong text-only LLM is insufficient to match end-to-end multimodal models. Cascaded models achieve only 28.1–33.1% (Table 3), substantially below Gemini 2.5 Pro (64.2%) and even below the best vision-only model (44.0%).

Cascaded model results. The paper evaluates two cascaded pipelines:

Qwen3-VL + Qwen3-Omni-Audio captions → Qwen3-235B: 33.1%
Qwen3-VL + Qwen3-Omni-Audio captions → GPT-5.2: 28.1%

These numbers are surprising in two ways: (a) they are lower than Qwen3-235B and GPT-5.2 operating in text-only mode (37.5% and 40.7%, respectively), meaning that adding captions actually hurts performance for the stronger text-only models; (b) they are substantially lower than the caption-generating models themselves (Qwen3-VL-32B at 44.0%, Qwen3-Omni at 35.6% in audio-only mode). This suggests that the captioning process loses critical information—likely fine-grained temporal alignment between audio and visual events, precise spatial relationships, or subtle cross-modal interactions—and that the text-only LLM, when provided with these degraded representations, performs worse than if it relied solely on language priors. The paper interprets this as evidence that "providing a rich contextual audio-visual summary is not sufficient and indicates the need for joint end-to-end cross-modal perception" (Section 5).

Text-only LLM results. Text-only LLMs achieve non-trivial accuracy: GPT-5.2 at 40.7%, Qwen3-235B at 37.5%, GPT-4.1-mini at 33.9%. These numbers are well above the 10.0% random baseline, confirming that some MMOU questions can be answered through language priors alone—the question text and answer options contain enough information for a sophisticated language model to make educated guesses. However, these numbers are well below Gemini 2.5 Pro's 64.2%, confirming that genuine audio-visual perception provides substantial additional signal. The text-only performance also varies by domain (Table 3): GPT-5.2 achieves 45.1% on Animation (its best domain) but only 29.1% on Video Games (its worst), suggesting that some domains have stronger textual priors than others. The paper does not explore which question characteristics predict text-only solvability—this would be a valuable analysis for understanding benchmark bias.

Domain-specific text-only patterns. GPT-5.2's text-only accuracy on Animation (45.1%) exceeds Gemini 2.5 Flash's audio-visual accuracy on the same domain (49.5%) by only 4.4 points, and exceeds several open-source audio-visual models (Qwen2.5-Omni-7B at 20.5%, OmniVinci at 6.3%). This is a red flag: on Animation questions specifically, text-only language priors may be doing much of the work, and the benchmark's cross-modal dependency may be weaker in this domain. The paper does not investigate this domain-specific pattern, leaving open the possibility that certain domains or question types in MMOU are more susceptible to textual shortcuts than others.

Human Performance Baseline

Headline result. Five graduate students (non-authors) achieve 84.3% overall accuracy on MMOU, with per-domain scores ranging from 82.0% (Animation) to 90.0% (News) (Table 3, Appendix D). Annotators were allowed to pause and rewind videos as many times as needed.

Interpretation of human performance. The 84.3% figure serves as the practical ceiling for MMOU: it represents what educated humans can achieve with unlimited time and full access to video content. The fact that humans do not achieve 100% indicates that some MMOU questions are genuinely ambiguous, require fine-grained attention that even motivated humans miss, or have answer options that are difficult to discriminate. The 15.7% human error rate provides context for interpreting model performance: a model scoring 64.2% is not just 20.1 points below humans but is missing a substantial fraction of questions that humans find answerable.

Domain variation in human performance. Humans find News easiest (90.0%) and Animation hardest (82.0%)—a pattern that partially mirrors model performance (Gemini 2.5 Pro: News 61.8%, Animation 58.2%) but with a much smaller spread (8.0 points for humans vs. 13.2 points for Gemini). This suggests that domain difficulty has a larger relative effect on models than on humans, consistent with models having more brittle, domain-specific capabilities.

Limitations of the human baseline. The paper reports that five annotators participated, with scores averaged. It does not report inter-annotator agreement, standard deviation, or confidence intervals. With only five annotators, the human baseline is a point estimate with unknown variance. The paper also does not report whether annotators had domain-specific expertise (e.g., a sports fan evaluating sports questions) that might inflate their scores relative to a more representative human sample. The human baseline should therefore be interpreted as an approximate ceiling rather than a precisely measured gold standard.

Ablation Studies and Robustness Checks

Because MMOU is an evaluation benchmark rather than a method with components to ablate, the paper does not contain traditional ablation studies in the sense of removing components from a proposed system. However, several analyses serve an analogous function by testing the robustness of the benchmark's properties and the reliability of its findings:

Modality-restricted evaluation as a benchmark validity check (Table 3): The inclusion of vision-only, audio-only, and text-only baselines serves as an ablation of the cross-modal dependency claim. The finding that vision-only Qwen3-VL-32B achieves 44.0% and audio-only Audio Flamingo 3 achieves 17.7%—both far below Gemini 2.5 Pro's 64.2%—confirms that MMOU questions genuinely require both modalities. If vision-only models had matched audio-visual models, the benchmark's central design claim (mandatory cross-modal dependency) would be invalidated. The fact that they do not is the paper's primary validation that its construction pipeline achieved its design goal.
Temporal answer position diversification as a design validation (Figure 6): The temporal position sensitivity analysis implicitly validates the annotation protocol's instruction to "diversify" answer evidence positions (Section 3.4, step 4). If all answer evidence clustered in the first two minutes of videos, the temporal degradation finding would be impossible to observe. The fact that models show a clear downward slope (Figure 6) confirms that the benchmark contains sufficient temporal diversity to measure this effect.
Option-order randomization as a position-bias check (Section 4.2): The five-fold option randomization serves as an implicit ablation: if models were exploiting option-position biases, the majority vote across five randomizations would not converge to a consistent answer, and accuracy would be lower than with a fixed option order. The paper does not report accuracy with fixed vs. randomized options, so the magnitude of position bias cannot be quantified from the reported results. However, the protocol itself is a robustness measure against a known failure mode (Liu et al., 2024; Yuan et al., 2025).
Multiple prompt variants as a prompt-sensitivity check (Section 4.2): The paper evaluates each model under "multiple prompt variants" and reports "the best-performing prompt configuration for all MLLMs." This is a robustness check against prompt sensitivity: if a model's performance varies dramatically across prompts, reporting only a single prompt would overstate or understate its capability. By reporting the best-performing prompt, the paper provides an upper bound on what each model can achieve with optimal prompting. However, the paper does not enumerate the prompt variants, report performance across all variants, or quantify the variance—so the reader cannot assess how sensitive models are to prompt engineering or whether the reported differences between models exceed prompt-induced variance.
"None of the above" manipulation as a guessing-strategy check (Section 3.4.4, step 5): The balanced NOTA design (13% NOTA-as-correct, 13% NOTA-as-distractor) serves as an implicit ablation: if models adopted a strategy of always selecting NOTA or never selecting NOTA, they would score below chance on the manipulated subset. The paper does not report model performance specifically on NOTA questions vs. non-NOTA questions, which would directly test whether models are exploiting NOTA-related heuristics. This is a missed opportunity for a diagnostic analysis that the benchmark's design explicitly enables.
Manual audit of cross-modal dependency (Section 3.3): The random 20% sample audit finding "100% answer correctness and 100% strict audio-visual dependency" is a direct validation of the quality control pipeline. This is not an ablation of a model component but an ablation of a benchmark design claim: if the audit had found questions answerable from a single modality, the entire benchmark's validity would be compromised.
Cascaded models as an end-to-end perception check (Table 3): The cascaded baselines (caption → text-only LLM) test whether textual descriptions of modalities are sufficient for MMOU. The finding that cascaded models underperform even text-only LLMs (33.1% vs. 37.5% for Qwen3-235B) is a robustness check on the claim that MMOU requires genuine multimodal perception—textual summaries lose information critical for answering the questions, validating the benchmark's design.
Custom-trained judge model as a rubric consistency check (Appendix G.5): The paper trains a Qwen-3.5-0.8B model to act as an LLM judge for open-ended evaluation, as an alternative to the GPT-5 judge. This serves as a robustness check on the open-ended scoring: if the custom judge and GPT-5 produce substantially different scores, the rubric-based evaluation may be unreliable. However, the paper does not report the custom judge's scores or compare them to GPT-5's scores, making this a stated capability rather than an actual robustness check in the current version. The paper notes that "our custom judge model will be released soon" (Appendix G.5), suggesting this analysis is planned but not yet complete.

Missing ablations that would strengthen the paper:

Per-skill accuracy numbers for Figure 5. The bar chart is presented without numerical labels, preventing precise quantitative comparisons.
NOTA-specific performance analysis. The paper does not report whether models perform differently on questions where NOTA is correct vs. a distractor vs. absent, which would reveal whether models are exploiting or struggling with the NOTA mechanism.
Prompt variance quantification. Reporting accuracy across all prompt variants (not just the best) would enable assessment of prompt sensitivity and statistical reliability of model comparisons.
Inter-annotator agreement for skill tagging. With 11 annotators tagging questions with skill categories (averaging 2.71 skills per question), inter-annotator agreement metrics would indicate how reliably the skill taxonomy can be applied, which affects the validity of per-skill analysis.
Domain-specific cross-modal dependency validation. The paper reports 100% cross-modal dependency on a 20% random sample but does not report whether this holds uniformly across domains. If some domains (e.g., Animation) have weaker cross-modal dependency, the per-domain results in Table 3 would need to be interpreted differently.
Duration bucket vs. temporal position interaction. The paper reports accuracy by duration bucket (Table 3) and by temporal position (Figure 6) separately, but does not analyze their interaction—for example, whether answers at minute 10 show different accuracy in 15-minute vs. 60-minute videos.

Critical Assessment

This section evaluates whether the experimental results support the paper's central claims, identifies genuine weaknesses and missing experiments, and specifies the precise conditions under which claims hold.

Claim 1: "MMOU evaluates joint audio-visual understanding and reasoning on long and complex real-world videos under realistic conditions."

What the experiments demonstrate. The modality-restricted baselines (Table 3) provide strong evidence for this claim. Vision-only Qwen3-VL-32B (44.0%) and audio-only Audio Flamingo 3 (17.7%) perform substantially below the best audio-visual model (Gemini 2.5 Pro, 64.2%), confirming that MMOU questions cannot be solved from a single modality. The manual audit of 20% of questions finding 100% strict audio-visual dependency (Section 3.3) further validates the design. The cascaded model results (28.1–33.1%, Table 3) confirm that textual summaries are insufficient—the benchmark requires genuine multimodal perception.

What the experiments do not demonstrate. The claim that questions require "joint" reasoning implies that the model must simultaneously integrate audio and visual information, not merely access both modalities. The modality-restricted baselines show that both modalities are necessary, but they do not show that they must be integrated—a model could, in principle, process audio and visual streams independently, answer sub-questions from each, and combine the results without genuine cross-modal attention. The paper does not include baselines that test this distinction, such as late-fusion models that process modalities separately and combine only at the decision level. The cascaded baselines are a partial test (captions are generated separately and fused), but they involve information loss through captioning, so their poor performance may reflect caption quality rather than the necessity of joint processing.

Specific conditions. The claim holds most strongly for the 20% audited subset (where cross-modal dependency was manually verified). For the remaining 80%, cross-modal dependency is asserted based on the annotation protocol and filtering process but has not been independently verified. The domain-specific variation in text-only performance (GPT-5.2 achieves 45.1% on Animation, only 4.4 points below Gemini 2.5 Flash's audio-visual score of 49.5%) suggests that some domains may have weaker cross-modal dependency than others, even if the overall benchmark meets the standard. The claim of "realistic conditions" is supported by the use of web-collected videos with natural audio, but the paper does not quantify the realism (e.g., background noise levels, recording quality distributions, presence of editing artifacts) or test whether model performance degrades under less realistic conditions.

Claim 2: "The best closed-source model achieves only 64.2% accuracy, with open-source models performing substantially worse (46.8%), revealing significant gaps in current multimodal reasoning capabilities."

What the experiments demonstrate. Table 3 directly supports these numbers. The 17.4-percentage-point gap between Gemini 2.5 Pro (64.2%) and MiniCPM-o 4.5 (46.8%) is well-measured across 15,000 questions. The human baseline of 84.3% provides a ceiling that makes the 64.2% figure interpretable as a 20.1-point gap to human-level performance.

What the experiments do not demonstrate. The claim that these numbers reveal "significant gaps in current multimodal reasoning capabilities" is an interpretation, not a direct measurement. Several factors could contribute to the 64.2% figure beyond reasoning capability: (a) the benchmark's inherent difficulty (questions may require extremely fine-grained attention that humans with unlimited time achieve 84.3% on, suggesting the benchmark itself is challenging); (b) the 10-option format, which increases cognitive load compared to standard 4-option MCQ; (c) the NOTA manipulation, which specifically penalizes models that rely on elimination strategies. The 64.2% is a function of both model capability and benchmark design; the paper does not disentangle these.

Missing analysis. The paper does not report model performance stratified by question characteristics that might predict difficulty independently of reasoning requirements—for example, question length, answer length, number of skill tags, or specific skill combinations. Such an analysis would help distinguish whether low accuracy reflects reasoning failures or more mundane factors like processing long question texts.

The open-source/closed-source gap interpretation. The paper treats the 64.2% vs. 46.8% gap as evidence of a capability deficit in open-source models. An alternative interpretation is that Gemini 2.5 Pro benefits from training on YouTube videos (the paper explicitly hypothesizes this in Section 5: "we hypothesize the relatively strong performance of Gemini to pre-training on YouTube videos"), which may give it an unfair advantage on a benchmark sourced from publicly available online platforms. If Gemini has effectively "seen" some of the test videos or similar content during training, its advantage may reflect memorization or familiarity rather than superior reasoning. The paper acknowledges this possibility in Section 7 ("Our benchmark is derived from publicly available web videos, which may introduce content biases and potential train–test leakage in closed and open-weight models") but does not attempt to quantify or control for it. This is a significant threat to the validity of cross-model comparisons, particularly between models with unknown training data (Gemini) and models with publicly documented training data.

Claim 3: "We conduct an in-depth analysis of model predictions, uncovering systematic failure modes."

What the experiments demonstrate. The paper provides three specific analyses that support this claim: (a) the skill-wise performance breakdown (Figure 5), which shows that Counting, Temporal Understanding, and Needle-in-the-Haystack are consistently the hardest skills; (b) the temporal position sensitivity analysis (Figure 6), which shows accuracy degradation with later answer evidence; and (c) the open-ended evaluation (Table 4, Figure 7), which shows that MCQ format inflates apparent understanding and that models are fluent but frequently incorrect.

What the experiments do not demonstrate. "Systematic failure modes" suggests that the paper identifies why models fail—the mechanisms underlying the observed patterns. The analyses are primarily descriptive (showing that accuracy varies across skills, temporal positions, and evaluation formats) rather than explanatory (showing why Counting is hard or why late-temporal-position evidence degrades accuracy). Potential mechanistic explanations—attention dilution in long contexts, inability to track discrete event instances, modality-specific processing bottlenecks—are suggested but not tested. The paper does not, for example, analyze model attention patterns to determine whether late-temporal degradation is due to attention fading or to interference from intervening content, or whether counting failures are due to perception errors (not detecting events) or enumeration errors (detecting events but miscounting them).

Missing failure mode analyses. Several analyses that would strengthen the "systematic failure modes" claim are absent:

Error categorization: Do models fail by selecting plausible distractors (indicating they understand the video but make fine-grained discrimination errors) or by selecting out-of-context distractors (indicating they fundamentally misunderstand the video)? The balanced distractor design enables this analysis but it is not reported.
Confusion matrices: Which skills are models most likely to confuse? If a question is tagged with both "Counting" and "Temporal Understanding," do models fail on one dimension while succeeding on the other?
Qualitative error analysis: The paper includes examples of skill types (Tables 7–8) but does not present examples of model failures with analysis of what went wrong—analogous to the qualitative failure analysis in Appendix M of the scaling-test-time-compute paper, which showed concrete examples of verifier over-optimization producing degenerate outputs.
NOTA-specific failures: When models fail on NOTA-correct questions, do they select a specific incorrect option (suggesting they are confidently wrong) or distribute errors across options (suggesting uncertainty)? When models incorrectly select NOTA on NOTA-as-distractor questions, is it because they cannot identify any correct option or because they adopt a NOTA-biased strategy?

Claim 4: "Current models frequently fail to apply even fundamental skills in long videos."

What the experiments demonstrate. Figure 5 shows that skills described as "basic and essential" (temporal understanding, counting, needle-in-the-haystack reasoning) are indeed among the lowest-accuracy categories for all models. The temporal position degradation (Figure 6) shows that performance decreases as videos get longer or answer evidence appears later. The >30-minute duration bucket in Table 3 shows catastrophic degradation for some models (MiniCPM-o 4.5 drops to 9.1%).

What the experiments do not demonstrate. The claim that these skills are "fundamental" is an assertion about what capabilities should be easy for MLLMs, not an empirical finding. The paper does not establish that these skills are easier for humans (though the human baseline of 84.3% suggests humans find MMOU questions answerable overall) or that models succeed at these skills in short-video contexts but fail specifically in long-video contexts. The latter comparison would require evaluating the same skills on short vs. long videos while controlling for other factors—an analysis the paper does not perform.

The "long videos" specificity. The paper's title and framing emphasize long videos as the distinctive challenge, but the experimental design does not isolate video length as a causal factor. The primary evidence for length-specific difficulty is the duration-bucket breakdown (Table 3) and the temporal position analysis (Figure 6). However, the duration buckets confound video length with content type (longer videos may be systematically different in domain, production quality, or complexity), and the temporal position analysis does not control for total video duration. A model that fails on a question at minute 40 of a 45-minute video might succeed on the same question at minute 40 of a 90-minute video (where the answer is relatively early) or fail at minute 5 of a 10-minute video (where context is limited). Without these comparisons, "long videos" as the explanatory variable remains an interpretation rather than a demonstrated causal factor.

Additional weaknesses in experimental design

No statistical significance testing. The paper reports accuracy numbers without confidence intervals or hypothesis tests. With 15,000 questions, small differences in accuracy can be statistically significant, but the paper provides no way to determine whether, for example, MiniCPM-o 4.5's 46.8% is reliably different from Qwen3-Omni-30B-A3B-Instruct's 46.0% (a 0.8-point gap) or whether this difference is within sampling error. This is particularly problematic for per-skill comparisons where the number of questions per skill is smaller (Figure 3d shows skill distributions, but exact counts per skill are not reported).
Single evaluation protocol without sensitivity analysis. All MCQ results are based on a single evaluation protocol (five-fold option randomization, best prompt variant). The paper does not report how sensitive results are to the number of randomizations (would 10-fold give different results?), to the specific prompt variants tested (are some models more prompt-sensitive than others?), or to the parsing method for extracting predicted options (are parsing errors inflating or deflating any model's score?). These are standard robustness checks for benchmark evaluation that would strengthen confidence in the reported rankings.
No analysis of model calibration or confidence. The paper reports only accuracy—whether the model selected the correct option. It does not report whether models are well-calibrated (do they assign high confidence to correct answers and low confidence to incorrect ones?), whether they exhibit option-position biases before randomization (does the majority vote meaningfully change answers?), or whether they show systematic patterns in their errors (are certain distractor types more likely to be selected?). These analyses would enrich the "systematic failure modes" claim considerably.
The human baseline is underpowered and underdocumented. Five annotators is a small sample for estimating human performance with precision. The paper does not report inter-annotator agreement, standard deviation, or whether annotators had domain expertise that might inflate scores. The 84.3% figure should be accompanied by a confidence interval—with only five annotators, the standard error could be several percentage points, making the 20.1-point human-model gap less precisely estimated than the paper implies.
Domain and subcategory imbalances are not discussed as potential confounds. News (4,221 questions) and Sports (3,484 questions) dominate the benchmark, while Animation (184 questions) and Music (630 questions) are sparsely represented. The overall accuracy numbers are therefore weighted averages dominated by News and Sports performance. If models are systematically better or worse on these overrepresented domains, aggregate accuracy may not reflect generalized audio-visual reasoning capability. The paper reports per-domain accuracy but does not discuss how domain imbalance affects the interpretation of aggregate scores.
The GPT-5 judge for open-ended evaluation is proprietary and non-reproducible. The open-ended evaluation relies on GPT-5 as an LLM judge, but GPT-5's scoring behavior is a black box—it may have its own biases, calibration issues, or inconsistencies that the paper does not characterize. The custom-trained judge (Qwen-3.5-0.8B) is mentioned as an alternative but its scores are not reported or compared to GPT-5's. This means the open-ended evaluation results are not independently reproducible without access to GPT-5, and the extent to which they reflect genuine response quality versus judge-specific preferences is unknown.
No longitudinal or version-specific analysis. The paper evaluates a snapshot of models available at the time of writing. It does not discuss how rapidly MMOU scores might saturate (will the next generation of models close the 20.1-point human gap?), whether there are diminishing returns to scale (do larger models within the same family show predictable improvements?), or which benchmark design features create the most headroom for future progress. The paper positions MMOU as infrastructure for tracking progress but does not calibrate its difficulty trajectory.

6. Limitations and Trade-offs

Limitation 1: Train–Test Leakage from Web-Sourced Videos Cannot Be Ruled Out and May Inflate Closed-Source Model Scores

The assumption or constraint. MMOU is constructed entirely from publicly available web videos (YouTube), and the paper evaluates both open-source and proprietary closed-source models whose training data is unknown. The authors explicitly acknowledge this in Section 7:

"Our benchmark is derived from publicly available web videos, which may introduce content biases and potential train–test leakage in closed and open-weight models."

They further hypothesize in Section 5 that Gemini 2.5 Pro's strong performance "may be attributable to pre-training on YouTube videos." This is not merely a generic caveat—it is a specific concern that the benchmark's source material overlaps with the training distribution of the models being evaluated, and that the degree of overlap is unknowable for closed-source systems.

The consequence. If Gemini 2.5 Pro (or any other model) has been trained on some of the same YouTube videos that appear in MMOU, its 64.2% accuracy may partly reflect memorization of video content rather than genuine cross-modal reasoning capability. The model might recall that "in this specific video, at this specific timestamp, this event occurred" without performing the audio-visual integration that the benchmark intends to measure. This would mean that the 20.1-percentage-point gap between Gemini 2.5 Pro (64.2%) and the human baseline (84.3%) overestimates the system's reasoning capability—the true gap for unseen video content could be larger. More problematically, it means that cross-model comparisons between proprietary and open-source models are confounded: Gemini 2.5 Pro's 17.4-point advantage over MiniCPM-o 4.5 may reflect training data access rather than architectural or algorithmic superiority. The paper's central claim—that MMOU reveals "substantial performance gaps" in multimodal reasoning—becomes difficult to interpret if we cannot distinguish reasoning gaps from memorization gaps.

What evidence exists in the paper. The paper provides no direct measurement of train–test leakage. There is no analysis of whether model accuracy correlates with video popularity (more-viewed YouTube videos are more likely to appear in training corpora), no held-out verification that models fail on videos from sources known to be excluded from training, and no comparison between closed-source models with different training data policies. The domain-level results in Table 3 provide circumstantial but inconclusive evidence: Gemini 2.5 Pro achieves 71.4% on Academic Lectures—a domain where YouTube content is abundant and likely included in training—but also achieves 71.0% on Pranks, a domain where training data may be less systematically curated. The paper's cascaded model analysis (28.1–33.1% for caption-then-answer pipelines, Table 3) does not address leakage because captions would not capture the memorized video-level information that a leaked model might exploit.

Mitigation status. The paper acknowledges the limitation explicitly (Section 7) but makes no attempt to measure, bound, or control for it. Section 7 proposes future work including "continuously expanding the benchmark"—presumably to include content less likely to appear in training corpora—but no concrete steps are taken in the current version. This is a fundamental limitation for any benchmark constructed from publicly available web content, and it is particularly consequential for MMOU because (a) YouTube is a known source of training data for large multimodal models, (b) the benchmark uses full-length videos rather than synthetically generated content that is guaranteed novel, and (c) the best-performing model is a proprietary system whose training data is opaque. Until leakage can be ruled out or quantified, MMOU's scores for closed-source models should be interpreted as upper bounds on reasoning capability that may be inflated by an unknown amount.

Limitation 2: The Benchmark Operates at a Single Snapshot in Time With No Built-In Mechanism for Evolving Difficulty or Preventing Saturation

The assumption or constraint. MMOU is a static benchmark: 15,000 fixed questions on 9,038 fixed videos, with no process for adding new questions, retiring saturated ones, or dynamically adjusting difficulty. The paper positions MMOU as infrastructure for tracking progress in multimodal understanding, stating that "MMOU emphasizes joint audio-visual perception across a diverse set of reasoning skills that are central to real-world understanding" (Section 7), but the benchmark's fixed nature means that as models improve—and the paper already documents rapid progress from open-source models (19.4% for the worst omni-modal model, 46.8% for the best)—the benchmark will saturate. Once models approach or exceed the human ceiling of 84.3%, MMOU will cease to discriminate between systems, and the remaining headroom (the 20.1-point gap between Gemini 2.5 Pro and humans) will close over an unknown but potentially short timeframe.

The consequence. The value of a benchmark as a research instrument depends on its ability to differentiate between models over time. A benchmark that saturates rapidly—where all frontier models cluster near the ceiling within one or two model generations—provides little signal for further progress and may even mislead by suggesting that the underlying capability (omni-modal video understanding) has been solved when in fact the benchmark has merely been exhausted. The paper does not provide a difficulty calibration that would allow the community to estimate how quickly saturation will occur. We do not know, for example, whether scaling model size, improving training data, or developing better cross-modal fusion mechanisms will close the gap—and at what rate. The finding that even the best model achieves only 64.2% suggests substantial headroom remains, but the paper's own evidence on model scaling is mixed: Qwen3-Omni-30B-A3B-Instruct (46.0%) outperforms Qwen2.5-Omni-7B (31.3%) by 14.7 points within the same model family across one generation, suggesting rapid progress is possible.

What evidence exists in the paper. Table 3 shows the performance spread across models: 19.4% (Qwen3-Omni-30B-A3B-Thinking) to 64.2% (Gemini 2.5 Pro). The gap between the best open-source and best closed-source model (17.4 points) is large enough to suggest headroom, but the paper provides no analysis of which questions contribute to this headroom—are there question categories where all models fail, providing durable headroom, or is the gap distributed uniformly such that incremental improvements will close it quickly? The skill-wise analysis (Figure 5) shows that Counting and Needle-in-the-Haystack are hard for all models, which may provide some durable difficulty, but the paper does not quantify the ceiling for these skills or project how rapidly they might improve. The paper's own open-source baselines include models from different generations (Qwen2.5-Omni vs. Qwen3-Omni), but the paper does not analyze the rate of improvement across generations to forecast saturation timelines.

Mitigation status. Section 7 states that future work includes "continuously expanding the benchmark to incorporate emerging concepts and scenarios" and "extending coverage beyond curated online content to include unstructured real-world videos, such as egocentric or driving scenarios." These are aspirational statements without concrete mechanisms. The paper does not describe a process for adding new questions, a schedule for benchmark updates, a difficulty tiering system (e.g., separating questions that current models can answer from those they cannot), or a method for detecting and retiring saturated questions. The benchmark is released as a static artifact, and its long-term utility as a progress-tracking instrument depends entirely on community maintenance practices that the paper does not establish. This is not unique to MMOU—most benchmarks face the saturation problem—but it is particularly salient given (a) the rapid pace of multimodal model development, (b) the relatively modest human ceiling of 84.3%, and (c) the paper's framing of MMOU as infrastructure for a research program rather than a one-time evaluation.

Limitation 3: The Open-Ended Evaluation Protocol Is Not Self-Contained—It Depends on an Unvalidated, Proprietary LLM Judge With Unknown Reliability

The assumption or constraint. The paper's open-ended evaluation (Section 6, Table 4, Figure 7) uses GPT-5 as an LLM judge to score model responses on a four-dimensional rubric (Correctness, Completeness, Faithfulness, Clarity, each on a 1–5 scale). GPT-5 receives the question, ground-truth answer, a detailed audio-visual caption describing the video content, and the model response to evaluate. The paper implicitly assumes that GPT-5's judgments are accurate, consistent, and unbiased—that its rubric scores meaningfully reflect the quality of model responses rather than artifacts of the judge's own limitations. The paper states that a custom-trained judge (Qwen-3.5-0.8B) has been developed as an alternative, but explicitly notes that its scores are not reported: "our custom judge model will be released soon" (Appendix G.5).

The consequence. Every quantitative claim derived from the open-ended evaluation is contingent on GPT-5 being a reliable evaluator. If GPT-5 is lenient (tending to assign higher scores than human evaluators would), the reported scores overstate model capability. If GPT-5's judgments are noisy or inconsistent across questions, the rankings in Table 4 may be unreliable—a model scoring 2.86 (Qwen3-Omni-30B-Instruct) might not be genuinely better than one scoring 2.66 (Qwen3-Omni-30B-Think) if judge variance exceeds the 0.20-point difference. If GPT-5 exhibits systematic biases—favoring certain response styles, penalizing brevity, misunderstanding domain-specific content—the per-skill analysis in Figure 7 may reflect judge preferences rather than model capabilities. The paper's key methodological claim—that "MCQ performance may overestimate true understanding" (Section 6)—rests on the validity of the open-ended evaluation as ground truth. If the open-ended scores themselves are unreliable, the comparison with MCQ scores loses its evidentiary force.

What evidence exists in the paper. The paper provides essentially no validation of the GPT-5 judge. There is no comparison between GPT-5 scores and human evaluator scores on the same responses (which would establish judge alignment), no measurement of inter-judge agreement between GPT-5 and the custom judge (which would establish consistency), no analysis of judge score distributions (are scores clustered at certain values? do they exhibit ceiling or floor effects?), and no sensitivity analysis of the rubric weighting (Equation 1) to determine whether the 0.5 weight on Correctness drives the overall ranking. The paper reports that Clarity scores are consistently high across all models and skills (4.03–4.70, Table 4 and Figure 7), which could indicate that models are genuinely clear communicators, or that GPT-5's Clarity judgments are insufficiently discriminating—the paper cannot distinguish these interpretations without human-validated Clarity scores. The detailed rubric and scoring protocol (Appendix G) is well-specified, but specification does not guarantee execution quality.

Mitigation status. The paper acknowledges the need for judge validation implicitly by developing the custom-trained Qwen-3.5-0.8B judge (Appendix G.5), described as an effort toward "scalable and reproducible open-ended scoring without relying on a proprietary API." However, because the custom judge's scores are not reported, this remains a promissory note rather than a robustness check. Section 7's future work includes "developing more robust evaluation protocols for open-ended audio-visual QA," which directly acknowledges the limitation. But in the current version, all open-ended conclusions—the 3.90 overall score for Gemini 2.5 Pro, the 21.1% MCQ rescue rate for open-ended failures, the per-skill rubric breakdown in Figure 7—are unvalidated measurements from a single, proprietary, uncalibrated instrument. A practitioner deciding whether to trust MMOU's open-ended evaluation as a measure of their model's capability has no basis for assessing the reliability of the numbers beyond faith in GPT-5's judgment.

Limitation 4: Difficulty Estimation, Temporal Position Analysis, and Domain Breakdowns Confound Multiple Variables—Video Length, Content Type, and Answer Position Are Not Disentangled

The assumption or constraint. The paper analyzes model performance along several axes: video duration buckets (<5, 5–10, 10–20, 20–30, >30 minutes) in Table 3, temporal position of answer evidence in Figure 6, video domains (10 categories) in Table 3, and skill categories (13 types) in Figure 5. Each analysis treats its axis of interest as the explanatory variable, but the axes are systematically correlated in ways the paper does not disentangle. Longer videos are not random supersets of shorter videos—they differ in domain composition (academic lectures and sports matches tend to be long; prank videos tend to be short), production characteristics, and the types of questions that can be asked about them. Later temporal positions occur disproportionately in longer videos and in specific domains (a 40-minute answer position can only occur in a 40+ minute video). Skill categories are not uniformly distributed across domains (Counting questions may be more natural in sports videos; Audio-Visual Stitching questions more natural in film content).

The consequence. When the paper reports that accuracy degrades for >30-minute videos (Table 3) or for late temporal positions (Figure 6), we cannot determine whether the degradation is caused by video length, by temporal position per se, by the types of content that happen to occupy those duration/position buckets, or by interactions among these factors. For example, Gemini 2.5 Pro drops from 66.2% accuracy at 5–10 minutes to 58.5% at >30 minutes (an 7.7-point decline). Is this because Gemini cannot process 30+ minutes of context, or because >30-minute videos are disproportionately sports matches and academic lectures—domains where Gemini scores 61.2% and 71.4% respectively, which would actually predict higher not lower accuracy—or because the answer evidence in >30-minute videos tends to be positioned later, and the temporal position effect (Figure 6) is the true driver? The paper cannot answer this question with the reported analyses.

Similarly, the counterintuitive finding that accuracy increases from the <5-minute bucket (62.2%) to the 5–10-minute bucket (66.2%) for Gemini 2.5 Pro could reflect that very short videos lack sufficient context for some question types, or that short videos are disproportionately from domains where Gemini performs poorly, or simply sampling noise given the per-bucket question counts (which are not reported in Table 3). Without controlling for domain, skill, and temporal position when analyzing duration effects—or vice versa—all of the paper's difficulty-axis analyses are confounded.

What evidence exists in the paper. The paper provides all the data necessary for a controlled analysis—each question has a domain tag, skill tags, temporal position annotations, and a video duration—but reports only marginal analyses along each axis independently. Table 3 reports accuracy by domain (columns 2–11) and by duration (columns 12–16) but does not report, for example, accuracy by domain within each duration bucket or accuracy by duration within each domain. Figure 6 plots accuracy against answer position but does not condition on video duration—we cannot see whether answers at minute 20 show different accuracy in 25-minute videos vs. 60-minute videos. Figure 5 plots accuracy by skill but does not break down by domain or duration—we cannot see whether Counting is equally hard in sports videos (where events are rapid and numerous) vs. academic lectures (where counting questions may be less frequent).

Mitigation status. The paper does not acknowledge this as a limitation or attempt to disentangle the confounded variables. The temporal position analysis (Figure 6) is presented as revealing "a fundamental limitation in long-horizon temporal reasoning and context retention" (Section 6), implying that temporal position is the causal factor. But duration is an equally plausible explanation for the same data pattern, and the paper provides no analysis that distinguishes them. The infrastructure for controlled analysis exists in the dataset (the paper has all the necessary annotations), but the analyses that would resolve the confounding are not performed. This limits the diagnostic value of MMOU: the benchmark can tell you that your model degrades on long videos or late answer positions, but not why, and without knowing why, a model developer cannot determine whether to invest in longer context windows, better temporal attention, domain-specific training data, or some other intervention.

The assumption or constraint. The paper's central design principle is that every MMOU question requires both audio and visual information—removing either modality should make the question unanswerable. This is enforced through the annotation protocol (annotators must create questions requiring simultaneous audio and visual understanding), the quality control process (questions solvable from a single modality are removed), and validated through manual audit (a random 20% sample achieves 100% audio-visual dependency, Section 3.3). The paper treats this as establishing that MMOU measures "joint audio-visual perception and reasoning" and "genuine cross-modal integration."

However, necessity of both modalities is not the same as integration of both modalities. A model could process the audio stream and visual stream entirely independently—transcribing speech, detecting objects and actions, tracking temporal events in each modality separately—and then combine the results at a late stage through a reasoning module that never computes cross-modal attention or learns cross-modal representations. Such a model would have access to both modalities (satisfying the necessity condition) but would not perform the kind of integrated, cross-modal reasoning that humans do when, for example, connecting a spoken reference to a visually identified entity at a precise temporal moment. The paper's central claim that MMOU evaluates "omni-modal understanding" implies, but does not test, that models succeed through integration rather than parallel unimodal processing followed by textual reasoning.

The consequence. If models can solve MMOU questions through parallel unimodal processing, the benchmark is measuring a weaker capability than the paper claims. A system that runs state-of-the-art ASR and visual recognition independently and then applies a text-only LLM to reason over the combined textual outputs might achieve respectable accuracy without ever computing a cross-modal representation. The cascaded baselines (Table 3) partially test this: the Qwen3-(VL+O-A) + LLM pipelines achieve only 28.1–33.1%, suggesting that naive caption-then-reason approaches fail. But a more sophisticated parallel-processing system—one that preserves fine-grained temporal alignment between modalities, passes structured representations rather than natural language captions, or uses the LLM to iteratively query each modality—might substantially outperform the caption-based cascades without ever truly integrating audio and visual signals at the representational level. The paper provides no diagnostic for distinguishing between integrated and parallel processing, so a model's MMOU score conflates the two.

What evidence exists in the paper. The cascaded baselines are the closest the paper comes to testing this distinction, but they are a weak test because (a) they introduce information loss through captioning, so poor performance may reflect caption quality rather than the necessity of integration, and (b) the text-only LLMs receiving the captions perform worse than text-only LLMs with no captions at all (GPT-5.2: 40.7% text-only vs. 28.1% with captions), suggesting the captioning pipeline is actively harmful rather than merely insufficient. The paper does not evaluate a late-fusion model that processes modalities separately and combines at a decision level, a model that uses cross-modal attention but is ablated to remove it, or any other diagnostic that would isolate the contribution of cross-modal integration specifically. The modality-restricted baselines (vision-only, audio-only) show that both modalities are necessary, but necessity can be satisfied by parallel processing.

Mitigation status. The paper does not acknowledge this as a limitation or propose diagnostics for distinguishing integrated from parallel cross-modal processing. Section 7's future work mentions "extending coverage beyond curated online content" but does not address the measurement validity question of whether MMOU's scores reflect genuine integration. This is a subtle but important limitation: the benchmark's design establishes a lower bound on what models must do (use both modalities), but not an upper bound on how they use them. A model that achieves 64.2% on MMOU might genuinely integrate audio and visual information at the representational level, or it might be a sophisticated parallel processor that treats each modality as a separate information channel. The paper's evaluation framework cannot distinguish these, and its claims about "omni-modal understanding" should be interpreted accordingly—as measuring a capability that requires multiple modalities without specifying the mechanism by which they are combined.

Limitation 6: The Cost and Reproducibility of Evaluation Are Not Addressed—Running the Full Benchmark Requires Substantial Compute and API Access That May Limit Adoption

The assumption or constraint. MMOU is a large-scale benchmark: 15,000 questions on 9,038 videos averaging 711.6 seconds (nearly 12 minutes) each. Evaluating a single model requires processing approximately 1,800 hours of video content (9,038 videos × 711.6 seconds / 3600), plus audio processing, plus running inference through the model for each of the 15,000 questions. For models that process video at native frame rates with audio, this is computationally intensive—the paper does not report wall-clock evaluation time, FLOPs, or cost estimates for any model, but a back-of-the-envelope calculation suggests that evaluating a single open-source multimodal model on MMOU could require days of GPU time, and evaluating proprietary models requires API access and associated costs. The paper also does not discuss whether the benchmark can be subsetted (e.g., evaluating on a random 1,000-question sample) while preserving statistical reliability.

The consequence. The computational cost of full MMOU evaluation creates several practical barriers. First, it limits who can run the benchmark: academic groups with constrained compute budgets may be unable to evaluate their models on the full benchmark, creating a de facto barrier where only well-resourced industry labs can participate in the MMOU evaluation ecosystem. Second, it limits the frequency of evaluation: if evaluating a model takes days of GPU time, rapid iteration during model development becomes impractical—teams cannot use MMOU as a development-phase diagnostic, only as a final evaluation. Third, it complicates reproducibility: if different groups evaluate the same model with slightly different preprocessing, frame sampling, or prompting, the reported scores may not be comparable, and the cost of evaluation makes it difficult to systematically test sensitivity to these factors. Fourth, for API-based proprietary models, the cost of evaluating on 15,000 long videos (roughly 1,800 hours of video input tokens) may be prohibitive—the paper evaluated Gemini 2.5 Pro and Flash, but does not report the API cost, and other researchers attempting to reproduce or extend these results would need to budget for substantial API expenditure.

What evidence exists in the paper. The paper provides no analysis of evaluation cost, no recommendations for subset evaluation, and no discussion of reproducibility infrastructure (e.g., standardized evaluation containers, frame sampling protocols, prompt templates). The paper evaluates 20+ models, which represents a substantial computational investment by the authors, but the cost is borne by a well-resourced industrial research lab (NVIDIA) and is not quantified for the benefit of the community. The paper does not report whether all models were evaluated under identical video preprocessing (frame rate, resolution, audio sampling rate) or whether these parameters were tuned per-model—an omission that affects reproducibility.

Mitigation status. The paper does not acknowledge evaluation cost as a limitation or provide guidance for cost-effective evaluation. Section 7's future work mentions "developing more robust evaluation protocols" but does not explicitly address scalability or cost. The benchmark is released on Hugging Face (indicated in the project URL), which is positive for accessibility, but the paper does not describe whether the release includes standardized evaluation scripts, preprocessing pipelines, or recommended subset configurations. For a benchmark that the paper positions as infrastructure for the community, the absence of practical deployment guidance—how to run it, what it costs, how to get statistically reliable results without evaluating on all 15,000 questions—is a significant gap. Practitioners deciding whether to adopt MMOU as an evaluation tool need to know not just whether it measures the right capabilities, but whether they can afford to run it. The paper provides no information to answer that question.

7. Implications and Future Directions

How This Work Changes the Landscape

MMOU does not introduce a new model architecture, training paradigm, or inference strategy. It is a measurement instrument—a benchmark—and its impact on the field must be understood in those terms: as a change in what we measure, how we measure it, and what those measurements reveal about the capabilities we thought we had.

This is a diagnostic reframing, not a paradigm shift. The paper's primary contribution is to make visible a capability gap that existing evaluation frameworks systematically obscured. Before MMOU, the field could reasonably believe that strong performance on vision benchmarks (Video-MME, MVBench) plus strong performance on audio benchmarks (MMAU, MMAU-Pro) implied strong performance on joint audio-visual tasks. That belief was never rigorously tested; the benchmarks simply did not demand cross-modal integration. MMOU's design enforces mandatory audio-visual dependency as a necessary condition for every question—and the result, validated through the modality-restricted baselines in Table 3 (vision-only Qwen3-VL-32B at 44.0%, audio-only Audio Flamingo 3 at 17.7%), is that removing either modality makes the benchmark essentially unsolvable at human-competitive levels. This reframes "multimodal understanding" from an additive property (vision capability + audio capability) to an integrative property (vision ∪ audio → understanding that neither alone supports), and in doing so, reveals that the additive assumption was wrong—or at least, that it systematically overestimated what current models could do.

The reframing matters because it changes what the field optimizes for. If benchmarks reward unimodal performance, model developers optimize unimodal encoders and training data, and the resulting models may score well on existing evaluations while lacking genuine cross-modal reasoning. MMOU creates an incentive gradient toward integration: a model that processes audio and visual streams independently but expertly will underperform a model with weaker unimodal components but stronger cross-modal fusion, all else equal. The paper's cascaded baseline results (Table 3: Qwen3-(VL+O-A) + Qwen3-235B achieves only 33.1%, and the same pipeline with GPT-5.2 drops to 28.1%—below text-only GPT-5.2 at 40.7%) make this concrete: state-of-the-art unimodal captioning plus state-of-the-art text reasoning is not merely insufficient, it is counterproductive, producing worse results than the text-only LLM operating without any captions at all. The information loss from decoupled processing is not recoverable by stronger text reasoning. This is a specific, falsifiable claim about the insufficiency of late-fusion approaches that prior benchmarks could not have generated.

MMOU reconciles a tension in the multimodal evaluation literature. Prior omni-modal benchmarks (OmniBench, AV-Odyssey, WorldSense, DailyOmni, OmniVideoBench) included both audio and visual modalities but did not enforce mandatory cross-modal dependency as a design requirement. The consequence, documented by Chen et al. (2025) and cited in the paper, is that these benchmarks "often limit questions to a single dominant modality" and "fall short of requiring genuine cross-modal reasoning" (Section 3.3). The field therefore faced a tension: multiple benchmarks claimed to evaluate audio-visual understanding, but models achieved widely varying scores on them, and it was unclear whether this reflected genuine capability differences or differences in how much cross-modal reasoning each benchmark actually demanded. MMOU resolves this by making cross-modal dependency a verified property (100% strict audio-visual dependency on the audited 20% sample, Section 3.3) rather than an aspirational design goal. A model that scores 64.2% on MMOU is not simply good at "video questions"—it is good at questions that cannot be answered without integrating audio and visual evidence. This provides a calibrated reference point against which other benchmarks can be compared: if a model scores 80% on Benchmark X but 50% on MMOU, the gap quantifies how much Benchmark X's scores reflect unimodal shortcuts rather than genuine cross-modal reasoning.

The skill taxonomy creates a new evaluation ontology. The 13-category skill decomposition (Tables 7–8) is not merely a finer-grained version of existing taxonomies—it defines categories that exist only at the intersection of modalities. "Audio-visual stitching" (understanding narrative continuity across edited segments), "tracking spurious correlations" (identifying surprising audio-visual coincidences that language priors cannot predict), and "referential grounding" (linking audio references to visual entities) are skills that have no analog in vision-only or audio-only benchmarks. By tagging every question with these skill labels (averaging 2.71 skills per question, Table 2), MMOU provides the infrastructure for differential diagnosis: a model developer can learn not just that their model scores 46.8% overall, but that it achieves near-ceiling on holistic reasoning while collapsing on counting and temporal understanding (Figure 5). This shifts evaluation from a single-number ranking exercise to a capability profiling exercise, which is more actionable for model improvement. The stability of the skill difficulty ranking across open and closed models (Figure 5: all three plotted models show Counting as the hardest or near-hardest skill and Context/Holistic Reasoning as the easiest) suggests that the taxonomy captures something fundamental about audio-visual reasoning difficulty—something that scaling alone does not overcome.

Long-form evaluation becomes a first-class measurement axis. The paper's temporal position sensitivity analysis (Figure 6) introduces a diagnostic that is distinct from, and more informative than, the duration-based analyses common in prior work (including MMOU's own Table 3 duration buckets). The finding that accuracy degrades steadily with answer evidence position—and that this occurs even for the best model (Gemini 2.5 Pro drops from ~55–60% at early positions to ~35–40% at late positions)—establishes that context retention and temporal retrieval, not just total video length, are the operative bottlenecks. This has architectural implications: it suggests that research investment should target mechanisms for temporal attention and retrieval from earlier video segments, rather than simply scaling context windows. It also provides a new evaluation protocol: future video understanding benchmarks can (and, the paper implicitly argues, should) annotate answer evidence timestamps and report temporal-position-conditioned accuracy as a standard diagnostic.

The open-ended vs. MCQ calibration is a methodological contribution. The paper's conditional analysis—among questions where Gemini 2.5 Pro scored poorly on open-ended correctness (<2/5), it answered 21.1% correctly in MCQ format (Section 6)—provides a specific, reusable methodology for quantifying how much MCQ format inflates apparent understanding. The finding that this inflation varies by skill (Subscene: 29.1% rescue rate; Holistic Reasoning: 10.5%) means that benchmark designers cannot simply apply a uniform correction factor—the degree of overestimation depends on what is being measured. This is a caution that applies to all MCQ benchmarks, not just MMOU, and it provides a template for future benchmarks to include built-in open-ended calibration subsets that quantify the MCQ-to-open-ended gap.

Research directions that become more attractive: (1) Cross-modal fusion mechanisms that operate at fine temporal granularity, since the cascaded baselines show that late fusion through captioning fails catastrophically. (2) Temporal attention and retrieval architectures specifically designed for long videos, since Figure 6 shows this is the dominant failure mode. (3) Training data curation strategies that emphasize tightly synchronized audio-visual content (the paper hypothesizes Gemini 2.5 Pro's advantage comes from YouTube pretraining, Section 5)—purely unimodal or loosely-coupled multimodal training data will not produce MMOU-competitive models. (4) Evaluation methodology research on rubric-based open-ended scoring with LLM judges, since the paper's open-ended evaluation (while unvalidated against human judgments) demonstrates the value of multi-dimensional scoring beyond binary correctness.

Research directions that become less attractive: (1) Incremental scaling of existing architectures without addressing cross-modal fusion—the paper shows that even models with strong standalone vision and audio capabilities (Qwen3-VL-32B at 44.0%, Qwen3-Omni at 35.6%) are far from competitive MMOU performance, and simply making these components larger is unlikely to close the gap without architectural changes that enable genuine integration. (2) Benchmarks that evaluate modalities independently or that do not enforce cross-modal dependency—MMOU raises the bar for what counts as a valid audio-visual evaluation, and benchmarks that fall below this bar will increasingly be seen as measuring a weaker, less relevant capability. (3) Reliance on MCQ-only evaluation for multimodal systems—the paper's demonstration that MCQ format inflates understanding by ~21% for the best model, and that this inflation is skill-dependent, weakens the case for treating MCQ accuracy as a sufficient measure of multimodal competence.

Follow-Up Research This Work Enables

1. Measuring and mitigating train–test leakage from web-sourced video benchmarks. The paper explicitly acknowledges that MMOU's web-sourced videos may introduce "potential train–test leakage in closed and open-weight models" (Section 7), and hypothesizes that Gemini 2.5 Pro's advantage "may be attributable to pre-training on YouTube videos" (Section 5). This is a specific, testable concern. A follow-up study would construct a leakage-probing experiment: curate a subset of MMOU videos that were uploaded to YouTube after a known cutoff date (e.g., videos uploaded after December 2024, when the most recent models in the benchmark were trained) and compare model accuracy on pre-cutoff vs. post-cutoff videos. If Gemini 2.5 Pro's advantage is partly due to memorization, its accuracy on post-cutoff videos should be significantly lower than on pre-cutoff videos, while models with documented training data that excludes recent YouTube content should show no such gap. Alternatively, one could compare performance on "popular" videos (high view count, likely included in web-scale training corpora) vs. "obscure" videos (low view count, unlikely to be included), controlling for video duration and domain. A positive finding (significant leakage) would require the community to rethink how we construct video benchmarks—moving toward synthetically generated content, content from sources known to be excluded from training, or continuously updated benchmarks that retire videos after a fixed period. A negative finding (no detectable leakage) would increase confidence in MMOU's scores as measuring genuine reasoning rather than memorization, and would be valuable evidence in the broader debate about benchmark contamination.

2. Disentangling video duration, temporal position, and domain as causal factors in model degradation. The paper's analyses in Table 3 (duration buckets), Figure 6 (temporal position), and Table 3 (domains) are reported marginally—we cannot determine whether the degradation on >30-minute videos is caused by video length, by the temporal position of answer evidence (which is correlated with duration), or by the types of content that occupy long-duration buckets. A follow-up study would use the paper's existing annotations (every question has a domain tag, skill tags, video duration, and temporal answer position) to perform a controlled regression: predict model accuracy from duration, temporal position, domain, and skill, including interaction terms. This would reveal, for example, whether temporal position remains a significant predictor after controlling for duration (if so, the mechanism is likely context retention rather than total context length) and whether the duration effect varies by domain (if sports videos show steep duration degradation but academic lectures do not, the mechanism may be event density rather than raw length). The same analysis could identify ceiling effects: are there question categories (specific skill × domain × duration combinations) where even the best model performs near chance, providing durable headroom for benchmark validity? This would also enable the construction of a difficulty-predicting model that estimates, from a question's metadata alone, how hard it will be for a given model family—a tool that could guide efficient benchmark subsetting for resource-constrained evaluation.

3. Validating LLM judges for rubric-based open-ended evaluation of multimodal outputs. The paper's open-ended evaluation (Section 6, Table 4) uses GPT-5 as an LLM judge with a four-dimensional rubric, but provides no validation of the judge against human evaluators. This is a critical gap because all conclusions derived from the open-ended evaluation—the 3.90 overall score for Gemini 2.5 Pro, the 21.1% MCQ rescue rate, the per-skill rubric breakdown in Figure 7—depend on GPT-5's judgments being accurate and unbiased. A follow-up study would collect human judgments (from, say, 3–5 trained evaluators) on a stratified sample of 500–1000 model responses spanning multiple models and skill categories, using the same four-criterion rubric. This would enable: (a) computing inter-annotator agreement among humans to establish an upper bound on achievable judge reliability; (b) measuring correlation between GPT-5 scores and mean human scores on each criterion, identifying any systematic biases (does GPT-5 consistently overrate Clarity? underrate Faithfulness?); (c) comparing the custom-trained Qwen-3.5-0.8B judge (Appendix G.5) against both GPT-5 and humans, determining whether a small open-source judge can match proprietary judge quality; and (d) establishing whether the weighted overall score formula (0.5 × Correctness + (0.5/3) × (Completeness + Faithfulness + Clarity)) produces rankings that agree with human holistic preferences, or whether the weights need calibration. The paper already has the infrastructure for this study (the custom judge is trained, the rubric is defined, the model responses exist); the missing piece is human validation data. A negative result—finding that GPT-5's judgments diverge substantially from human judgments on certain criteria or skill categories—would be equally valuable, as it would establish boundary conditions for LLM-as-judge approaches in multimodal evaluation.

4. Developing and benchmarking cross-modal fusion mechanisms against late-fusion baselines on MMOU. The paper demonstrates that late-fusion through captioning fails on MMOU (cascaded models score 28.1–33.1%, Table 3), but does not test whether this failure is due to information loss in captioning (which might be mitigated by better captioning models) or due to a fundamental need for cross-modal attention at the representational level. A follow-up study would design a controlled experiment: take a single base LLM and equip it with three processing pathways—(a) early fusion (cross-modal attention between audio and visual tokens throughout all transformer layers), (b) mid fusion (independent unimodal processing for the first N layers, cross-modal attention thereafter), and (c) late fusion (fully independent unimodal processing, with cross-modal attention only at a final fusion layer or through a textual summary bottleneck). All three variants would have identical parameter counts and be trained on identical data, differing only in the fusion architecture. Evaluating these on MMOU would answer a fundamental question: how much does fusion architecture matter for the specific skills that MMOU tests, and which skills benefit most from early vs. late fusion? The hypothesis, based on the paper's cascaded baseline results, is that early fusion will substantially outperform late fusion, and that the gap will be largest for skills requiring fine-grained temporal alignment between modalities (referential grounding, temporal understanding, audio-visual stitching). This experiment would directly test the paper's implicit claim that MMOU measures genuine cross-modal integration, not just multimodal access.

5. Constructing a longitudinal benchmark suite with built-in saturation monitoring and dynamic difficulty refresh. The paper acknowledges in Section 7 that MMOU, as a static benchmark, will eventually saturate as models improve, but provides no mechanism for tracking or preventing saturation. A follow-up project would construct a "living benchmark" protocol: (1) establish a baseline saturation measurement by identifying the subset of MMOU questions that are already solved by the best model (Gemini 2.5 Pro answers correctly) vs. those that remain challenging, and tracking this solved fraction over time as new models are evaluated. (2) Define difficulty tiers based on current model performance: Tier 1 (solved by >80% of evaluated models), Tier 2 (solved by 40–80%), Tier 3 (solved by <40%), and report per-tier accuracy for each new model to provide headroom calibration. (3) Implement a quarterly refresh cadence where new videos (uploaded after a cutoff date, to control for leakage) are annotated using the same protocol and added to the benchmark, while Tier 1 questions (solved by all models) are optionally retired to a legacy set. (4) For the refresh, prioritize domains and skills where the headroom is narrowest—the paper's Figure 5 and Table 3 provide the baseline data to identify which domains (Animation, Music) and skills (Holistic Reasoning, Context Understanding) are closest to saturation and thus most urgently need new content. This project would transform MMOU from a static artifact into a sustainable evaluation infrastructure, and the protocols developed would be reusable for any large-scale multimodal benchmark.

6. Probing whether cross-modal reasoning failures are perceptual or cognitive through error decomposition. The paper's skill-wise analysis (Figure 5) shows that Counting and Temporal Understanding are consistently hard, but does not distinguish whether models fail because they do not perceive the relevant events (perceptual failure: the model does not detect that a horn sounded or that shadows are present) or because they cannot reason about perceived events (cognitive failure: the model detects the events but cannot count them or order them correctly). A follow-up study would design a two-stage probing experiment on a subset of MMOU Counting and Temporal Understanding questions: Stage 1 asks the model simple detection questions about individual events ("Was a horn heard in this video?" "Were shadows visible on the skate rink?"), establishing whether the perceptual components are intact. Stage 2 presents the original reasoning question. If models succeed at Stage 1 (detection) but fail at Stage 2 (counting/ordering), the bottleneck is cognitive—the model's reasoning over perceived events is the limiting factor, and improvements should target reasoning architectures. If models fail at Stage 1, the bottleneck is perceptual—the audio and visual encoders are missing or misidentifying events, and improvements should target encoder quality or temporal resolution. This decomposition would provide actionable guidance for model development and would calibrate the interpretation of MMOU's skill-level scores: a low Counting score with high perceptual accuracy means something very different (and calls for different interventions) than a low Counting score with low perceptual accuracy. The paper's annotation infrastructure (timestamped answer evidence, skill tags) makes this experiment feasible without additional data collection—the probing questions can be derived from the existing QA pairs by isolating the perceptual components of each reasoning question.

Practical Applications and Downstream Use Cases

1. Model selection and procurement for video-heavy enterprise deployments. Organizations deploying AI systems for video understanding tasks—media monitoring, educational content analysis, security footage review, video archive search—need to choose among competing models with limited information. MMOU provides a standardized, multi-dimensional evaluation that directly measures the capability these deployments require: joint audio-visual reasoning over long, real-world videos. The per-domain breakdown in Table 3 enables domain-specific model selection: an organization primarily analyzing sports content can see that Gemini 2.5 Pro achieves 61.2% on Sports while MiniCPM-o 4.5 achieves 30.4%—a 30.8-point gap that may justify the cost of a proprietary API over an open-source alternative. An organization focused on academic lecture understanding can see that Gemini 2.5 Pro achieves 71.4%, while even the best open-source model (MiniCPM-o 4.5) achieves only 43.6%—a 27.8-point gap. The skill-wise analysis (Figure 5) further enables capability-specific procurement: if the deployment requires accurate counting of events in video (e.g., counting vehicles in traffic footage, counting actions in manufacturing quality control), the paper's finding that even the best model struggles with Counting provides a sobering calibration—expect substantial error rates regardless of which model is chosen, and budget for human-in-the-loop verification for counting-critical applications.

2. Diagnostic tool for multimodal model development. The paper's skill taxonomy and per-skill accuracy breakdown (Figure 5) enable model developers to identify specific weaknesses in their systems rather than just tracking an aggregate accuracy number. If a team is developing a new audio-visual MLLM and evaluates it on MMOU, they receive not just "46.8% overall" but a 13-dimensional capability profile showing that their model excels at holistic reasoning and context understanding but fails at counting, temporal understanding, and needle-in-the-haystack reasoning. This profile directly informs data collection priorities: if counting is the weakest skill, the team can curate or synthesize training data that specifically targets counting of audio-visual events—videos with repeated, distinct, cued events (sports plays, manufacturing steps, musical phrases) labeled with count annotations. The temporal position sensitivity analysis (Figure 6) provides an additional diagnostic: if a team's model shows steep degradation at late temporal positions, they can prioritize architectural improvements to long-context retention (e.g., better positional encodings, retrieval-augmented generation over video segments, or hierarchical attention) rather than general scaling. The open-ended evaluation framework (Table 4, Figure 7) provides a further layer: if a model scores high on MCQ but low on open-ended, the team knows their model may be relying on recognition and elimination rather than genuine understanding, and can prioritize training for generative capabilities.

3. Calibration of user-facing AI systems that describe or answer questions about video content. The paper's finding that models are consistently clear and fluent (Clarity scores of 4.03–4.62 in Table 4) even when factually wrong (Correctness scores as low as 1.30–3.71) has direct implications for deployment safety. A user-facing video QA system built on Qwen3-Omni-30B-Instruct (Correctness 2.27, Clarity 4.62) will produce answers that sound authoritative and well-articulated while being substantially incorrect, potentially misleading users who lack the expertise to verify the answers. The paper's quantification of this "fluent-but-wrong" gap—Clarity exceeds Correctness by 2.35 points for Qwen3-Omni-Instruct, and by 0.91 points even for Gemini 2.5 Pro—provides a specific risk metric that can inform deployment decisions. A product team can set thresholds: if a model's Clarity-Correctness gap exceeds some value (say, 1.5 points), the system should include explicit uncertainty communication ("I'm not fully confident about this answer") or defer to human review. The per-skill breakdown in Figure 7 further enables skill-specific calibration: for Holistic Reasoning, where Gemini 2.5 Pro's Correctness (4.24) and Clarity (4.60) are closely aligned, the system can be more confident; for Counting, where Correctness (3.01) lags Clarity (4.60) by 1.59 points, the system should be explicitly uncertain. This kind of calibrated deployment is only possible because MMOU provides multi-dimensional evaluation rather than a single accuracy number.

4. Benchmarking progress in the open-source multimodal community. The paper evaluates a comprehensive set of open-source audio-visual models (Table 3: nine models spanning from Qwen2.5-Omni-7B at 31.3% to MiniCPM-o 4.5 at 46.8%) and establishes a clear performance hierarchy. This provides a baseline against which future open-source releases can be measured, and enables the community to track the rate of progress. The finding that Qwen3-Omni-30B-A3B-Instruct (46.0%) substantially outperforms Qwen2.5-Omni-7B (31.3%)—a 14.7-point improvement within one model generation of the same family—suggests rapid progress is possible, but the finding that the reasoning-enhanced Qwen3-Omni-30B-A3B-Thinking actually regresses to 19.4% (a 26.6-point drop from its non-thinking counterpart) reveals that progress is not monotonic and that reasoning enhancements do not automatically transfer to omni-modal video understanding. For open-source model developers, MMOU serves as a standardized report card that captures both overall capability and the specific dimensions where improvement is needed. The benchmark's release on Hugging Face (indicated in the project URL) makes it accessible to the community, and the paper's detailed evaluation protocol (five-fold option randomization, best-prompt reporting, regular-expression parsing) provides a reproducible methodology.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark2023· evaluation← LLM-judged similar
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark2023· evaluation← in-domain predecessor
- Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities2026· evaluation← similar topic
- Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding2026· evaluation← similar topic
- Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs2026· evaluation← similar topic
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark2024· evaluation← similar topic
- MMLU: Measuring Massive Multitask Language Understanding2020· evaluation← similar topic
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems2019· evaluation← similar topic
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections2026· evaluation← in-domain predecessor
- Deep Research: A Systematic Survey2025· uncategorized← LLM-judged similar
- MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents2026· evaluation← in-domain predecessor
- How is ChatGPT's behavior changing over time?2023· evaluation← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation2026· evaluation→ in-domain predecessor
- Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models2026· evaluation→ in-domain predecessor
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark2023· evaluation→ similar topic
- Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities2026· evaluation→ similar topic
- Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs2026· evaluation→ similar topic
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018· evaluation→ similar topic
- SQuAD 2.0: The Stanford Question Answering Dataset2018· evaluation→ in-domain predecessor
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems2019· evaluation→ similar topic
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark2024· evaluation→ similar topic