MMGR: Multi-Modal Generative Reasoning

🎯 Pitch

Despite generating photorealistic videos, top models like Veo-3 and Sora-2 achieve under 10% accuracy on abstract reasoning tasks like ARC-AGI and often produce the right final answer through completely invalid steps—a ‘hallucination of competence’ where 74% of math outcomes are correct but only 12% of the reasoning processes are.

1. Executive Summary

This paper introduces MMGR (Multi-Modal Generative Reasoning Evaluation and Benchmark), a principled evaluation framework and benchmark suite that systematically assesses generative reasoning in video and image models across five core abilities—Physical, Logical, 3D Spatial, 2D Spatial, and Temporal reasoning—spanning three complementary domains: Abstract Reasoning (e.g., Maze solving, Sudoku, ARC-AGI, Math), Embodied Navigation (e.g., 3D real-world navigation, simultaneous localization and generation), and Physical Commonsense (e.g., sports scenarios, material interactions). Benchmarking state-of-the-art models including Veo-3, Sora-2, Wan-2.2, Nano-banana Pro, and GPT-4o-image, the work reveals a pronounced modality asymmetry—image models dominate abstract reasoning (Nano-banana Pro achieves 30.54% on ARC-AGI v1 vs. video models' <10%), while video models exhibit a "hallucination of competence" where they frequently generate correct final answers despite invalid intermediate reasoning (74% outcome success but only 12% process success on GSM8K for Veo-3)—and identifies verifier over-optimization and context drift as core failure modes, where video models corrupt static demonstration examples during generation. Human evaluation exposes that automated metrics systematically overestimate model competence by 2–5× (Veo-3's maze-solving rated at 42–52% by AutoEval vs. 0–20% by humans), establishing that current generative models prioritize visual plausibility over causal correctness and can only simulate surface-level patterns rather than execute genuine multi-step reasoning.

2. Context and Motivation

The Core Problem: We Evaluate Video Models on Visual Quality, Not Reasoning

The fundamental question this paper tackles is straightforward yet has been largely overlooked by the field: do video generation models actually reason about the content they produce, or do they merely synthesize visually plausible sequences? This matters because the long-term ambition of the field—articulated explicitly by major research labs—is to build "world models" or "world simulators" (Ha & Schmidhuber, 2018; LeCun, 2022) that internalize the physical, logical, and spatial constraints governing reality. If a model generates a billiards shot where balls pass through one another, or a navigation sequence where an agent teleports through walls, it has failed as a world simulator regardless of how photorealistic the output appears.

This gap is significant for several practical and theoretical reasons the authors highlight (Section 1):

Embodied AI and robotics: Video generation models are increasingly being positioned as foundations for robotic planning and simulation. If these models lack robust physical reasoning—understanding object permanence, collision geometry, momentum conservation—they cannot serve as reliable simulators for training or deploying embodied agents.
Scientific and educational applications: Applications in scientific visualization, medical training, or educational content require causal fidelity, not just visual appeal. A generated video of a chemical reaction that looks realistic but violates conservation of mass is actively misleading.
Trustworthiness of generated media: As synthetic video becomes ubiquitous, distinguishing between content that is merely visually convincing versus content that is causally faithful becomes a critical societal challenge. Evaluation frameworks must evolve to capture this distinction.
Guiding model development: Without diagnostic benchmarks that reveal why models fail, research risks optimizing for the wrong objectives—pursuing higher FVD scores while ignoring fundamental reasoning deficits that will cap progress toward genuine world modeling.

The paper is motivated by a genuine crisis in how video generation models are evaluated. As the authors state in Section 2:

"Metrics such as FVD and Inception Score (IS) capture perceptual fidelity, while more recent benchmarks focus on text–video alignment and basic temporal consistency. These tools, however, are insufficient for probing world modeling capabilities."

This critique is more specific than a general complaint about metrics. The paper identifies three distinct failure modes of existing evaluation paradigms:

1. Perceptual metrics reward the wrong behavior. Fréchet Video Distance (FVD) (Unterthiner et al., 2018a) measures the distributional distance between generated and real video features, essentially asking "does this look like a video from the training distribution?" A model that generates a visually stunning billiards shot where balls pass through one another achieves excellent FVD because the hallucination is visually subtle—individual frames look photorealistic. The metric has no mechanism to detect that the output violates physical law. The same applies to Inception Score (IS) (Salimans et al., 2016) and CLIP-based similarity metrics (Radford et al., 2021), which measure semantic alignment between video and text but cannot verify whether the depicted dynamics are physically possible.

2. Existing video benchmarks test understanding, not generative reasoning. Prior benchmarks in video understanding—such as Something-Something (Goyal et al., 2017) for action recognition, CATER (Girdhar & Ramanan, 2020) for object tracking, or even ARC-AGI (Chollet, 2019) for abstract reasoning—evaluate discriminative models. They ask: given a video, can the model classify the action, track the object, or deduce the rule? This is fundamentally different from asking: can the model produce a video that demonstrates reasoning? A model might be able to recognize that a maze solution is valid without being able to generate one itself. The paper explicitly shifts the paradigm:

"Our work shifts this paradigm from understanding to generation: requiring models to not merely interpret a video, but to manifest reasoning processes through synthesis."

3. Embodied AI benchmarks rely on rigid simulators, not generative models. Benchmarks like Habitat (Savva et al., 2019) and Matterport3D (Chang et al., 2017) provide photorealistic environments for training and evaluating navigation agents, but these operate within deterministic physics engines where actions have guaranteed outcomes. They test whether a learned policy can navigate a static environment—a fundamentally different capability from generating a coherent navigation video from a single prompt while respecting physical constraints.

The Conflicting Signals in Prior Work

The paper is also motivated by genuinely contradictory signals from recent research about what video models can and cannot do:

On the optimistic side, several recent studies suggest emergent reasoning capabilities in large video models:

Wiedemer et al. (2025) identified emergent "Chain-of-Frames" (CoF) reasoning in models like Veo-3, suggesting that these models can maintain coherent multi-step visual reasoning across frames—essentially "thinking" in visual space.
Tong et al. (2025) demonstrated competitive performance by Sora-2 across various vision tasks when using video generation as a reasoning paradigm, proposing that "thinking with video" could be a viable reasoning modality.
Guo et al. (2025) used their MME-CoF benchmark to probe geometric consistency in video models, finding both strengths and specific failure modes.

On the pessimistic side, these same studies expose critical limitations:

Guo et al. (2025) simultaneously found that models like Veo-3 suffer from significant geometric consistency failures—objects change shape, spatial relationships drift, and causal chains break down.
The "Chain-of-Frames" reasoning that Wiedemer et al. (2025) observed appears to be fragile, working on some tasks but collapsing on others with no clear explanation of when or why.
None of these prior studies provided a systematic framework across multiple reasoning dimensions to characterize which reasoning abilities are present, which are absent, and why the boundary exists.

This creates a confusing landscape: are video models nascent world simulators, or sophisticated pattern matchers with no genuine understanding? Different papers, targeting different tasks with different methodologies, reach different conclusions. The field lacks a unified diagnostic framework to reconcile these findings.

Where Prior Approaches Fall Short: Systematic Gaps

The paper identifies specific limitations in prior evaluation work that motivate MMGR's design:

No unified reasoning taxonomy. Prior work studied reasoning capabilities in isolation—physical reasoning (VideoPhy; Bansal et al., 2024), abstract visual reasoning (ARC-AGI; Chollet, 2019), embodied navigation (Habitat; Savva et al., 2019)—but there was no framework connecting these abilities into a coherent picture of what a "reasoning-aware" video model should possess. The authors argue (Section 1) that robust world simulation requires the coordination of multiple reasoning abilities—a model solving a navigation task must reason about 3D space, physical constraints, temporal causality, and logical goal conditions simultaneously. Testing these in isolation misses the integration challenge that defines real-world reasoning.

Missing diagnostic granularity. Benchmarks like VBench (Huang et al., 2024) and EvalCrafter (Liu et al., 2024b) provide multi-dimensional evaluation of video quality—assessing temporal consistency, motion smoothness, object permanence, and text-video alignment. However, these metrics are aggregated into quality scores; they do not decompose failures into specific reasoning dimensions that would guide model development. A model scoring poorly on "temporal consistency" in VBench provides no signal about whether the failure stems from physical implausibility, logical inconsistency, or spatial misalignment. The MMGR framework explicitly maps tasks to reasoning abilities (Table 1) and defines fine-grained, diagnostic metrics per task (e.g., Cross Wall vs. Maze Changed vs. Action Reflection in the Maze task) to enable precise failure-mode analysis.

Domain-specific isolation. VideoPhy (Bansal et al., 2024) evaluates physical commonsense in video generation but is limited to atomic physical interactions—splashing, collision, pouring. It does not assess spatial reasoning, abstract logic, or embodied planning. Conversely, ARC-AGI evaluates abstract visual reasoning but was designed for image-based pattern completion, not generative video synthesis. Neither benchmark addresses multiple reasoning abilities simultaneously, nor do they operate under a shared evaluation protocol that enables cross-domain comparison of model strengths and weaknesses. The authors note in Section 3:

"These domains are strategically complementary; together, they exercise the five core reasoning abilities outlined in Section 1."

No systematic comparison across modalities. A critical gap in prior work is the lack of comparison between video generative models and image generative models on reasoning tasks. Intuitively, one might expect video models—with their temporal processing capabilities—to excel at tasks requiring sequential reasoning (like maze navigation or math problem-solving). The MMGR results reveal the opposite: image models consistently outperform video models on abstract reasoning tasks, suggesting that the temporal dimension actually interferes with logical consistency rather than enhancing it. This finding (which the paper terms the "temporal tax" on reasoning) would be invisible without a benchmark that evaluates both modalities on identical tasks.

Automated evaluation validation is absent. Prior benchmarks largely assume that VLM-based automated evaluation is reliable for reasoning tasks. MMGR is motivated in part by skepticism of this assumption. The paper conducts extensive human evaluation studies (Tables 5, 7, 16-17, 27, 29, 31, 33, 38) that reveal systematic discrepancies between AutoEval and human judgment—with AutoEval overestimating success by 2–5× on temporally dense tasks like maze navigation, and underestimating performance on physical commonsense tasks where humans are more tolerant of minor visual artifacts. This validation work establishes that evaluation methodology itself needs to be benchmarked, not just the models.

How This Paper Positions Itself

The paper positions MMGR as filling the gap between surface-level quality evaluation and genuine reasoning assessment. The key intellectual moves that distinguish it from prior work are:

From quality metrics to reasoning diagnostics. Rather than aggregate performance into a single score, MMGR provides per-task, per-metric breakdowns that map directly to specific reasoning abilities. The Maze task doesn't just report "success rate"—it reports Action Reflection (does the model explore or execute directly?), Cross Wall (does the model respect impermeable boundaries?), Maze Changed (does the model preserve the problem definition?), and Target Achievement (does the model reach the goal?) as separate diagnostics (Section 5.3). This enables the discovery of the "illusion of competence" pattern: Veo-3 achieves 62% Target Achievement but only 42% Overall Success because it reaches the goal through physically impossible means.

A five-ability cognitive framework grounded in developmental psychology. The paper explicitly draws on theories of core knowledge and cognitive development (Spelke & Kinzler, 2007; Lake et al., 2017) to motivate its five reasoning pillars. This is more than taxonomic—it provides a theoretical justification for which abilities matter and why they should be evaluated together. Physical reasoning corresponds to "core knowledge" of intuitive physics that infants possess (Baillargeon, 1987); logical reasoning corresponds to System 2 cognitive processes (Kahneman, 2011); 3D spatial reasoning maps to "cognitive map" formation (Tolman, 1948; Gibson, 1979). By grounding the evaluation framework in cognitive science rather than arbitrary task selection, the paper argues that MMGR measures abilities fundamental to intelligence itself, not just video generation quality.

Generative adaptations of established reasoning benchmarks. Rather than creating entirely new tasks, MMGR adapts established reasoning benchmarks—ARC-AGI, Sudoku, MATH, embodied navigation environments—into a generative format. This provides a bridge between the reasoning literature (which has well-established difficulty hierarchies and construct validity) and the generative evaluation literature (which has focused on visual quality). The adaptation is non-trivial: ARC-AGI was designed for discrete image completions; making it a video generation task where the model must animate its solution while preserving static demonstration examples introduces the "context drift" failure mode that the paper identifies as characteristic of video models.

Systematic difficulty stratification. Each task in MMGR includes a hard-level control system that varies difficulty along semantically meaningful axes. Maze difficulty scales with grid size and start-goal placement; Sudoku difficulty with grid size and clue sparsity; ARC-AGI difficulty through a quantitative scoring system based on grid features (Section 7.1.1); Embodied Navigation across 24 configurations defined by environmental complexity, view fidelity, trajectory distance, and destination specification (Section 9.2). This stratification enables the paper's key analytical contribution: showing that model failure is not uniform but difficulty-dependent, with distinct scaling behaviors across model types (Nano-banana Pro's robust stability across ARC-AGI difficulty levels vs. Sora-2's 93% collapse from v1 to v2).

Human evaluation as a first-class component. The paper does not treat human evaluation as a small-scale validation study but as a core empirical contribution. The finding that human evaluators rate Veo-3 at 0% overall success across all 98 ARC-AGI cases (Section 7.4.2), compared to AutoEval's 4.72%, fundamentally changes the interpretation of the benchmark results. This positions MMGR not just as a benchmark for models but as a benchmark for evaluation methodology itself, exposing the reliability limits of VLM-based auto-evaluation and establishing human ground truth as essential for reasoning tasks.

Summary of the Motivation

The paper's central argument is that the field of video generation has reached a critical juncture where models can produce visually stunning output but may possess negligible reasoning capabilities, and that existing evaluation frameworks are structurally incapable of detecting this gap. The motivation is simultaneously practical (if we deploy these models as world simulators, the consequences of reasoning failures could be severe), scientific (understanding what video models actually learn about the world is a fundamental AI question), and methodological (we need new tools to measure what matters, not just what's easy to measure).

MMGR's response is to build a unified evaluation framework that (1) defines what reasoning abilities matter, grounded in cognitive science; (2) designs tasks that demand the coordination of multiple abilities; (3) provides fine-grained, diagnostic metrics per task that reveal specific failure modes; (4) implements systematic difficulty control to characterize when and how models fail; and (5) validates automated evaluation against human judgment to establish measurement reliability. This is an ambitious scope—far exceeding any prior generative evaluation benchmark—and the paper's contributions are primarily empirical: the systematic characterization of reasoning deficits in SOTA models that this framework enables, rather than the proposal of a new model or training method.

3. Technical Approach

3.1 Reader Orientation

MMGR is not a single model or algorithm but an evaluation framework and benchmark suite — a principled diagnostic system for measuring whether generative models (both video and image) can reason about the content they produce across five core cognitive abilities. The problem it solves is that existing evaluation metrics for generative models are blind to reasoning: they measure whether a generated video looks realistic (perceptual fidelity) but cannot detect whether the events depicted are physically possible, logically consistent, or spatially coherent. The shape of the solution is a multi-domain, multi-ability benchmark with fine-grained per-task metrics, systematic difficulty control, and extensive human validation — designed so that a model's score on MMGR reveals not just how well it generates, but what specific reasoning capabilities it possesses and where it fails.

3.2 Big-Picture Architecture (Diagram in Words)

The MMGR system has five major components that operate in a sequential pipeline:

Task Specification and Hard-Level Control: Defines each benchmark task with precise visual templates, rules for generating evaluation instances, and axes of difficulty variation (grid size, trajectory distance, puzzle difficulty). Produces controlled input images or scene configurations.
Generation Engine: Takes the task specification and produces 5 candidate outputs per prompt for each model under test. This engine handles model-specific APIs (for closed-source models like Veo-3, Sora-2, GPT-4o-image) and recommended configurations (for open-weights models like Wan-2.2, Qwen-image), ensuring zero-shot, fair comparison.
VLM-Based Automated Evaluator (AutoEval): A Gemini-2.5-Pro instance that receives the generated output, the ground-truth reference, and a structured task-specific evaluation prompt. It produces binary scores on fine-grained metrics (e.g., Cross Wall = 1/0, Pattern Recognition = 1/0) and a brief textual justification of its judgment.
Human Evaluation Pipeline (HumanEval): A web-based annotation platform with video playback controls (frame-by-frame navigation, adjustable playback speed). Six trained annotators assess a curated subset of generated outputs using identical evaluation criteria as the VLM evaluator, providing ground-truth calibration for AutoEval reliability.
Metrics Aggregation and Diagnostic Analysis: Computes primary metrics (strict holistic scores requiring simultaneous satisfaction of all sub-metrics) and generates difficulty-stratified breakdowns across domains, task types, and model architectures.

Information flows as follows: a task specification is constructed under hard-level control parameters → the specification is rendered as an input image and prompt → each model under test generates 5 candidate outputs → Gemini-2.5-Pro evaluates each candidate against ground-truth references using structured rubrics → fine-grained binary metrics are extracted from the evaluator's responses → primary holistic metrics are computed as logical conjunctions of sub-metrics → human evaluation is conducted on a subset for calibration → results are stratified by difficulty level and analyzed per reasoning ability.

3.3 Roadmap for the Deep Dive

First, the five-ability reasoning framework (Section 1 of the paper, referenced throughout Section 3): what are the five reasoning abilities, how are they grounded in cognitive science, and how do they map to specific tasks. This is the conceptual foundation — every evaluation metric is designed to probe one or more of these abilities.
Second, how hard-level control works across all tasks: the systematic axes of difficulty variation (grid sizes, trajectory distances, environmental complexity) that enable the paper's key analytical contribution — showing that model failure is difficulty-dependent rather than uniform.
Third, the structured evaluation prompt and rubric system: how the VLM evaluator is prompted, what the evaluation rubrics look like per task, and how they enforce binary (pass/fail) judgments for each fine-grained metric. This is the computational mechanism that converts raw generated video into diagnostic scores.
Fourth, the metrics aggregation strategy: how fine-grained metrics are combined into gated primary metrics (overall success = ALL sub-metrics pass simultaneously), why this strict conjunction is necessary (to prevent the "illusion of competence" where partial success masks fundamental failure), and how the metric cascade reveals performance bottlenecks.
Fifth, the human evaluation protocol: the annotation interface design, the training and calibration process, and the statistical methodology for comparing AutoEval against HumanEval — establishing the reliability boundaries of automated reasoning evaluation.
Sixth, the model configuration and generation protocol: what models are evaluated, why these specific models, what default parameters are used, and how the zero-shot evaluation guarantees fairness.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical evaluation paper whose core idea is that generative models can be diagnosed for reasoning capability through a multi-domain benchmark with fine-grained, binary diagnostic metrics, systematic difficulty control, and rigorous human validation of automated evaluation.

The Five-Ability Reasoning Framework

The conceptual foundation of MMGR is a taxonomy of five reasoning abilities that the authors argue are necessary and (with the right tasks) sufficient to characterize whether a generative model is a genuine "world model" or merely a visual pattern matcher. The framework is explicitly grounded in theories of core knowledge and cognitive development (Spelke & Kinzler, 2007; Lake et al., 2017), and each ability is operationally defined through specific benchmark tasks:

Physical Reasoning refers to understanding intuitive physics: object permanence (objects continue to exist when occluded), gravity (unsupported objects fall), collisions (solid objects do not interpenetrate), material properties (rigid vs. deformable, fluid vs. solid), and momentum conservation. In the benchmark, this ability is probed by Physical Commonsense tasks — the Physical Concept sub-task (leveraging the VideoPhy ontology of solid-solid, solid-fluid, and fluid-fluid interactions) and the Sports sub-task (which requires compositional physical reasoning integrating multiple physical laws simultaneously). The evaluation metrics directly test whether generated videos obey physical laws: Physics Accuracy checks for violations of gravity, momentum, and collision constraints; Object Semantic Score in Embodied Navigation checks whether the agent collides with or passes through solid structures.

Logical Reasoning refers to manipulating abstract concepts, following formal rules, and performing deductive operations — what Kahneman (2011) terms "System 2" reasoning. In the benchmark, this is probed by tasks with explicit logical constraints: Sudoku (digit uniqueness across rows, columns, and subgrids), Maze (a valid path must not cross walls), Math (step-by-step deduction from premises to solution), and ARC-AGI (inferring latent transformation rules from examples). The evaluation metrics directly test logical constraint satisfaction: Constraints Violation in Sudoku checks whether the model's output violates the uniqueness rules; Intermediate Correctness in Math checks whether all reasoning steps are logically valid.

3D Spatial Reasoning refers to understanding volumetric spatial relationships, depth, occlusion, and viewpoint transformation — building an internal "cognitive map" (Tolman, 1948) of 3D environments. This is probed by Embodied Navigation tasks that require the model to interpret scene geometry from third-person cutaway views (3D Real-World Navigation), panoramic egocentric views (Last-Mile Navigation), and cross-view alignment (SLAG, which requires synchronizing 3D navigation with 2D map generation). The evaluation metrics test spatial understanding: Spatial Alignment Score checks whether the agent's heading, motion direction, and elevation changes are coherent with the expected physical layout; Trajectory Alignment Score in SLAG checks whether the 2D projected route matches the 3D motion path.

2D Spatial Reasoning refers to interpreting planar layouts, shapes, relative positions, and grid-based patterns in the projected image plane. This is probed by tasks operating on 2D grids: Maze (understanding the topology of white paths vs. black walls on a 2D grid), Sudoku (interpreting the 2D grid structure of rows, columns, and subgrids), ARC-AGI (manipulating colored cells on 2D grids), and Top-down View Navigation (interpreting bird's-eye floor plans). The evaluation metrics test 2D spatial accuracy: Cross Wall in Maze checks whether the agent stays on white paths; Grid Integrity in ARC-AGI checks whether the output preserves correct grid dimensions and layout; Success Score 2D in Top-down Navigation checks whether the agent's final position lies within the goal footprint in the 2D map.

Temporal Reasoning refers to modeling causality, event ordering, and long-range dependencies — segmenting continuous streams into discrete causal events (Michotte, 1963; Zacks & Tversky, 2001). This is probed by tasks that require the model to generate coherent temporal sequences: Maze (executing a path plan sequentially), Sudoku (filling digits progressively without overwriting correct entries), Math (animating step-by-step deduction), and all Embodied Navigation tasks (generating smooth trajectories over multiple frames). The evaluation metrics test temporal coherence: Action Reflection in Maze and Sudoku checks whether the model exhibits meaningful step-by-step behavior vs. instantaneous jumps; Motion Quality in Physical Commonsense checks for smooth, continuous motion without jerky discontinuities; Scene Consistency across all navigation tasks checks whether the environment remains static throughout the video — a failure on this metric indicates temporal drift in world-state maintenance.

The authors explicitly justify the separation of 2D and 3D spatial reasoning (a design choice that might initially seem like over-splitting) by arguing they "rely on fundamentally different perceptual and computational mechanisms" (Section 1). 2D reasoning operates on planar relationships — adjacency, relative positioning, in-plane transformations — while 3D reasoning necessitates depth estimation, viewpoint transformation, and occlusion handling. This mirrors human cognition, where flat representations (maps) are processed differently than volumetric environments. The benchmark targets this dichotomy explicitly: abstract tasks like Sudoku and ARC-AGI probe 2D grid-based logic, whereas embodied navigation tasks demand coherent 3D spatial understanding.

Table 1 in the paper provides the complete mapping: each of the ten benchmark tasks is annotated with which of the five reasoning abilities it evaluates. For example, Maze is marked with Logical, 2D Spatial, and Temporal reasoning (the model must plan a valid route on a 2D grid and execute it sequentially). 3D Real-World Navigation is marked with Physical, 3D Spatial, and Temporal reasoning (the model must navigate a 3D environment while respecting physical constraints and maintaining temporal coherence). This mapping is not merely descriptive — it is prescriptive for evaluation design: every metric defined for a task is designed to probe one of the marked abilities.

Hard-Level Control: Systematic Difficulty Stratification

A central design principle of MMGR is that tasks must not be uniform in difficulty — evaluating models only on easy cases overestimates capability, while evaluating only on hard cases provides no signal about where the capability boundary lies. The paper implements systematic hard-level control across all ten tasks, varying difficulty along semantically meaningful axes that enable fine-grained analysis of when and how models fail.

Maze (Section 5.2). Difficulty is controlled along three independent axes: maze generation algorithm (2 types — Depth-First Search and Wilson's Algorithm — producing topologically diverse layouts), grid size (10 levels from 3×3 to 13×13), and start-goal placement scheme (4 types — corner-to-corner, corner-to-random, random-to-corner, random-to-random). A minimum start-goal distance is enforced to prevent trivial solutions. The 240 total mazes are stratified into Easy (3×3–5×5, 80 mazes), Medium (6×6–9×9, 80 mazes), and Hard (10×10–13×13, 80 mazes). The purpose of varying the generator is to test whether models have overfit to maze topology — DFS produces mazes with long corridors (a single solution path with no branches), while Wilson's algorithm produces more uniform, bushy mazes.

Sudoku (Section 6.1). Difficulty is controlled along two axes: grid size (2 levels — 4×4 with digits 1–4, and 9×9 with digits 1–9) and puzzle difficulty (3 levels — Easy, Medium, Hard — modulated by the number of initial clues, with fewer clues creating harder search). The three difficulty levels are: Easy (many pre-filled clues, small search space), Medium (moderate clue count, larger search space but still uniquely solvable), and Hard (minimal clues while ensuring a unique solution — the maximum difficulty before the puzzle becomes ambiguous). This yields a balanced set of 300 puzzles: 50 per grid-size/difficulty combination (e.g., 50 of 4×4 Easy, 50 of 9×9 Hard). The key insight in difficulty design is that 9×9 grids are not simply "larger" 4×4 grids — they require deeper multi-step logical inference because the constraints (rows, columns, 3×3 subgrids) create more complex interdependencies.

ARC-AGI (Section 7.1). Difficulty is controlled through a sophisticated two-level classification system. Level 1 is Shape Consistency: Match cases (316 instances) where input and output grids have the same dimensions (requiring in-place transformations like color change or pattern filling), and Mismatch cases (140 instances) where input and output have different dimensions (requiring spatial restructuring — cropping, extraction, concatenation). Level 2 is a Quantitative Difficulty Classification based on five grid-level features scored from 0 to 2:

Grid Size: minimum side length < 8 → score 0; 8–15 → score 1; ≥ 16 → score 2.
Color Count: distinct colors in input ≤ 3 → score 0; 4–6 → score 1; ≥ 7 → score 2.
Object Count: distinct connected components ≤ 4 → score 0; 5–10 → score 1; > 10 → score 2.
Occupancy Ratio: non-background cells / total cells ≤ 0.25 → score 0; (0.25, 0.55] → score 1; > 0.55 → score 2.
Change Ratio (∆IO, Match cases only): cell changes / total cells ≤ 0.2 → score 0; (0.2, 0.5] → score 1; > 0.5 → score 2.

A case's overall difficulty is the sum of applicable feature scores: for Match cases (5 features), Easy ≤ 3, Medium 4–6, Hard ≥ 7; for Mismatch cases (4 features), Easy ≤ 2, Medium 3–4, Hard ≥ 5. The distribution across 456 cases (381 v1 + 75 v2) is reported in Table 8, and v2 is designed to be harder overall (containing only 1 Easy Match case vs. 102 in v1), enabling a robustness test against distribution shift. This quantitative difficulty system replaces human judgment with objective, computable features — a design choice that enables reproducibility and avoids the circularity of defining difficulty post-hoc based on model performance.

Math (Section 8.1). Difficulty is controlled by selecting problems from five existing benchmarks spanning a wide difficulty spectrum: GSM8K (grade school, 50 problems), MATH500 (high school competition, 50 problems), AIME 2024 and 2025 (invitational competition, 30 problems each), and Omni-MATH (Olympiad-level, 167 problems). Omni-MATH provides an additional internal difficulty taxonomy with five levels (T0–T4) and eight mathematical categories (Algebra, Applied Math, Calculus, Discrete Math, Geometry, Precalculus, Number Theory, and Other), enabling fine-grained domain-specific analysis. The key design insight is that problem difficulty here is not a function of visual complexity (the problems are presented as plain text images) but purely of mathematical reasoning depth — T0 problems are accessible to middle school students, while T4 problems represent Olympiad-level challenges.

Embodied Navigation (Section 9.2). Difficulty is controlled along four hard-level axes applied uniformly across all four navigation subtasks (Last-Mile Navigation, Top-down View, 3D Real-World Navigation, SLAG), with task-specific interpretations where needed:

Environmental Complexity (2 levels): Single-floor environments (floor01) vs. multi-floor environments (floor02plus) with vertical transitions (staircases, elevators). Multi-floor environments in the 3D rendered tasks (Top-down, 3D R.-W., SLAG) include fully connected floors with additional branching regions — basements, attics, outdoor areas — requiring models to reason about verticality and inter-floor connectivity.
View Fidelity (3 levels): quality03, quality04, quality05. For Panoramic Last-Mile Navigation, fidelity captures the extent and spatial distribution of occlusions from foreground objects (how much of the room layout is visible; how many landmarks can be seen). For the other three tasks, fidelity reflects holistic scene realism — presence of holes or cracks, furnishing quality, door openness, and plausibility of interaction. These are scored by human raters on the 3–5 scale.
Trajectory Distance (2 levels): Short trajectories (relatively direct, no major turns, may include vertical movement) vs. Long trajectories (at least one significant turn, sharing partial path structure with corresponding short cases at the same hard level for comparability). Distance is defined as the geodesic separation between start and destination.
Destination Specification (2 levels): Visual marker (color mark — a pure red #ff0000 overlay on the target region) vs. natural language description (location description — text specifying the target location with semantically meaningful landmarks). For ambiguous descriptions where multiple similar regions exist, annotators include disambiguating details (floor identifiers, spatially anchored landmarks). Figure 15 visualizes the vocabulary distribution of these descriptions, showing dominance of spatial prepositions ("front," "top," "next," "right") and structural landmarks ("staircase," "window," "entrance," "living room").

This four-axis, fully crossed design produces 24 distinct configuration slots (2 × 3 × 2 × 2 = 24), with each slot containing 5 evaluation samples across all four navigation subtasks (Table 23), yielding 120 samples per subtask and 480 total for Embodied Navigation. The key design insight is that by varying these axes independently, the benchmark can attribute performance changes to specific factors — for example, isolating whether a model fails due to environmental complexity (more rooms) or destination ambiguity (textual vs. visual specification), which would be confounded in a less controlled design.

Physical Commonsense (Section 14.2). Difficulty is controlled by stratification across three dimensions: interaction type (Solid-Solid, Solid-Fluid, Fluid-Fluid — testing different material properties and physical regimes), scenario context (controlled Physical Concepts vs. dynamic Sports Scenarios — testing generalization from atomic physics to compositional real-world settings), and interaction complexity (Simple single-object motion, Complex multi-object interactions with simultaneous forces, Chain-Reaction causal sequences). The evaluation set is balanced at 50 total samples (25 Physical Concepts, 25 Sports Scenarios), with equal representation across interaction types, ensuring that fundamental physics and compositional reasoning are weighted equally in aggregate scores.

Structured Evaluation Prompts and Rubric Design

The core computational mechanism of MMGR is the VLM-based evaluator, which must convert raw generated video or image into structured, binary diagnostic scores. This is a non-trivial design problem: the evaluator must apply consistent criteria across diverse tasks, produce judgments that are granular enough to diagnose specific failure modes, and output scores that are interpretable and can be logically combined into holistic metrics.

Evaluator Model. All automated evaluation uses Gemini-2.5-Pro (Comanici et al., 2025) as a unified evaluator. The choice is justified by the model's strong multimodal reasoning capabilities, but also by the practical consideration that using a single evaluator across all tasks ensures cross-task comparability — differences in model performance cannot be attributed to differences in evaluator capability or bias.

Input Structure. The evaluator receives three or four inputs depending on the task:

The model-generated video or image (the output being evaluated).
The ground-truth reference: for Maze, the solution image showing the valid path; for Sudoku, the completed puzzle grid; for ARC-AGI, the ground-truth test output; for Math, the correct answer and solution steps; for navigation tasks, the specified destination and scene configuration.
A structured evaluation prompt that is task-specific and modality-specific (different prompts for video generation vs. image generation, since video evaluation requires temporal analysis while image evaluation is static).
For ARC-AGI specifically, a fourth input: the demonstration examples (input-output pairs from which the transformation rule must be inferred).

Prompt Design for Diagnostic Decomposition. The evaluation prompts are the most critical design element of the system. Each prompt decomposes the "is this correct?" question into multiple binary sub-questions, each targeting a specific failure mode. The prompts follow a consistent template: (1) state the evaluation criteria in precise operational terms, (2) ask for a binary 0/1 judgment per criterion, and (3) request a brief textual justification. This design forces the VLM to make explicit, criterion-by-criterion judgments rather than producing a holistic quality score that would obscure diagnostic information.

For the Maze task (Section 5.3), the evaluation prompt decomposes the assessment into three binary questions:

"Does the green square (start) reach and stop on the red square (end)?" → Target Achievement (1/0)
"Does the green square ever touch or cross a black wall?" → Cross Wall (1/0)
"Does the layout of the black walls or the position of the red square change at any time?" → Maze Changed (1/0)

From the VLM's structured responses, a fourth fine-grained metric is computed: Action Reflection = 1 if the video shows exploratory behavior (backtracking, trying multiple paths) vs. a single direct route (0). This metric is derived from the VLM's textual justification rather than asked directly — the VLM describes the agent's behavior in its justification, and Action Reflection is extracted as a positive signal if that description includes backtracking or multi-path exploration.

For Sudoku (Section 6.2), the decomposition targets the specific logical constraints of the puzzle:

Are any original digits (clues) modified, removed, or displaced? → Clues Changed (1/0)
Are all row, column, and subgrid uniqueness constraints satisfied? → Constraints Violation (inverted: 0 = satisfied, 1 = violated)
What fraction of empty cells are correctly filled? → Completion Accuracy (continuous, compared to ground truth)
Does the sequence show interpretable step-by-step reasoning (gradual cell updates without overwriting correct entries)? → Action Reflection (1/0, video only)

For ARC-AGI (Section 7.2), the decomposition targets the sub-components of visual reasoning:

Did the model identify the transformation pattern from the demonstrations? → Pattern Recognition (1/0)
Does the output preserve the correct grid dimensions and structural layout? → Grid Integrity (1/0)
Are all colors applied correctly according to the transformation rule? → Color Accuracy (1/0)

For Math (Section 8.2), the decomposition separates outcome correctness from process validity:

Does the final answer match the ground truth? → Final Correctness (1/0)
Are all reasoning steps logically valid and mathematically sound? → Intermediate Correctness (1/0)
Does the video exhibit self-correction behavior (revising incorrect steps)? → Action Reflection (1/0, video only)

For Embodied Navigation (Section 9.3), the evaluation is substantially more complex because a generated video must simultaneously satisfy geometric, physical, and instruction-following constraints. The evaluation rubric is decomposed into three families of metrics:

Task Completeness Metrics (geometry only, ignoring visual fidelity):

Success Score 3D (S.S. 3D): binary — 1 if the agent's final position lies within the destination volume in 3D.
Oracle Success Score 3D (O.S. 3D): binary — 1 if the agent enters the destination vicinity at any point during the trajectory (more lenient than S.S. 3D, providing partial credit for near-misses).
Success Score 2D (S.S. 2D): binary — 1 if the agent's final position lies within the goal footprint in the 2D overhead map (for tasks with 2D maps: Top-down View and SLAG).
Oracle Success Score 2D (O.S. 2D): binary — 1 if the agent's path ever intersects the goal region on the 2D map.
Trajectory Alignment Score: binary (SLAG only) — 1 if the agent's 2D projected route is consistent with its 3D motion path, focusing on major turns and spatial transitions.

Physical Understanding Metrics (assessing physical plausibility and spatial coherence):

Object Semantic Score (Obj. Sem.): binary — 1 if the agent does not collide with, pass through, or visually intersect solid structures (walls, furniture, appliances). This directly tests Physical Reasoning.
Agent Consistency Score (Agent Con.): binary — 1 if the agent's trajectory is continuous (no teleportation) and exactly one agent appears throughout the sequence. For image generation: additionally requires that the trajectory remains continuous across frames and the number of agents is preserved.
Spatial Alignment Score (Spa. Ali.): binary — 1 if the agent's heading, motion direction, and elevation changes remain coherent with the expected physical layout. For image generation: additionally requires that the initial position is visually identifiable (when provided) and the agent's initial facing direction aligns with its first movement.

Instruction Following Metrics (assessing whether the model respects the problem definition):

Destination Integrity Score (Des. Inte.): binary — 1 if the destination region is preserved unchanged (size, position, texture, appearance) and the agent does not rely on hallucinated alternatives. According to supplementary rules: the red-marked target region must remain unchanged; the agent must not fabricate new goal markers or look-alike objects; the agent must stop within the original destination region.
Scene Consistency Score (Scene Con.): binary — 1 if the environment remains static throughout the video — no objects, lighting, geometry, or layout elements appear, disappear, deform, or shift in a way that violates the static-scene assumption.

For Physical Commonsense (Section 14.3), the evaluation decomposition targets the distinction between visual quality and physical correctness that is central to the paper's thesis:

Physics Accuracy (1/0): evaluates whether the video strictly obeys fundamental physical laws. Checks if motion adheres to gravity, momentum, and friction; verifies that object interactions are plausible. Score 0 for violations such as floating objects, unrealistic speeds, incorrect trajectories, or deviation from the scenario's "Physics Focus."
Motion Quality (1/0): assesses temporal coherence and naturalness of movement. Verifies that motion follows the expected pattern and remains smooth, continuous, and free of jerky discontinuities or unnatural accelerations.
Visual Realism (1/0): measures visual fidelity and believability. Checks if objects and materials appear realistic, lighting and shadows are consistent, and the scene composition is plausible. Score 0 for significant visual artifacts, glitches, or lack of photorealism — critically, this is separate from Physics Accuracy.
Prompt Adherence (1/0): verifies that all key elements (objects, setting) from the prompt are present and that the specified action actually occurs. Score 0 for significant mismatches between generated content and text prompt.

The key design insight across all evaluation prompts is the modality-specific adaptation. For tasks where both video and image generation are possible (Maze, Sudoku, ARC-AGI, Embodied Navigation), the prompts have separate variants for video vs. image evaluation. The video variants include temporal checks (frame-by-frame analysis, detecting transient violations), while the image variants focus on static correctness. For example, Cross Wall for images is: "1 if the blue path touches or crosses black walls, 0 if fully contained within white corridors," while for video it is: "1 if the green square crosses a black wall in any frame, 0 only if it stays on white paths at all times." The temporal variant requires the VLM to maintain state across frames, which is fundamentally harder than static evaluation and is a source of the systematic AutoEval errors identified in the human evaluation comparisons.

Metrics Aggregation: From Fine-Grained Diagnostics to Holistic Primary Metrics

The fine-grained metrics would be misleading if reported in isolation because models can achieve high scores on individual dimensions while failing catastrophically on the complete task. The paper's key design choice is the use of strict holistic primary metrics that require simultaneous satisfaction of all sub-metrics.

For each task, the primary metric (called Overall Score for Maze/Sudoku, Valid Solution for ARC-AGI, Overall Success Rate for Math, Overall Success for Embodied Navigation, Overall for Physical Commonsense) is defined as a logical conjunction:

For the Maze task:

$\text{Overall} = (\text{Maze Changed} = 0) \land (\text{Cross Wall} = 0) \land (\text{Target Achievement} = 1)$

where the three conditions are binary predicates. This means that a model that successfully reaches the target but crosses a wall in the process (Cross Wall = 1) receives Overall = 0 — the partial success on Target Achievement is invalidated by the rule violation.

For Sudoku:

$\text{Overall} = (\text{Clues Changed} = 0) \land (\text{Constraints Violation} = 0) \land (\text{Completion Accuracy} = 1)$

where Completion Accuracy = 1 means 100% of empty cells are correctly filled. A model that fills 80% of cells correctly but violates one row constraint receives Overall = 0.

For Math:

$\text{Overall} = (\text{Final Correctness} = 1) \land (\text{Intermediate Correctness} = 1)$

This conjunction is particularly important because it captures the "hallucination of competence" pattern: Veo-3 achieves 74% Final Correctness on GSM8K but only 12% Intermediate Correctness, resulting in 12% Overall Success — because the model frequently arrives at the correct answer through invalid reasoning steps.

For Embodied Navigation tasks, the conjunction is substantially more restrictive because there are up to 7 or 10 binary conditions. For 3D Real-World Navigation:

$\text{Overall} = \text{S.S.3D} \land \text{O.S.3D} \land \text{Obj.Sem.} \land \text{Agent Con.} \land \text{Spa.Ali.} \land \text{Des.Inte.} \land \text{Scene Con.}$

For SLAG, with its additional 2D and trajectory alignment metrics:

$\text{Overall} = \text{S.S.3D} \land \text{O.S.3D} \land \text{S.S.2D} \land \text{O.S.2D} \land \text{Traj.Ali.} \land \text{Obj.Sem.} \land \text{Agent Con.} \land \text{Spa.Ali.} \land \text{Des.Inte.} \land \text{Scene Con.}$

For Physical Commonsense:

$\text{Overall} = \text{Physics Accuracy} \land \text{Motion Quality} \land \text{Visual Realism} \land \text{Prompt Adherence}$

The paper also defines intermediate Gate Metrics (composite metrics that intersect geometric success with semantic and physical validity) to identify where the bottlenecks occur. For example, in Embodied Navigation:

Success (3D) with Original Destination = S.S.3D ∧ Des.Inte. ∧ Scene Con. — this gate catches models that achieve geometric success through scene manipulation (fabricating a target at the agent's feet or warping the room layout).
Physics Validness = Obj.Sem. ∧ Agent Con. ∧ Spa.Ali. — this gate catches models that reach the destination through physically impossible means (clipping through walls, teleporting).

Why strict conjunction rather than weighted sum: The paper explicitly justifies this design choice in Section 4.3: "We prioritize this holistic measure to address the disparity between partial success and complete correctness—a gap that typically inflates performance estimates by 1.2–4× when ignored." The empirical evidence supporting this claim is visible throughout the results tables: Veo-3 achieves 90% Success Score 3D on Panoramic Last-Mile Navigation (floor01) but only 73% Overall Success (Table 26) — a 1.23× inflation factor. On 3D Real-World Navigation, Veo-3 achieves 85% Success Score 3D (floor01) but only 3.33% Overall Success under human evaluation (Table 31) — a 25.5× inflation factor. The gap is not constant across tasks; it is largest for tasks requiring fine-grained physical constraint satisfaction (navigation, maze wall-following) and smallest for tasks where partial correctness approximates full correctness (image-based Sudoku where the model either solves the puzzle or doesn't).

The metric cascade — the progressive drop-off from least to most restrictive metric — is itself a diagnostic tool. The paper uses this cascade to identify which specific sub-ability is the bottleneck. For Nano-banana on ARC-AGI v1 (Table 9): Grid Integrity 55.79% → Pattern Recognition 28.42% → Color Accuracy 12.63% → Valid Solution 9.21%. This cascade reveals that Color Accuracy is the critical bottleneck (only 44% of pattern-recognizing outputs achieve correct colors), implying that the model can learn structural rules but struggles with precise color-based execution. This type of diagnostic would be invisible if only the primary metric were reported.

Human Evaluation Protocol: Calibrating Automated Evaluation

The paper's human evaluation is not a small-scale validation addendum but a core methodological contribution. The protocol is designed to answer a specific question: can VLM-based evaluation reliably assess reasoning in generated content, or does it introduce systematic biases that inflate or deflate performance estimates?

Annotation Interface Design. The authors developed a web-based annotation platform (Figure 3) with purpose-built features for temporal analysis:

Full video playback controls including frame-by-frame navigation and adjustable playback speed — essential for detecting transient violations that occur within single frames (e.g., the green square passing through a wall between frame 3 and frame 4 in a maze navigation video).
The original task prompt displayed alongside the generated video — ensuring annotators always have the correct problem context.
Structured evaluation forms tailored to each task type, with binary checkboxes for each fine-grained metric and optional free-text fields for annotator comments.
Task-specific visual aids: for Maze, the ground-truth solution image is shown for comparison; for ARC-AGI, the demonstration examples are displayed alongside the generated output.

Annotator Recruitment and Training. The paper recruited 6 annotators with bachelor education backgrounds. The training process was structured to ensure reliable, calibrated judgments:

A 4-hour instruction session covering the evaluation criteria for all tasks, with emphasis on distinguishing between genuine physical violations and acceptable visual artifacts.
A 50-video practice phase where annotators independently evaluated the same set of videos and then discussed discrepancies in calibration meetings.
Calibration meetings where annotator disagreements were resolved through consensus discussion and refinement of the evaluation guidelines.

The multi-stage training design addresses a well-known challenge in reasoning evaluation: different annotators may apply different standards for what constitutes a "physical violation" or "logical error." The calibration meetings serve to align these standards before the main evaluation.

Evaluation Protocol. The human evaluation covers a curated subset of generated outputs, focusing on the top-performing video model (Veo-3) because it exhibits the most interesting failure modes:

Maze: Veo-3 outputs across both DFS and Wilson's generators, all three difficulty levels (specific sample counts per setting reported in Table 5).
Sudoku: Veo-3 outputs across both grid sizes (4×4, 9×9) and all three difficulty levels (Table 7).
ARC-AGI: 98 Veo-3 generated videos — 60 from v1, 38 from v2 (Tables 16 and 17).
Embodied Navigation: Veo-3 outputs across all four navigation subtasks, all 24 hard-level configurations (Tables 25, 27, 29, 31, 33).
Physical Commonsense: 45 Veo-3 generated videos across Physical Concepts and Sports Scenarios (Table 38).

The human evaluators use the same evaluation prompts and binary scoring rubrics as the VLM evaluator, ensuring that discrepancies between AutoEval and HumanEval can be attributed to differences in evaluator capability rather than differences in evaluation criteria.

Metrics for Comparing AutoEval and HumanEval. The paper reports side-by-side tables comparing AutoEval vs. HumanEval scores for each metric in each task setting. The comparison is not summarized with a single correlation coefficient but is presented as raw pass rates, enabling readers to identify where and in which direction discrepancies occur. Key patterns identified include:

AutoEval overestimates physical constraint satisfaction: For Maze (Table 5), AutoEval reports Cross Wall rates of 16–26% while HumanEval detects 70–100% — a 3–5× increase. The VLM systematically misses transient wall-crossing events that occur in single frames, likely due to frame-dropping or insufficient temporal resolution.
AutoEval overestimates overall success on reasoning tasks: For ARC-AGI v1 (Table 16), AutoEval reports 4.72% Valid Solution while HumanEval finds 0.00% across all 60 evaluated cases. The VLM occasionally credits visually plausible but logically incorrect outputs as correct.
AutoEval underestimates Color Accuracy on ARC-AGI: For ARC-AGI v1 (Table 16), AutoEval reports 7.61% Color Accuracy while HumanEval reports 23.33% — a 3.1× increase. The VLM may be overly sensitive to minor color deviations that humans perceive as acceptable matches.
AutoEval underestimates Action Reflection: For Maze (Table 5), AutoEval reports 0–3% Action Reflection while HumanEval reports 40–90%. Humans are sensitive to subtle exploratory behaviors (hesitations, micro-adjustments) that the VLM fails to classify as meaningful reflection.
AutoEval underestimates Physical Commonsense performance: For Physical Concepts (Table 38), AutoEval reports 41.67% Overall while HumanEval reports 77.27% — a 29-point gap. Human evaluators are more tolerant of minor visual artifacts than the VLM, which applies overly strict criteria for temporal coherence and physical accuracy.

These discrepancies are not random noise — they reveal systematic biases in VLM-based evaluation that are task-dependent and metric-dependent. The paper's key methodological finding is that VLM-based evaluation cannot be assumed reliable for reasoning assessment without task-specific human calibration, and that the reliability of AutoEval varies dramatically across tasks (reasonably reliable for static image evaluation, severely unreliable for temporally dense video evaluation).

Model Configuration and Generation Protocol

Model Selection. The paper evaluates seven generative models spanning both video and image modalities, including closed-source and open-weights models (Table 3):

Video models: Sora-2 (OpenAI, closed), Veo-3 (Google DeepMind, closed), Wan-2.2 (open-weights, from the Wan team).
Image models: Nano-banana (Google DeepMind, closed — same family as Gemini), Nano-banana Pro (Google DeepMind, closed), GPT-4o-image (OpenAI, closed), Qwen-image (open-weights, from the Qwen team).

The selection is motivated by coverage of the SOTA landscape: the top-performing video models (Sora-2 and Veo-3), the top-performing image models (Nano-banana Pro), the most capable open-weights alternatives (Wan-2.2, Qwen-image), and a representative mid-tier model (GPT-4o-image). The authors argue that these models are "representative of the capabilities of contemporary generative models" (Section 4.4) and span the range from near-zero to substantial reasoning performance.

Generation Settings. To ensure robust performance estimation and fair comparison, the paper implements several methodological controls:

Multiple samples per prompt: 5 samples are generated per prompt for every model. This accounts for stochastic variability in generation (due to random seeds, temperature, or diffusion noise) and provides a more reliable estimate of model capability than single-sample evaluation.
Default API parameters: For closed-source models (Sora-2, Veo-3, GPT-4o-image, Nano-banana, Nano-banana Pro), the paper strictly adheres to default API parameters without task-specific tuning. This ensures that the evaluation reflects out-of-the-box zero-shot performance rather than cherry-picked hyperparameters.
Recommended configurations: For open-weights models (Wan-2.2, Qwen-image), the paper uses the configurations recommended in their respective release documentation, again avoiding task-specific optimization.
Zero-shot evaluation: No model receives task-specific fine-tuning, few-shot examples (beyond what is inherent in the task design — e.g., ARC-AGI's demonstration examples), or prompt engineering beyond the standardized evaluation prompts. The authors explicitly state this in Section 4.2: they benchmark "zero-shot reasoning capabilities" following the paradigm established by Wiedemer et al. (2025), Guo et al. (2025), and Tong et al. (2025).

Task-to-Model Mapping. Not all models are evaluated on all tasks. Physical Commonsense is evaluated only on video models because "static image generators lack temporal modeling capabilities and cannot represent causal interactions unfolding over time" (Section 14.3). The Math task's Omni-MATH sub-analysis is noted to include only video generation evaluation results (Table 19 note: "These results represent video generation evaluations only; image generation was not performed for this dataset"). For other tasks (Maze, Sudoku, ARC-AGI, Embodied Navigation), both video and image models are evaluated, enabling the paper's central modality comparison.

Evaluation Scale. The full benchmark comprises 1,853 testing samples across three domains and ten tasks (Table 2). With 5 samples per prompt per model, the evaluation produces 5 × 1,853 = 9,265 generated outputs per model for full coverage (in practice, not all models run on all tasks). Across 7 models, this yields tens of thousands of evaluations, each of which requires VLM inference for fine-grained metric extraction — a computationally substantial undertaking that the paper manages through automated evaluation infrastructure.

Summary of Design Choices and Their Justifications

Five-ability framework grounded in cognitive science rather than arbitrary task taxonomy: ensures that the benchmark measures abilities fundamental to intelligence itself, not just video generation quality. The grounding in developmental psychology (Spelke & Kinzler, 2007; Lake et al., 2017) provides theoretical validity for which abilities are selected and why they should be evaluated together.
Strict binary metrics with logical conjunction rather than continuous quality scores or weighted averages: forces models to satisfy all constraints simultaneously, preventing the "illusion of competence" where partial success masks fundamental failure. The empirical justification is that ignoring the gap between partial and complete correctness inflates performance estimates by 1.2–4× (and up to 25× in the worst cases).
Systematic hard-level control with semantically meaningful axes rather than random difficulty variation: enables attribution of performance changes to specific factors (grid size, trajectory distance, textual vs. visual goal specification), supporting diagnostic rather than merely descriptive evaluation.
VLM-based automated evaluation with extensive human calibration rather than assuming AutoEval is reliable: treats evaluation methodology itself as something to be benchmarked. The human evaluation reveals that VLM-based AutoEval has systematic, task-dependent biases (overestimating physical constraint satisfaction; underestimating action reflection), establishing that human ground truth is essential for reasoning evaluation.
Multi-domain, multi-modality design rather than focusing on a single domain: enables the discovery of cross-domain patterns (the "temporal tax" on reasoning in video models; image models' superiority on abstract logic) that would be invisible in a domain-specific benchmark.
Default zero-shot generation rather than task-specific fine-tuning or prompting: reflects how these models are used in practice (as general-purpose generation APIs) and prevents benchmark-specific optimization from masking the fundamental reasoning deficits that MMGR aims to characterize.

4. Key Insights and Innovations

Innovation 1: Generative Reasoning as a Distinct Capability from Visual Quality

The central conceptual contribution of MMGR is not any single metric or task, but the decomposition of generative model evaluation into two independent axes: perceptual quality and reasoning correctness, and the empirical demonstration that these axes are decoupled—sometimes anticorrelated—in current state-of-the-art models.

The dominant assumption in the field prior to this work was that visual quality metrics (FVD, IS, CLIP-score) and reasoning capability would correlate positively in large generative models—that a model sophisticated enough to produce photorealistic video would, by virtue of its scale and training, internalize the physical and logical constraints that govern the depicted content. This assumption is implicit in the positioning of video generation models as "world simulators" (OpenAI, 2024b; Ha & Schmidhuber, 2018) and in the practical enthusiasm for using these models as foundations for robotic planning and scientific simulation.

MMGR's results systematically dismantle this assumption. The most striking evidence comes from the Physical Commonsense task (Table 35), where Wan-2.2 achieves 84–96% Visual Realism across both Physical Concepts and Sports Scenarios, yet records only 24% Overall Success—because its Physics Accuracy (50.67%) and Prompt Adherence (30.00%) collapse. Wan-2.2 produces videos that look photorealistic but depict physically impossible interactions. Conversely, Sora-2 achieves 70% Overall Success with only marginally higher Visual Realism (92%), demonstrating that physical correctness and visual fidelity are not the same optimization target. The model that "looks best" is not the model that reasons best.

This decoupling is not a minor calibration issue—it is a fundamental reframing of how the field should evaluate generative models. If visual quality and reasoning correctness can vary independently, then optimizing for visual quality (as current training objectives do) provides no guarantee of progress on reasoning. The paper makes this explicit in the concluding diagnosis (Section 15): "optimization objectives that reward perceptual fidelity rather than logical validity" are identified as one of three fundamental bottlenecks. This is a negative result with significant implications: it implies that scaling current architectures and training recipes will not automatically produce reasoning-capable models; explicit intervention in training objectives, data composition, or architecture is required.

The decoupling is further substantiated by the Math task (Table 20), where the "hallucination of competence" pattern—high Outcome Success but low Process Success—is a direct consequence of models optimizing for the appearance of correctness rather than actual logical derivation. Veo-3 achieves 74% Final Correctness on GSM8K but only 12% Intermediate Correctness, meaning that 62% of its "correct" answers were produced through invalid reasoning. The video generation process morphs visual elements to reach a memorized answer pattern rather than constructing a valid proof. This diagnostic would be invisible under any evaluation framework that reports only answer accuracy—a standard practice in LLM evaluation that this paper demonstrates is dangerously misleading when applied to generative video.

The significance of this innovation is that it changes the evaluation target. Prior work asked: "How good does this model's output look?" MMGR asks: "Does this model understand what it is generating?" The answer—that current SOTA models largely do not—has implications for deployment (these models should not be trusted as simulators), for research prioritization (improving visual quality is hitting diminishing returns for reasoning), and for the broader AI narrative that scaling alone will produce general intelligence.

Innovation 2: The "Temporal Tax"—Video Generation as an Impediment to Reasoning

Perhaps the most counterintuitive finding in MMGR is that video generation models consistently underperform image generation models on tasks requiring abstract reasoning, despite the intuitive expectation that temporal processing should enhance sequential reasoning. The paper identifies and names this phenomenon: the "temporal tax" on reasoning.

The prior expectation—implicit in work like Wiedemer et al. (2025) on Chain-of-Frames reasoning and Tong et al. (2025) on "thinking with video"—was that video models, with their capacity to maintain and manipulate visual state across frames, would possess inherent advantages for tasks requiring multi-step reasoning. The temporal dimension, in this view, provides a workspace for explicit computation: a model can show its work, backtrack from errors, and construct solutions incrementally.

The MMGR results systematically refute this expectation. On ARC-AGI—the purest test of abstract visual reasoning in the benchmark—Nano-banana Pro achieves 30.54% on v1 while Sora-2, the best video model, achieves 20.18% (Table 9). On v2, the gap widens: Nano-banana Pro maintains 30.36% while Sora-2 collapses to 1.33%—a 93% relative decline (Table 10). On Math, Nano-banana Pro achieves 97.83% Overall Success on GSM8K versus Sora-2's 30.00%, and 66.67% on AIME 2025 versus 0.00% for Sora-2 (Table 20). Across all abstract reasoning tasks, the modality hierarchy is consistent: image models outperform video models.

The paper's diagnostic metrics reveal why this inversion occurs. Video models do not fail because they lack temporal capability—they fail because the temporal dimension introduces an additional consistency burden that competes with logical coherence. The ARC-AGI case studies (Figures 11a, 11b) show the defining failure mode: video models cannot maintain static demonstration examples across frames. The demonstrations—which define the transformation rule—undergo progressive hallucination (color shifts, pattern deformations, structural degradation), effectively corrupting the problem definition itself. This "context drift" means that by the time the model generates what it thinks is a solution, it is solving a different problem than the one specified in the initial prompt. Image models avoid this failure mode entirely because they produce a single static output with no intermediate frames to corrupt.

Similarly, in the Sudoku task, video models exhibit "temporal drift" where digits morph or overwrite across frames (Section 6.3, Figure 8). Veo-3 demonstrates visible self-reflection—editing entries to resolve conflicts—but these edits frequently introduce new violations because the model lacks a persistent constraint representation. The temporal generation process becomes a source of noise rather than a reasoning aid. Image models, producing a single static grid, sidestep this instability entirely.

This finding is significant beyond the specific numbers because it challenges a core assumption of the video generation-as-reasoning paradigm. The idea that temporal generation provides a natural scaffold for reasoning—that "showing your work" in video form enhances correctness—appears to be false for current architectures. The temporal tax is not a minor overhead; it can reduce performance by 3-10× on complex reasoning tasks compared to the image-modality baseline. The paper's diagnosis (Section 15) identifies this as stemming from architectures that "treat mathematical derivation as a visual texture to be morphed, rather than a semantic chain to be constructed"—a fundamental mismatch between temporal video generation mechanisms and symbolic reasoning requirements.

This insight is a negative result with positive implications for research direction: it suggests that bridging video generation with reasoning will require architectural innovations that decouple reasoning state from visual rendering—mechanisms to maintain persistent symbolic constraints across frames that are not subject to visual drift. The current approach, where every frame is generated from the previous frame's visual features, is structurally incompatible with the invariant representation that logical reasoning requires.

Innovation 3: The "Hallucination of Competence"—A Diagnostic for Distinguishing Pattern Matching from Reasoning

MMGR introduces a diagnostic pattern that the paper terms the "hallucination of competence"—situations where a model achieves high outcome success (correct final answers) but low process success (valid intermediate reasoning). This pattern serves as a litmus test for distinguishing genuine reasoning from sophisticated pattern matching, and the paper demonstrates it across multiple tasks and modalities.

Prior evaluation paradigms in both LLM and generative model assessment have overwhelmingly focused on outcome metrics: did the model produce the right answer? This is true of standard math benchmarks (GSM8K, MATH), code generation benchmarks (HumanEval), and most generative evaluation frameworks (VBench, EvalCrafter). The implicit assumption is that outcome correctness implies process correctness—that a model arriving at the right answer probably reasoned correctly to get there.

MMGR's design—specifically the decomposition of success into process and outcome components—enables the systematic detection of cases where this assumption fails. The Math task's separation of Final Correctness and Intermediate Correctness (Section 8.2) provides the cleanest operationalization. Veo-3 on GSM8K: 74% Final Correctness vs. 12% Intermediate Correctness (Table 20). This 62-percentage-point gap means that in the majority of cases where Veo-3 produces the correct answer, it does so through invalid reasoning steps—visual morphing that "teleports" to the answer rather than deriving it. The model has memorized answer patterns from its training data and learned to generate video sequences that terminate at those answers, but it has not learned the logical transformations that connect premises to conclusions.

The diagnostic generalizes beyond math. In the Maze task, Veo-3 achieves 62% Target Achievement but only 42% Overall Success (Table 4, DFS Easy, AutoEval) because it reaches the goal through physically impossible means (wall-crossing). In humans, the equivalent would be solving a maze by drawing a line that ignores the walls—the outcome is "reached the goal" but the reasoning is "ignored the constraints." The Maze task's Cross Wall metric operationalizes process validity in the spatial domain, and the gap between Target Achievement and Overall quantifies the hallucination of competence.

The concept is significant because it provides a portable diagnostic criterion: any evaluation framework that reports only outcome metrics is likely overestimating model capability by precisely this mechanism. The paper quantifies the inflation factor as 1.2–4× in most cases (Section 4.3), but it can be much higher—up to 25× for 3D Real-World Navigation (Veo-3: 85% Success Score 3D vs. 3.33% Overall under human evaluation, Table 31). This is not a static correction factor; the inflation varies by model, task, and difficulty. The implication for the broader evaluation community is that process-level evaluation is not a luxury for detailed analysis but a necessity for accurate assessment.

The hallucination of competence also illuminates the training data dynamics driving current model behavior. The Math Omni-MATH deep dive (Tables 21, 22) reveals "inverse scaling" patterns—Veo-3 performs better on the hardest problems (T4: 5% Overall) than on intermediate ones (T3: 1.47% Overall)—suggesting that T4 problems follow canonical Olympiad templates that the model has memorized, while T3 problems require genuine intermediate reasoning that the model cannot perform. The hallucination of competence is thus not random noise; it is a systematic consequence of training on datasets where problem-answer pairs are abundant but reasoning trajectories are scarce. The model learns the answer distribution without learning the reasoning process that generates it.

Innovation 4: VLM-Based Evaluation as a First-Class Problem Rather Than a Solved Tool

MMGR treats evaluation methodology itself as an object of empirical study, systematically comparing VLM-based automated evaluation against human judgment across tasks, metrics, and difficulty levels. The results establish that VLM-based evaluation for generative reasoning is not a solved problem—it introduces systematic, task-dependent biases that can inflate or deflate performance estimates by factors of 2-5×.

Prior work in generative model evaluation (Huang et al., 2024; Liu et al., 2024b; Wiedemer et al., 2025) has increasingly adopted VLM-based evaluators as a scalable alternative to human evaluation, with the implicit assumption that modern VLMs like Gemini are sufficiently capable to serve as reliable judges for visual reasoning tasks. This assumption has generally been validated by reporting aggregate correlation with human judgments, but it has rarely been stress-tested across task types, difficulty levels, and specific failure modes.

MMGR's human evaluation results reveal that this assumption fails in specific, predictable ways. The most critical failure mode is AutoEval's blindness to transient violations in temporally dense tasks. On the Maze task (Table 5), AutoEval reports Cross Wall rates of 16-26% while HumanEval detects 70-100%—the VLM systematically misses wall-crossing events that occur within single frames, likely due to frame-dropping or insufficient temporal resolution when processing fast-moving agents. This results in AutoEval reporting moderate competence (42-52% Overall) while HumanEval reveals near-zero true performance (0-20% Overall). On ARC-AGI (Tables 16, 17), AutoEval reports 4.72% Valid Solution for Veo-3 on v1, while HumanEval finds 0.00% across all 60 evaluated cases.

Conversely, AutoEval can be overly strict on metrics that humans treat as approximate. On ARC-AGI Color Accuracy (Table 16), AutoEval reports 7.61% while HumanEval reports 23.33%—the VLM appears to detect subtle RGB deviations that humans perceive as acceptable matches. On Physical Commonsense, AutoEval reports 41.67% Overall for Veo-3 on Physical Concepts while HumanEval reports 77.27% (Table 38)—a 29-point gap, with the VLM penalizing minor temporal artifacts that human observers either do not notice or do not consider violations.

The direction of AutoEval error is not uniform—it depends on the metric, the task, and the failure mode. This means there is no simple calibration factor that can correct VLM-based evaluation across the board. The paper's contribution is not to discredit VLM-based evaluation (which remains the only scalable approach for large benchmarks) but to establish that task-specific human calibration is essential and to identify which types of metrics are most vulnerable to AutoEval error. Temporally fine-grained metrics (Cross Wall, Scene Consistency in navigation) are systematically overestimated; color-precision metrics are systematically underestimated; action recognition metrics (Action Reflection) are systematically underestimated.

This insight is significant because it raises the bar for evaluation rigor in the generative AI field. A benchmark that reports only VLM-based evaluation without human validation is, based on these results, reporting numbers that may be substantially inaccurate for the most diagnostically interesting metrics. The paper's practice of presenting AutoEval and HumanEval side-by-side—rather than reporting a single "validated" metric—models a more rigorous approach that the field would benefit from adopting.

Innovation 5: Difficulty-Dependent Reasoning Profiles as a Diagnostic for Generalization vs. Memorization

Beyond reporting aggregate performance, MMGR's systematic difficulty stratification enables the construction of difficulty-dependent reasoning profiles—curves that reveal whether a model's performance degrades gracefully (suggesting genuine generalization) or collapses abruptly (suggesting memorization of specific patterns). The paper uses this diagnostic to distinguish between models that have internalized abstract transformation rules and models that have memorized training distribution artifacts.

The cleanest demonstration is the ARC-AGI difficulty breakdown (Table 11 and the v1-to-v2 comparison). Nano-banana Pro exhibits what the paper calls "robust stability": performance remains nearly flat across Easy (30.89%), Medium (30.39%), and Hard (30.23%) cases on the combined v1+v2 dataset. This flat difficulty curve is exactly what would be expected from a model that has learned abstract transformation rules—harder instances (larger grids, more objects, more complex color mappings) should not be substantially more difficult if the underlying rule-reasoning mechanism is the same. The model's Grid Integrity also shows only modest degradation (86.18% → 86.74% → 77.91%), consistent with a reasoning process that is difficulty-agnostic but limited by execution precision.

In contrast, Sora-2 exhibits "difficulty-sensitive collapse": performance drops from 22.22% (Easy) to 16.33% (Medium) to 10.64% (Hard), and Pattern Recognition plummets from 76.07% to 40.43%. More dramatically, Sora-2 collapses from 20.18% on v1 to 1.33% on v2—a 93% relative decline (Tables 9, 10). This pattern is consistent with a model that has memorized transformation templates from the v1 training distribution and cannot generalize to the novel pattern families in v2. The difficulty curve is not flat because the model's "reasoning" is actually retrieval: harder cases deviate further from memorized templates and thus fail more frequently.

The distinction matters because it reveals that scale alone does not produce generalization. Sora-2 and Veo-3 are among the largest generative models ever built, trained on enormous video corpora, yet their reasoning profiles are consistent with sophisticated pattern matching rather than abstract rule induction. Nano-banana Pro, an image model with substantially different architecture, achieves genuine (if imperfect) generalization. The paper's difficulty stratification makes this distinction empirically visible and quantifiable.

The Omni-MATH results (Tables 21, 22) provide additional evidence with a different structure. The "inverse scaling" pattern—Veo-3's Overall Success rising from 6.06% (T0) to 1.47% (T3) to 5.00% (T4)—is a difficulty-dependent profile that is inconsistent with genuine reasoning (where harder problems should be strictly harder) but consistent with template memorization (where the hardest problems follow the most rigid, canonical templates that are easiest to retrieve). The category breakdown supports this: Veo-3 achieves its best performance in Geometry (8.33%, Table 22), where visual-spatial proof templates are most stereotyped, and collapses in abstract domains like Calculus and Discrete Math (0.00%) where problem structures are more variable.

This innovation is significant because it provides an empirical criterion for distinguishing reasoning from memorization in generative models—a problem that has been central to AI evaluation since at least the ARC-AGI benchmark's design (Chollet, 2019) but has not been systematically operationalized for generative video. The difficulty-dependent reasoning profile is a portable diagnostic: any model that shows flat performance across difficulty levels with graceful degradation passes the generalization test; any model that shows abrupt collapse at specific difficulty thresholds or between dataset versions fails it. The paper's contribution is both the diagnostic concept and the empirical demonstration that current SOTA models span this spectrum—from Nano-banana Pro's robust stability to Sora-2's memorization-driven collapse—providing a concrete roadmap for what generalization looks like and establishing that it is achievable (if only in specific architectures and domains).

5. Experimental Analysis

Evaluation Methodology

Dataset. The MMGR benchmark comprises 1,853 evaluation samples across three domains and ten tasks (Table 2). Abstract Reasoning contributes 1,323 samples (Maze: 240, Sudoku: 300, ARC-AGI: 456, Math: 327), Embodied Navigation contributes 480 samples (120 per subtask), and Physical Commonsense contributes 50 samples. For Abstract Reasoning and Embodied Navigation, difficulty is systematically stratified; for Physical Commonsense, samples are balanced across interaction types and scenario contexts. The benchmark sources scenes from Matterport3D, HM3D, and Habitat for navigation, adapts ARC-AGI v1 and v2 for abstract reasoning, and uses VideoPhy ontologies for physical commonsense.
Base model(s). Seven models spanning two modalities are evaluated (Table 3): three video generation models (Veo-3, Sora-2, Wan-2.2) and four image generation models (Nano-banana, Nano-banana Pro, GPT-4o-image, Qwen-image). Veo-3 and Sora-2 represent the current closed-source state-of-the-art in video generation; Nano-banana Pro represents the strongest image generation model; Wan-2.2 and Qwen-image are the primary open-weights alternatives. The authors justify this selection as "representative of the capabilities of contemporary generative models" (Section 4.4), spanning the range from near-zero to substantial reasoning performance.
Metrics. Every task defines a set of fine-grained binary diagnostic metrics (0/1) that probe specific reasoning failures, and a strict holistic primary metric computed as the logical conjunction of all fine-grained metrics—a sample passes only if every sub-metric is simultaneously satisfied. For example, Maze Overall requires Maze Changed=0 AND Cross Wall=0 AND Target Achievement=1 (Section 5.3); Math Overall requires Final Correctness=1 AND Intermediate Correctness=1 (Section 8.2); Embodied Navigation Overall requires satisfaction of 7–10 binary checks spanning Task Completeness, Physical Understanding, and Instruction Following (Section 9.3). The conjunction design is motivated by empirical observation that partial success inflates performance estimates by 1.2–4× compared to complete correctness (Section 4.3).
Baselines. The primary comparison is cross-modal: video generation models vs. image generation models on identical tasks. Within each modality, all evaluated models serve as baselines for each other—Veo-3, Sora-2, and Wan-2.2 are compared as video baselines; Nano-banana, Nano-banana Pro, GPT-4o-image, and Qwen-image as image baselines. The paper does not include a "random" or "chance" baseline, as most tasks involve complex generative outputs where chance performance would be near-zero and uninformative. For Physical Commonsense, only video models are evaluated (image models cannot represent temporal physical interactions; Section 14.3).
Generation budget / compute accounting. Five samples are generated per prompt for every model (Section 4.2), with default API parameters for closed-source models and recommended configurations for open-weights models. The paper measures performance as pass@1 averaged across 5 samples (i.e., for each prompt, 5 independent generations are produced and evaluated; the reported metric is the fraction of these 5 generations that achieve the primary metric, averaged across all prompts). This accounts for stochastic variability in generation and provides a more robust estimate than single-sample evaluation.
Cross-validation / statistical protocol. The benchmark employs two distinct evaluation tracks: VLM-based automated evaluation (AutoEval) using Gemini-2.5-Pro with structured, task-specific rubrics, and systematic human evaluation (HumanEval) on curated subsets of Veo-3 outputs for calibration (Section 4.5). Six annotators with bachelor education backgrounds undergo a 4-hour instruction session, a 50-video practice phase, and calibration meetings before evaluation. HumanEval sample sizes are: Maze (all difficulty-generator combinations for Veo-3), Sudoku (all grid-size difficulty combinations), ARC-AGI (98 videos: 60 v1 + 38 v2), Embodied Navigation (across all 24 hard-level configurations, with per-task sample sizes reported in Tables 25, 27, 29, 31, 33), and Physical Commonsense (45 Veo-3 videos; Table 38). No cross-validation is needed for strategy selection because MMGR is a fixed benchmark with a deterministic evaluation protocol—all metric definitions and difficulty stratifications are specified a priori.

Main Quantitative Results

The MMGR results span three domains and ten tasks, each with task-specific metrics and breakdowns. The following synthesis organizes findings by the paper's central analytical themes: cross-modal comparison, the hallucination of competence, difficulty-dependent scaling, and the AutoEval-HumanEval reliability gap.

The most consistent finding across the benchmark is that image generation models substantially outperform video generation models on tasks requiring abstract reasoning, symbolic manipulation, or logical constraint satisfaction. This result is counterintuitive given that video models possess temporal processing capabilities that might be expected to benefit sequential reasoning, and it holds across multiple tasks at multiple difficulty levels.

On Maze (Table 4), Nano-banana Pro achieves 17.50% Overall on DFS Easy (3×3–5×5), compared to Veo-3's 42.00%—but this apparent video advantage is misleading. The AutoEval inflation is severe: under HumanEval (Table 5), Veo-3's DFS Easy Overall collapses to 10.00%, while Nano-banana benefits from producing static solution paths that avoid temporal corruption. The image advantage becomes clear on Hard mazes: Nano-banana Pro achieves 5.00% Overall on DFS Hard (10×10–13×13), compared to Veo-3's 0.00% under HumanEval.

On Sudoku (Table 6), the modality gap is stark and consistent. For 9×9 Hard—the most demanding logical reasoning setting—Nano-banana Pro achieves 30.86% Overall, while the best video model (Sora-2) achieves 7.14%, Veo-3 achieves 2.57%, and Wan-2.2 achieves 1.00%. Image models achieve 19.94–39.28% across difficulties, while video models remain in single digits. The gap persists across grid sizes: even on the simplest 4×4 Easy setting, Nano-banana achieves 66.25% Overall compared to Veo-3's 11.38%.

On ARC-AGI (Tables 9, 10), Nano-banana Pro achieves 30.54% Valid Solution on v1 and 30.36% on v2, establishing clear state-of-the-art. The best video model, Sora-2, achieves 20.18% on v1 but collapses to 1.33% on v2—a 93% relative decline that exposes memorization rather than reasoning. Veo-3 achieves 5.16% (v1) and 4.00% (v2), while Wan-2.2 and GPT-4o-image effectively fail (near 0%).

On Math (Table 20), Nano-banana Pro dominates across all five benchmarks. On GSM8K: 97.83% vs. Sora-2's 30.00%. On MATH500: 91.84% vs. Sora-2's 31.91%. On AIME 2025: 66.67% vs. Sora-2's 0.00%. On Omni-MATH: 63.06% vs. Veo-3's 3.89%. The gap widens as problem difficulty increases, with video models failing to execute multi-step mathematical derivation.

On Embodied Navigation (Section 9.4, Table 24), the pattern is task-dependent. Nano-banana achieves the highest holistic scores on Panoramic Last-Mile Navigation (74.2% vs. Veo-3's 60.0%) and 3D Real-World Navigation (79.2% vs. Wan-2.2's 24.2% and Veo-3's 22.5%). However, Veo-3 outperforms Nano-banana on Top-down View Navigation (19.5% vs. 11.1%)—the only Abstract/Embodied task where a video model leads. On SLAG, Nano-banana achieves 28.8% vs. Sora-2's 12.9% and Veo-3's 11.2%. The Embodied results reveal that video models possess genuine ego-centric spatial perception capabilities (they understand scene layouts and can model agent-environment interactions from first-person views), but these capabilities fail to translate into functional navigation when long-horizon planning, multi-view alignment, or strict constraint following is required.

The Hallucination of Competence: Video Models Produce Correct Answers Through Invalid Reasoning

The separation of process and outcome metrics across multiple tasks enables the paper to quantify what it terms the "hallucination of competence": situations where models achieve high outcome success despite near-zero process validity. This pattern is observable primarily in video models and is diagnostic of pattern matching rather than genuine reasoning.

On Math GSM8K (Table 20), Veo-3 achieves 74.00% Final Correctness (Outcome Success) but only 12.00% Process Success (Intermediate Correctness)—a 62-percentage-point gap. Sora-2 shows a smaller but still substantial gap: 64.00% Outcome vs. 38.00% Process. In contrast, Nano-banana Pro achieves near-perfect alignment: 97.83% for both metrics—when the image model produces the correct answer, the reasoning is almost always valid. This pattern holds across more difficult benchmarks. On MATH500, Veo-3 shows 52.00% Outcome vs. 20.00% Process. On AIME 2025, Veo-3 shows 11.67% Outcome vs. 3.33% Process. The gap narrows on harder problems because Outcome Success itself collapses—the model cannot guess the right answer when problems are sufficiently novel.

On Maze (Tables 4, 5), the hallucination of competence manifests as the gap between Target Achievement and Overall. Veo-3 on DFS Hard (AutoEval, Table 4): 60.00% Target Achievement vs. 51.50% Overall—an 8.5-point gap driven by Cross Wall violations. Under HumanEval (Table 5), the gap is catastrophic: 70.00% Target Achievement vs. 0.00% Overall for DFS Hard. The model reaches the red square in 70% of cases, but in 100% of those cases (per human assessment), it crossed walls to get there—the competence is entirely illusory.

On Omni-MATH category analysis (Table 22), Veo-3 shows a particularly revealing version of this pattern in Precalculus: 21.74% Outcome Success vs. 4.35% Process Success (and 0.00% Overall). The model produces the right final answer in over 1/5 of cases, but the reasoning is valid in only 1/20—and because Overall requires both, the success rate is zero. The model is "hallucinating competence" through answer retrieval without understanding the derivation.

The hallucination of competence is not universal across models or tasks. Image models, particularly Nano-banana Pro, show tight coupling between process and outcome metrics (GSM8K: 97.83% both; Omni-MATH T4: 82.76% vs. 93.10%). On Physical Commonsense (Table 35), Sora-2 achieves better alignment (76.00% Overall from 86.00% Physics Accuracy and 76.00% Motion Quality) than Veo-3 (51.02% Overall from 71.43% Physics Accuracy). The pattern is most extreme in tasks combining visual generation with symbolic constraint satisfaction—precisely the combination that video generation architectures are least equipped to handle.

Difficulty-Dependent Scaling: Robust Stability vs. Sudden Collapse

The systematic difficulty stratification across tasks enables the paper to characterize how different models degrade as problem complexity increases, revealing distinct "difficulty profiles" that distinguish generalization from memorization.

On ARC-AGI (Table 11, combined v1+v2), Nano-banana Pro exhibits what the paper terms "robust stability": Overall accuracy remains nearly flat across Easy (30.89%), Medium (30.39%), and Hard (30.23%). Grid Integrity shows only modest degradation (86.18% → 86.74% → 77.91%). This flat difficulty curve is consistent with a model that has internalized abstract transformation rules—harder instances (larger grids, more objects, more colors) do not substantially increase the difficulty of rule application once the rule is identified.

In contrast, Sora-2 shows "difficulty-sensitive collapse": Overall drops from 22.22% (Easy) to 16.33% (Medium) to 10.64% (Hard). Pattern Recognition plummets from 76.07% to 40.43%. More dramatically, the v1-to-v2 transition (Tables 9, 10) collapses Sora-2 from 20.18% to 1.33%—a 93% relative decline—while Nano-banana Pro remains stable (30.54% → 30.36%). The v2 benchmark introduces novel pattern families explicitly designed to defeat memorization; Sora-2's collapse confirms that its v1 performance was retrieval-based rather than rule-based.

On Maze difficulty scaling (Table 4), Veo-3 shows an unexpected pattern: Overall remains stable across difficulty levels (Easy 42.00%, Medium 38.69%, Hard 51.50% for DFS under AutoEval), and Maze Changed actually improves (15.50% → 0.50% → 0.00%). However, HumanEval (Table 5) reveals that this apparent stability is an AutoEval artifact: true Overall is 10.00% (Easy), 0.00% (Medium), 0.00% (Hard) for DFS, with Cross Wall rates of 70–100% across all levels. The model consistently violates physical constraints; the AutoEval stability reflects the evaluator's blindness to these violations, not genuine difficulty invariance.

On Math dataset progression (Table 20), Nano-banana Pro defies the difficulty curve: 97.83% (GSM8K) → 91.84% (MATH500) → 31.82% (AIME24) → 66.67% (AIME25) → 63.06% (Omni-MATH). The jump from 31.82% to 66.67% between AIME24 and AIME25 is notable—it suggests that problem-year variance matters more than the theoretical difficulty gradient, possibly due to training data contamination varying by exam year. Video models show a more conventional collapse: Sora-2 drops from 30.00% (GSM8K) to 31.91% (MATH500) to 4.35% (AIME24) to 0.00% (AIME25) to 0.62% (Omni-MATH).

On Omni-MATH difficulty (Table 21), Veo-3 exhibits "inverse scaling": 6.06% (T0) → 3.03% (T1) → 5.56% (T2) → 1.47% (T3) → 5.00% (T4). The U-shaped curve—with T4 (Olympiad-level) outperforming T3—is inconsistent with genuine reasoning but consistent with template memorization: the hardest problems follow the most rigid, canonical templates (standard Olympiad proof structures) that are easiest for the model to retrieve and reproduce, while intermediate-tier problems have more variable structures that resist template matching.

On Sudoku (Table 6), the difficulty scaling is monotonic but the modality gap persists. Nano-banana Pro: 4×4 Overall drops from 56.12% (Easy) to 56.75% (Medium) to 57.38% (Hard)—essentially flat with a slight increase on Hard. 9×9 Overall: 39.28% (Easy) → 33.99% (Medium) → 30.86% (Hard)—a modest, graceful decline. Veo-3: 4×4 Overall 11.38% → 9.70% → 8.71%; 9×9 Overall 3.18% → 2.77% → 2.57%. Both modalities degrade, but image models degrade gracefully from a high baseline while video models degrade to near-zero.

On Embodied Navigation (Tables 26–33), difficulty scaling interacts with task type. Panoramic Last-Mile Navigation (Table 26): Veo-3 drops from 73.33% (floor01) to 46.67% (floor02plus) under AutoEval, and from 25.00% to 28.33% under HumanEval—the human scores are so low that the difficulty effect is hard to discern. Nano-banana: 73.33% → 75.00% (AutoEval), actually improving on more complex environments. Trajectory distance shows the expected penalty for Veo-3 (66.67% Short → 53.33% Long, AutoEval; 41.67% → 11.67%, HumanEval), confirming that long-horizon control is substantially harder. For SLAG (Table 32), the holistic metric is near floor across all difficulty axes: Veo-3 11.86% (floor01) → 10.53% (floor02plus); Sora-2 10.34% → 15.52%. The difficulty stratification for SLAG reveals less about model capability differences and more about the uniformly low ceiling imposed by the trajectory alignment bottleneck.

The AutoEval-HumanEval Reliability Gap

The systematic comparison of automated VLM-based evaluation against human judgment is a core empirical contribution of MMGR, establishing that AutoEval reliability is task-dependent, metric-dependent, and systematically biased in specific directions.

On Maze (Table 5), the gap is largest. Veo-3 AutoEval Overall ranges from 38.69% (DFS Medium) to 51.50% (DFS Hard), while HumanEval Overall ranges from 0.00% to 20.00%, with four of six settings at 0.00%. The primary driver is Cross Wall: AutoEval reports 15.50–25.63% violations while HumanEval detects 70–100%. The VLM evaluator systematically misses transient wall-crossing events that occur within single frames. Action Reflection shows the inverse pattern: AutoEval reports 0.00–3.00% while HumanEval reports 40–90%, because humans detect subtle exploratory behaviors (hesitations, micro-adjustments) that the VLM fails to classify.

On Sudoku (Table 7), the gap is also severe for Veo-3 but manifests differently. HumanEval finds 0.00% Overall across all six grid-size/difficulty combinations, compared to AutoEval's 2.57–11.38%. The primary discrepancy is in Completion Accuracy: AutoEval reports 13.45–37.61% while HumanEval reports 0.00–17.50%. On 9×9 Medium, AutoEval reports 13.66% Completion Accuracy while HumanEval finds 2.50%. The VLM is more generous in recognizing digits, likely accepting ambiguous or blurry characters that human annotators reject. Constraints Violation shows the opposite pattern: AutoEval reports 39.60–44.88% violations while HumanEval reports 0.00–20.00%, suggesting the VLM may over-detect rule violations by misinterpreting visual artifacts as digit repetitions.

On ARC-AGI (Tables 16, 17), the most critical finding is the Valid Solution collapse. For v1 (60 evaluated samples), AutoEval reports 4.72% Valid Solution while HumanEval finds 0.00%. For v2 (38 samples), AutoEval 2.67% vs. HumanEval 0.00%. Not a single Veo-3 generated video achieved exact match to the ground-truth solution under human judgment, despite the VLM occasionally crediting outputs as correct. Grid Integrity is systematically underestimated by humans vs. AutoEval (v1: 23.33% Human vs. 35.70% Auto)—humans are stricter about grid misalignment. Color Accuracy is systematically underestimated by AutoEval vs. humans (v1: 7.61% Auto vs. 23.33% Human)—the VLM appears overly sensitive to minor RGB deviations.

On Embodied Navigation (Tables 27, 29, 31, 33), the gap magnitude varies dramatically by task. Panoramic Last-Mile Navigation (Table 27): AutoEval Overall 73.33% (floor01) vs. HumanEval 25.00%. Top-down View Navigation (Table 29): AutoEval 37.14% (floor01) vs. HumanEval 10.34%. 3D Real-World Navigation (Table 31): AutoEval 25.00% (floor01) vs. HumanEval 3.33%. SLAG (Table 33): AutoEval 11.86% (floor01) vs. HumanEval 0.00%. Across all navigation tasks, Scene Consistency and Physics Validness are rated significantly lower by humans, indicating that AutoEval fails to detect environmental drift and physical law violations that humans immediately recognize.

On Physical Commonsense (Table 38), the pattern reverses compared to other tasks: HumanEval rates Veo-3 higher than AutoEval. Overall: 80.00% (Human) vs. 51.02% (AutoEval)—a 29-point gap in the opposite direction from Maze or ARC-AGI. Physics Accuracy: 84.44% vs. 71.43%. Motion Quality: 82.22% vs. 61.22%. The VLM evaluator applies overly strict criteria for temporal coherence and physical accuracy in domain-general physical scenarios, penalizing minor artifacts that human observers find acceptable.

Physical Commonsense: Video Models Show Genuine, If Imperfect, Physical Understanding

Physical Commonsense (Tables 35–40) stands apart from the other MMGR domains in yielding positive results for video models. Sora-2 achieves 70.00% Overall, Veo-3 achieves 51.02% (by AutoEval), and human evaluation further elevates Veo-3 to 80.00% Overall (Table 38). These scores, while not ceiling, substantially exceed video model performance on Abstract Reasoning tasks (<10% on ARC-AGI, <7% on Sudoku).

The fine-grained breakdown by scenario type (Table 35) reveals that Sports Scenarios are consistently easier than Physical Concepts across all models: Veo-3 improves from 41.67% to 60.00%; Sora-2 shows 76.00% to 64.00% (a slight reversal). The sport-specific breakdown (Table 36) reveals that Swimming and Skiing are easier than Ballet and Diving—consistent with the hypothesis that learned biomechanical motion patterns from training data drive performance, and that motions with complex rotational dynamics (Ballet pirouettes, Diving flips) pose greater challenges.

The states-of-matter breakdown (Table 37) reveals that Solid-Solid interactions are most difficult: Veo-3 achieves 0.00% Overall on Solid-Solid vs. 75.00% on Solid-Fluid and 50.00% on Fluid-Fluid (AutoEval). Under HumanEval (Table 40), Veo-3's Solid-Solid rises to 66.7%, suggesting that the AutoEval is overly strict on collision detection. Sora-2 achieves 100% on Solid-Solid and Fluid-Fluid but only 75% on Solid-Fluid, indicating model-specific weaknesses in different physical regimes.

The Embodied Navigation results (Tables 24–33) reveal a consistent dissociation between models' ability to generate convincing local motion and their ability to execute goal-directed navigation. Veo-3 achieves 93.3% Physical Understanding on Panoramic Last-Mile Navigation (Table 24), with Object Semantic Score of 87.50% and Agent Consistency Score of 92.50% under HumanEval (Table 25). The model understands that pillars are obstacles and maintains smooth, continuous agent motion.

However, the holistic success rate collapses to 26.67% under HumanEval (Table 25) because Instruction Following metrics fail: Destination Integrity is only 60.00% and Scene Consistency is 64.17%. The model generates physically plausible motion toward the wrong destination or in an unstable environment.

On SLAG (Table 25), the component scores reveal where video models fundamentally break: Scene Consistency is remarkably high at 93.90% (the model maintains visual coherence between the 3D and 2D panels), but Trajectory Alignment is near zero at 6.10%. The model can generate temporally consistent frames but cannot align the 2D trajectory projection with the 3D motion—the cross-view spatial reasoning required for SLAG exceeds current capability.

The full AutoEval results (Tables 26, 28, 30, 32) provide detailed breakdowns across the 24 hard-level configurations. For Panoramic Last-Mile Navigation (Table 26), destination specification matters: Veo-3 achieves 70.00% Overall with color marks vs. 50.00% with location descriptions—grounding text to spatial locations is harder than following visual markers. For Top-down View Navigation (Table 28), the same gap is 38.89% vs. 11.43%, confirming that text-to-space grounding is a consistent bottleneck. View fidelity shows positive scaling for Veo-3 (T.V.R.-W.Nav. quality03: 20.83%, quality04: 29.17%, quality05: 26.09%), indicating that clearer visual cues improve navigation generation.

Ablation Studies and Robustness Checks

MMGR is a benchmark rather than a method, so traditional ablations (varying model components or training hyperparameters) are not applicable. Instead, the paper's robustness is established through several forms of internal validation:

Task stratification across difficulty levels: Every task provides difficulty-stratified results, testing whether performance patterns are robust to problem complexity changes. The consistent finding that image models show gracefully degrading difficulty curves while video models collapse abruptly (especially on ARC-AGI and Math) is replicated across multiple tasks with independent difficulty definitions, reducing the likelihood that the pattern is an artifact of a particular difficulty metric.
Multi-generator validation for Maze: The Maze task uses two distinct generation algorithms (DFS and Wilson's) producing topologically diverse mazes (Section 5.2). Table 4 shows that the performance hierarchy across models and the pattern of AutoEval-HumanEval discrepancy is consistent across both generators. Veo-3 DFS Easy AutoEval Overall: 42.00%; Wilson's Easy AutoEval Overall: 46.50%. The Cross Wall discrepancy is present for both generators (DFS AutoEval 25.50% vs. HumanEval 80.00%; Wilson's AutoEval 21.50% vs. HumanEval 70.00%).
Multi-version robustness for ARC-AGI: The inclusion of both v1 (381 cases) and v2 (75 harder cases) tests whether model performance patterns generalize across dataset versions. Nano-banana Pro's stability (30.54% → 30.36%) vs. Sora-2's collapse (20.18% → 1.33%) is a robustness check that distinguishes genuine rule-based generalization from memorization. The v2 set was explicitly designed to contain novel pattern families not present in v1 (Section 7.1), making it a strong test of out-of-distribution generalization.
Multi-benchmark validation for Math: The Math task spans five independent benchmarks (GSM8K, MATH500, AIME24, AIME25, Omni-MATH) with 327 total problems (Table 18). The consistent finding that image models dominate video models—and that video models exhibit the hallucination of competence (Outcome > Process)—replicates across all five benchmarks, establishing that the pattern is not an artifact of a particular math dataset.
Modality comparison as architecture ablation: The consistent comparison of video and image models on identical tasks (Maze, Sudoku, ARC-AGI, Math, Embodied Navigation) serves as an informal ablation of the temporal generation dimension. The finding that image models outperform video models on abstract reasoning tasks isolates the temporal "tax" as a causal factor—the additional consistency burden of frame-by-frame generation harms logical coherence.
Negative result: Replicability of the hallucination of competence pattern: The pattern of high Outcome Success but low Process Success appears in Math (Veo-3 GSM8K: 74% vs. 12%), Maze (Veo-3 DFS Hard AutoEval: 60% Target Achievement vs. 51.50% Overall), and Omni-MATH category analysis (Veo-3 Precalculus: 21.74% Outcome vs. 0.00% Overall). The replication of this pattern across tasks with different reasoning demands (mathematical derivation, spatial path planning, domain-specific problem solving) suggests it is a fundamental property of current video generation architectures rather than a task-specific artifact.
Negative result: Veo-3's 0% ARC-AGI Valid Solution under HumanEval: The finding that Veo-3 achieves 0.00% exact-match accuracy across 98 evaluated ARC-AGI cases under human judgment (Tables 16, 17), despite AutoEval reporting 4.72% (v1) and 2.67% (v2), is an important negative result. It establishes that the automated metrics overestimate abstract reasoning capability and that true generalization to novel visual transformations is essentially absent in current video models—even when they occasionally produce outputs that pass automated checks.

Critical Assessment

The experiments in MMGR support the paper's central claims to varying degrees, with the strongest evidence concentrated on the decoupling of visual quality from reasoning, the hallucination of competence phenomenon, and the unreliability of VLM-based evaluation for certain metrics. Other claims—particularly the "temporal tax" interpretation and the generalization to training data explanations—receive only partial or indirect support.

Claim 1: Current generative models exhibit a critical gap between perceptual quality and reasoning capability, with visual quality metrics failing to detect reasoning failures. This claim is strongly supported across multiple domains. The Physical Commonsense results (Tables 35, 38) provide the cleanest evidence: Wan-2.2 achieves 84–96% Visual Realism but only 24.00% Overall because its Physics Accuracy and Prompt Adherence collapse. The Maze results demonstrate the same decoupling in the spatial domain: Veo-3 achieves 60+% Target Achievement while producing videos that cross walls in 70–100% of cases (Table 5 HumanEval). The Math results demonstrate it in the logical domain: Veo-3 achieves 74% Final Correctness with only 12% Process Success. The paper has established this decoupling across physical, spatial, and logical reasoning tasks, which is the most robust and important finding.

However, the claim is demonstrated for a specific set of models and tasks. The paper does not establish whether the decoupling is specific to video generation architectures or would also appear in other generative modalities (audio, code, 3D). The benchmark is limited to visual tasks, so the "multi-modal" framing in the title should be understood as multi-modal within the visual domain (video + image) rather than across fundamentally different modalities. This is a scope limitation acknowledged by the task design but not explicitly discussed as a constraint on the claims.

Claim 2: Image models consistently outperform video models on abstract reasoning tasks, revealing a "temporal tax" on reasoning. This claim is empirically well-supported by the cross-modal comparisons (Tables 4, 6, 9, 10, 20, 24) but the causal interpretation—that temporal generation causes the performance deficit—is not directly tested. The paper observes a correlation (video < image on abstract reasoning) and attributes it to the temporal dimension introducing consistency burdens. However, alternative explanations are not ruled out:

Training data differences: Video models and image models are trained on different data distributions. Video training data may contain proportionally less structured symbolic content (logic puzzles, math derivations, abstract spatial transformations) than image training data, which could explain the gap without invoking temporal processing as the causal mechanism. The paper acknowledges this possibility in its training data imbalance diagnosis (Section 15) but does not isolate it from the architectural explanation.
Scale differences: The models compared are not FLOPs-matched or parameter-matched. Nano-banana Pro may simply be a more capable model in absolute terms, and its advantage on abstract reasoning would persist even if Veo-3 were somehow given a static-image-generation head. The paper does not control for model scale, training compute, or data volume in the cross-modal comparison.
Task format confound: The image generation format may be inherently more suitable for certain abstract reasoning tasks. ARC-AGI, for example, was designed for discrete grid completions; asking a video model to animate its solution while preserving static demonstrations introduces an additional requirement (static context preservation) that is not part of the reasoning task per se. The video model's failure on ARC-AGI may reflect a failure of context maintenance rather than a failure of abstract reasoning—the model might be able to solve the puzzle but unable to keep the demonstration examples stable while doing so. The paper's case studies (Figures 11a, 11b) support this interpretation.

The "temporal tax" concept is a compelling interpretive framework that fits the data, but the paper does not run the experiments that would directly test it: for instance, giving video models a static image generation mode for reasoning tasks (to see if their performance improves when temporal processing is disabled), or training a single architecture on identical data in both video and image modes and comparing reasoning performance. These are reasonable directions for future work, but the current evidence supports a weaker claim: video models tested in their standard video generation mode underperform image models on abstract reasoning tasks; the reasons for this gap are likely multi-factorial.

Claim 3: The hallucination of competence—where models produce correct outcomes through invalid reasoning—is a widespread failure mode in video generation. This claim is well-supported by the Math results (Table 20) and to a lesser extent the Maze results (Table 4), but its generality is limited by the availability of process-level metrics in the benchmark. The hallucination of competence is directly measurable only in the Math task (where Intermediate Correctness vs. Final Correctness provides an explicit process-outcome decomposition) and indirectly inferable in Maze (where Target Achievement vs. Overall captures a similar pattern). For Sudoku, ARC-AGI, Embodied Navigation, and Physical Commonsense, the paper's metrics do not cleanly separate process from outcome, so the hallucination of competence cannot be quantified in those domains.

This is not a flaw in the benchmark design—many tasks do not have a natural process-outcome decomposition—but it limits the scope of the claim. The paper has convincingly demonstrated the phenomenon in mathematical reasoning; whether it extends to physical reasoning (e.g., video models generating physically correct outcomes through physically impossible intermediate states) is plausible but not directly tested. The Maze results hint at this (reaching the goal through wall-crossing), but the Maze task is too simple to draw strong conclusions about physical reasoning.

Claim 4: VLM-based automated evaluation is systematically unreliable for reasoning assessment and must be calibrated against human judgment. This claim is the most robustly supported finding in the paper because it emerges from the most extensive empirical comparison (Tables 5, 7, 16, 17, 25, 27, 29, 31, 33, 38, 39, 40). The evidence spans all three domains, multiple tasks, and both directions of error (AutoEval overestimation and underestimation). The specific mechanisms identified—temporal resolution limitations causing missed transient violations, color sensitivity causing overly strict Color Accuracy judgments, and difficulty recognizing subtle exploratory behaviors—are well-documented through both quantitative discrepancy tables and qualitative case studies.

However, several limitations temper this claim:

Single evaluator model: All AutoEval uses Gemini-2.5-Pro. The paper cannot distinguish between limitations specific to this Gemini version and limitations of VLM-based evaluation in general. A comparison across multiple VLM evaluators (e.g., GPT-4V, Claude 3.5, Gemini variants) would establish whether the reliability problems are evaluator-specific or fundamental to current VLM capabilities.
Human evaluation limited to one model (Veo-3) for most tasks: The HumanEval calibration establishes that AutoEval is unreliable for Veo-3, but it does not establish whether the bias is consistent across models. It is possible that AutoEval is more accurate for some models than others (e.g., image models with cleaner outputs might be evaluated more reliably than video models with motion artifacts). HumanEval for Sora-2 or Nano-banana Pro would be needed to establish evaluator-model interaction effects.
Small human evaluation samples: The 98 ARC-AGI samples, 45 Physical Commonsense samples, and the various navigation subsets (120 total per task but split across many conditions) provide reliable point estimates for the specific settings tested, but the confidence intervals around the discrepancy magnitudes are not reported. A 0.00% Valid Solution rate from 98 samples could still represent a true rate of up to ~3% at a 95% confidence interval (using the rule of three). The qualitative conclusion (AutoEval overestimates) is robust; the precise discrepancy magnitude is less so.
Human evaluator demographics: The paper reports 6 annotators with "bachelor education background" but does not specify their domain expertise, native language, or whether they received task-specific training beyond the 4-hour instruction session. For tasks requiring mathematical or physical reasoning judgment (evaluating whether a Sudoku solution is logically valid, whether a ballet pirouette obeys angular momentum conservation), domain expertise may substantially affect evaluation accuracy. The human evaluation scores should be understood as "educated layperson judgments" rather than expert-validated ground truth for the most technically demanding metrics.

Claim 5: Systematic difficulty stratification reveals distinct reasoning profiles that distinguish generalization from memorization. This claim is supported for ARC-AGI (where Nano-banana Pro's flat difficulty curve contrasts with Sora-2's collapse) and for Math (where the Omni-MATH U-shaped curve suggests template memorization). However, the claim is not tested for other tasks in the benchmark. Maze, Sudoku, and Embodied Navigation report difficulty-stratified results, but the paper does not analyze whether these stratifications reveal generalization-vs.-memorization profiles. This is an analytical gap rather than an experimental gap—the data exists in Tables 4, 6, 26–33, but the generalization interpretation is not systematically applied.

Additionally, the ARC-AGI v1 → v2 robustness test is confounded by the difficulty distribution shift: v2 contains only 1 Easy Match case vs. 102 in v1 (Table 8). Sora-2's 93% decline could partly reflect that v2 is simply harder (more Medium/Hard, more Mismatch) rather than reflecting a failure of generalization to novel patterns. The paper's difficulty-stratified results (v1+v2 combined, Table 11) partially address this by reporting Easy/Medium/Hard breakdowns, but the v1-v2 comparison itself conflates difficulty shift with novelty shift. A version-matched difficulty comparison (v1-only or v2-only stratified) would isolate the novelty effect.

Missing experiments that would have strengthened the paper:

Fine-tuning or few-shot adaptation: The paper evaluates zero-shot performance only. For embodied navigation and physical commonsense, providing a small number of task-specific demonstrations could substantially change the performance landscape. The failure modes identified (context drift, trajectory misalignment) might be addressable with in-context learning, but the paper's zero-shot design cannot distinguish between fundamental capability gaps and adaptation gaps.
Scaling analysis within model families: The paper evaluates one variant of each model. Comparing multiple size variants of the same architecture (e.g., Nano-banana vs. Nano-banana Pro is one data point, but a more systematic comparison of, say, small/medium/large video diffusion models) would reveal whether reasoning capability scales with model size or plateaus. This is particularly important for the "temporal tax" claim: does the tax shrink or grow with scale?
Multiple evaluator models: As noted above, comparing AutoEval across Gemini-2.5-Pro, GPT-4V, and other VLMs would establish whether the evaluation reliability problems are evaluator-specific or fundamental.
Controlled data ablation: The paper attributes model failures partly to training data imbalance (Section 15: "a severe scarcity of structured symbolic reasoning data"), but this is a post-hoc explanation. Training models on datasets with systematically varied proportions of symbolic reasoning data and measuring MMGR performance would provide causal evidence for this claim.
Latency and computational cost analysis: The paper does not report inference time, GPU-hours, or financial cost for generating the benchmark results. For a benchmark intended to guide practical model evaluation, understanding the computational cost of comprehensive MMGR evaluation (vs. simpler metrics like FVD) would inform adoption.

Genuine weaknesses in the experimental design:

Single benchmark: All results are from MMGR itself. There is no external validation against established reasoning benchmarks (e.g., the original ARC-AGI leaderboard, MATH benchmark text-only results for the base models that some of these image/video models might have been built on). This is inherent in MMGR being a new benchmark, but it means there is no independent verification that the task adaptations (e.g., making ARC-AGI a video generation task) preserve the difficulty and construct validity of the original benchmarks.
No statistical significance testing: The paper reports point estimates without confidence intervals or significance tests. This is standard in benchmark papers but limits the interpretability of small performance differences. For instance, Veo-3's 51.02% vs. Sora-2's 70.00% on Physical Commonsense—with only 50 total samples—has substantial uncertainty that is not quantified.
Potential prompt sensitivity: The evaluation prompts for Gemini-2.5-Pro are task-specific and carefully designed, but the paper does not ablate prompt variations to test whether small changes in evaluation prompt wording significantly affect AutoEval scores. Given the paper's own finding that AutoEval is unreliable, the robustness of the evaluation prompts themselves is an unexamined variable.
Training data contamination: The paper does not address whether the evaluated models might have been trained on the specific benchmark instances or their close variants. For public benchmarks like ARC-AGI, GSM8K, and MATH, training data contamination is a known concern that could inflate image model performance if these models were trained on text-image pairs that include the benchmark problems. Nano-banana Pro's 97.83% on GSM8K could partly reflect memorization rather than reasoning—a concern the paper does not discuss, even though the ARC-AGI v1-to-v2 collapse for Sora-2 demonstrates that memorization is a real factor.

In summary, the MMGR experiments provide strong evidence for the decoupling of visual quality from reasoning, the hallucination of competence in video math reasoning, and the systematic unreliability of VLM-based evaluation. The evidence for the "temporal tax" as a causal mechanism and for training data imbalance as a root cause is suggestive but not directly tested. The benchmark's scope—single model per architecture, single evaluator model for AutoEval, zero-shot evaluation only—limits the generality of some claims, but within these scope constraints, the experimental methodology is rigorous and the key findings are well-supported by the data.

6. Limitations and Trade-offs

The Difficulty Estimation and Hard-Level Control Overhead Is Not Accounted for in Practical Deployment

The Assumption or Constraint. MMGR's central analytical contribution—the systematic difficulty stratification that enables diagnostic reasoning profiles—depends on extensive pre-computation of difficulty labels through programmatic generation (grid sizes, clue counts, trajectory distances) and quantitative feature scoring (ARC-AGI's five-feature system in Section 7.1.1). For a practitioner seeking to use MMGR-style evaluation on their own task distribution, building equivalent hard-level control requires either access to the benchmark's specific generation infrastructure (Maze generators from maze-dataset (Ivanitskiy et al., 2023), Sudoku generation from sudoku-bench (Seely et al., 2025), Matterport3D/HM3D scene access for Embodied Navigation) or the design and validation of a new feature-based difficulty scoring system.

The paper does not frame this as a limitation, because MMGR is presented as a fixed benchmark rather than a deployable evaluation methodology. Yet the diagnostic value of the benchmark—its ability to distinguish generalization from memorization through difficulty-dependent scaling curves—depends entirely on the quality and granularity of difficulty stratification. This is the evaluation equivalent of the exploration-exploitation tradeoff: the information gained from fine-grained difficulty analysis requires substantial upfront investment in task design and labeling.

The Consequence. A practitioner attempting to build a reasoning evaluation for their own domain would face the following concrete obstacles:

Task Design Complexity. Each of MMGR's ten tasks required domain-specific engineering. The Maze task (Section 5.2) uses two generation algorithms (DFS, Wilson's), 10 grid sizes, and 4 start-goal configurations, yielding 240 uniquely generated mazes with verified single-solution paths. The Embodied Navigation tasks (Section 9.2) draw from 3 separate environment sources (Matterport3D, HM3D, Habitat) and require human annotation of view fidelity scores (quality03–quality05) and destination descriptions with disambiguation metadata. The ARC-AGI task (Section 7.1.1) required designing a quantitative difficulty scoring system from five grid features, calibrated across 456 cases. None of this infrastructure is packaged as reusable tooling; a practitioner would need to design equivalent systems de novo.
Difficulty Calibration Requires Ground-Truth Validation. The paper's difficulty stratifications are validated by construction (Maze grid size objectively determines search complexity; Sudoku clue sparsity objectively determines constraint-satisfaction difficulty). For novel domains where difficulty is less objectively quantifiable—open-ended dialogue, creative writing, design tasks—the mapping between task parameters and reasoning difficulty is non-obvious. MMGR provides no methodology for difficulty calibration beyond what is baked into its specific tasks.
Cost of Maintaining Difficulty Granularity at Scale. The 24-configuration Embodied Navigation design (2 × 3 × 2 × 2, Section 9.2.4) produces 120 samples per subtask. Each configuration slot requires 5 samples (for 5 models × 5 generations each = 25 generations per slot), yielding thousands of evaluated videos. Scaling this design to additional environments, model variants, or task settings incurs linear cost in both generation and evaluation compute. The paper does not estimate the total compute budget for running the full MMGR benchmark; based on the reported model APIs (Veo-3, Sora-2, GPT-4o-image) and generation counts (5 samples × 1,853 prompts = 9,265 generations per model for full coverage), the inference cost alone—excluding evaluation compute—is substantial and would represent a barrier for academic labs or smaller organizations.

What Evidence Exists in the Paper. Section 9.2 documents the four-axis, 24-configuration design for Embodied Navigation. Section 7.1.1 documents the quantitative difficulty scoring system for ARC-AGI. Section 4.1 summarizes the benchmark statistics. The paper's systematic difficulty stratification is one of its key innovations (Section 4, Innovation 5), and the overhead required to achieve it is inherent in the benchmark design. The paper is transparent about the task generation infrastructure (citing maze-dataset, sudoku-bench, Matterport3D, HM3D, Habitat, VideoPhy) but does not estimate the person-hours or compute cost required to assemble the full benchmark.

Mitigation Status. Not addressed. The paper does not discuss the cost of benchmark construction or the barriers to adapting MMGR-style evaluation to new domains. The authors do not release difficulty-estimation tooling (beyond the specific benchmark instances) or provide guidance on how practitioners might build difficulty-stratified evaluations for their own tasks. This is understandable for a benchmark paper—MMGR's contribution is the benchmark itself, not a general methodology for building benchmarks—but it limits the practical impact of the difficulty-stratification insight beyond the specific tasks tested.

The Assumption or Constraint. The paper's most prominent empirical claim—that image models consistently outperform video models on abstract reasoning tasks, revealing a "temporal tax" (Sections 4, 5, 15)—rests on a comparison between specific instantiations of image and video generation architectures that differ along multiple unmeasured dimensions. The models compared are not controlled for parameter count, training compute, training data composition, or training data volume.

The paper acknowledges this indirectly by referring to models as "representative" (Section 4.4) rather than matched, but never directly addresses the confound. The models span four different organizations (Google, OpenAI, Wan team, Qwen team), different training paradigms (proprietary vs. open-weights), and different release timelines (2024–2025). Nano-banana Pro, which dominates the abstract reasoning tasks, is a Google model from the same lab that produced Veo-3—yet the paper does not compare parameter counts, training FLOPs, or training data overlap between these two models to isolate whether the performance gap is truly a "temporal tax" or simply reflects that Google invested more compute in their image model than their video model (or trained the image model on more symbolically-structured data).

The Consequence. The "temporal tax" interpretation—that video generation actively harms reasoning because temporal processing competes with logical consistency—is not uniquely supported by the data. At least three alternative explanations are consistent with the observed results:

Scale Confound. Nano-banana Pro may simply be a larger or better-trained model than Veo-3 in absolute terms. If so, its advantage on abstract reasoning would reflect general capability rather than modality. The paper provides no parameter counts or training FLOPs to rule this out.
Training Data Confound. Image models may be trained on datasets with proportionally more structured symbolic content—textbooks, diagrams, puzzles, math problems—while video models are trained predominantly on naturalistic video (sports, daily activities, nature scenes), which would explain the domain asymmetry (video models do well on Physical Commonsense/Sports, poorly on Abstract Reasoning) without invoking temporal processing as a mechanism.
Task Format Confound. The adaptation of reasoning benchmarks to video format introduces additional requirements beyond reasoning. ARC-AGI for video models requires maintaining static demonstration examples while generating the solution (Section 7.3, Figures 11a-11b: "context drift"). The video model might be able to solve the puzzle in principle but fails because it cannot preserve the problem definition across frames—a failure of static context maintenance, not of abstract reasoning per se. The paper's case studies support this interpretation (Veo-3's ARC-AGI failures are predominantly context-destruction failures, not rule-application failures).

What Evidence Exists in the Paper. The cross-modal performance gaps are documented extensively: ARC-AGI (Tables 9, 10: Nano-banana Pro 30.54% vs. Sora-2 20.18% v1; 30.36% vs. 1.33% v2), Math (Table 20: Nano-banana Pro 97.83% vs. Sora-2 30.00% GSM8K), Sudoku (Table 6: Nano-banana Pro 30.86% vs. Veo-3 2.57% 9×9 Hard). The paper does not report model sizes, training compute, or training data statistics for any evaluated model. The "temporal tax" concept is introduced in Section 4 (Innovation 2) and Section 15 (Conclusion) as an interpretation of the observed gap, but no experiment isolates temporal processing from other confounds—there is no comparison of a single architecture trained identically in video and image modes, or a video model evaluated in a "static mode" where it generates only a single frame.

Mitigation Status. Partially addressed through converging evidence. While no direct causal isolation exists, the paper provides several pieces of evidence that strengthen the temporal tax interpretation beyond mere correlation:

Within-model evidence for temporal drift. The Sudoku case study (Section 6.3, Figure 8) and ARC-AGI case studies (Section 7.3, Figures 11a-11b) provide qualitative evidence that video models actively corrupt problem definitions across frames—a failure mode specific to temporal generation that image models avoid by construction. This is not conclusive (the model might have corrupted the problem even in a single frame if it had poor understanding), but it is consistent with the temporal tax hypothesis.
Cross-task asymmetry. Video models perform competitively on Physical Commonsense (Sora-2: 70.00% Overall, Table 35) and some Embodied Navigation subtasks (Top-down View: Veo-3 19.5% vs. Nano-banana 11.1%, Table 24), where temporal processing is genuinely beneficial for modeling physical dynamics. The temporal tax appears specifically on symbolic/abstract tasks, precisely where temporal processing provides no structural advantage. This task-dependent pattern is harder to explain with a simple scale confound (why would a weaker model perform better on physical tasks but worse on abstract ones?) than with the modality-specific interpretation.
The hallucination of competence pattern is video-specific. The Outcome-Process gap (Section 8.3.1: Veo-3 GSM8K 74% Outcome vs. 12% Process) does not appear in image models, which show tight coupling between process and outcome metrics (Nano-banana Pro GSM8K: 97.83% both, Table 20). This is consistent with video generation introducing "visual morphing" artifacts that produce correct outputs through invalid visual transitions—a mechanism unique to temporal generation.

The paper acknowledges the limitation implicitly in Section 15, where it identifies three "fundamental bottlenecks" including "a severe scarcity of structured symbolic reasoning data compared to naturalistic physical data"—this acknowledges that training data, not just architecture, may drive the performance patterns. But the paper does not separate these causal factors empirically.

VLM-Based Evaluation Unreliability Undermines the Benchmark's Core Measurement Mechanism

The Assumption or Constraint. MMGR's evaluation protocol relies on Gemini-2.5-Pro as a unified automated evaluator across all tasks, metrics, and models (Section 4.3). The paper's own human evaluation results demonstrate that this evaluator is systematically unreliable for specific metric types, with error magnitudes up to 3–5× relative to human judgment, and error directions that vary by task and metric. This creates a fundamental tension: the benchmark's diagnostic value depends on accurate metric measurement, but the measurement instrument is known to be biased in task-dependent and metric-dependent ways.

The paper's response to this tension is to treat human evaluation as a calibration mechanism, presenting AutoEval and HumanEval side-by-side where available (Tables 5, 7, 16, 17, 25, 27, 29, 31, 33, 38, 39, 40). However, human evaluation is conducted only for a subset of tasks (Maze, Sudoku, ARC-AGI, Embodied Navigation, Physical Commonsense), only for one model (Veo-3 in most cases), and only on a fraction of the total evaluation samples (e.g., 98 ARC-AGI videos from 456 total cases; 45 Physical Commonsense videos from 50 total; specific per-task sample counts in the human evaluation tables).

The Consequence. The benchmark's headline numbers—the model rankings, the performance gaps, the difficulty-dependent scaling curves—are only as reliable as the evaluation instrument. For tasks and metrics where AutoEval-HumanEval calibration has been performed, the reader can adjust expectations (e.g., treating AutoEval Maze Overall as substantially overestimated). For tasks and metrics where calibration has not been performed (the majority of model × task combinations), the error magnitude and direction are unknown. Specifically:

Uncalibrated model comparisons. HumanEval calibrates AutoEval for Veo-3, but the paper then compares Veo-3 against Sora-2, Nano-banana Pro, and other models using AutoEval only. If AutoEval's bias is model-dependent—for instance, if it evaluates Sora-2's outputs more accurately than Veo-3's because Sora-2 produces videos with different motion characteristics—then the model rankings from AutoEval could differ from true rankings. The paper provides no evidence on whether AutoEval bias is model-invariant.
Uncalibrated task comparisons. HumanEval has been performed on Maze, Sudoku, ARC-AGI, Embodied Navigation, and Physical Commonsense, but not on Math (beyond the implicit calibration that process-outcome metrics are computed against ground-truth answers, which is objective). The Math hallucination-of-competence finding (Section 8.3.1) depends on Intermediate Correctness being accurately judged by Gemini-2.5-Pro. If the VLM evaluator is unreliable for assessing step-by-step mathematical reasoning—and the Sudoku and Maze human evaluations suggest it can be unreliable for process-level reasoning assessment—then the Math process scores may be biased in unknown directions.
Uncalibrated difficulty-stratified results. The fine-grained difficulty breakdowns (ARC-AGI Match/Mismatch × Easy/Medium/Hard, Omni-MATH T0–T4 × 8 categories, Embodied Navigation 24 hard-level configurations) are all AutoEval-only except where specifically noted. If AutoEval error varies with difficulty (e.g., the VLM misses more wall-crossings on Hard mazes with faster agent movement), then the difficulty-dependent scaling curves conflate model capability changes with evaluator reliability changes.

What Evidence Exists in the Paper. The human evaluation tables (5, 7, 16, 17, 25, 27, 29, 31, 33, 38, 39, 40) document the calibration results in detail. Key findings about AutoEval unreliability include:

Maze (Table 5): AutoEval Overalls of 38.69–51.50% vs. HumanEval of 0.00–20.00%. Cross Wall: AutoEval 15.50–25.63% vs. Human 70–100%. Action Reflection: AutoEval 0.00–3.00% vs. Human 40–90%.
Sudoku (Table 7): HumanEval Overall 0.00% across all six conditions vs. AutoEval 2.57–11.38%. Completion Accuracy: AutoEval 13.45–37.61% vs. Human 0.00–17.50%.
ARC-AGI (Tables 16, 17): HumanEval Valid Solution 0.00% across all 98 cases vs. AutoEval 4.72% (v1) and 2.67% (v2). Color Accuracy: AutoEval 7.61% vs. Human 23.33% (v1).
Panoramic Last-Mile Navigation (Table 27): AutoEval Overall 73.33% (floor01) vs. Human 25.00%. Physics Validness: AutoEval 81.67% vs. Human 25.00%.
Physical Commonsense (Table 38): Reverse pattern: AutoEval Overall 51.02% vs. Human 80.00%. Human evaluators are more lenient on physical commonsense videos.

The paper demonstrates conclusively that AutoEval bias is large (factor of 2–5×), bidirectional (overestimation on some metrics/tasks, underestimation on others), and metric-dependent. The vulnerability of uncalibrated results to these biases is a direct consequence of these documented facts.

Mitigation Status. Partially addressed through extensive human evaluation, but the calibration coverage is incomplete. The paper conducts human evaluation for Veo-3 on a subset of tasks and samples, establishing the existence and magnitude of AutoEval bias. The paper presents AutoEval and HumanEval side-by-side where available, allowing readers to apply their own calibration. However, the following gaps remain:

No human evaluation for non-Veo video models. Sora-2's and Wan-2.2's results are AutoEval-only. If AutoEval bias is model-dependent, their true performance may differ systematically from the reported numbers.
No human evaluation for image models. Nano-banana, Nano-banana Pro, GPT-4o-image, and Qwen-image results are AutoEval-only. The paper's human evaluation focuses exclusively on Veo-3 (a video model), but the cross-modal comparison is the paper's central empirical contribution. If AutoEval evaluates image model outputs differently than video model outputs (e.g., being more reliable for static images than for videos), the image-vs-video performance gap could be substantially different from what AutoEval reports.
Human evaluation samples are modest. For tasks with many difficulty strata, the human evaluation sample per stratum is small. For ARC-AGI, 98 samples across v1 Match, v1 Mismatch, v2 Match, v2 Mismatch leaves limited per-stratum precision (Tables 16, 17 report per-subset results but with wide implicit confidence intervals). For Embodied Navigation, 120 total samples per subtask split across 24 configurations yields ~5 samples per configuration, which is insufficient for precise per-configuration calibration.
The paper does not propose a corrected metric. Given the documented biases, a natural response would be to produce bias-corrected AutoEval scores (e.g., scaling down Maze Overalls by the observed AutoEval/HumanEval ratio) or to report uncertainty intervals around AutoEval scores reflecting the calibration uncertainty. The paper does neither; AutoEval and HumanEval are reported separately with no attempt at integration.

Section 15 acknowledges the evaluation reliability problem as part of the broader conclusion that "VLM-based auto-evaluation tends to overestimate model performance" and calls for future work on improved evaluation, but within the paper's scope, the AutoEval unreliability means that many of the reported numbers should be treated as preliminary estimates subject to potential substantial revision if more extensive human calibration were performed.

The Benchmark Does Not Evaluate Whether Reasoning Failures Are Fixable Through Inference-Time Techniques

The Assumption or Constraint. MMGR evaluates models in a strict zero-shot setting: default API parameters, no task-specific prompting beyond the standardized generation prompt, no few-shot examples (except where inherent to the task design, as in ARC-AGI's demonstration examples), no chain-of-thought reasoning, and no test-time compute scaling (Section 4.2). This design choice is justified for establishing a clean baseline, but it systematically excludes a class of techniques—inference-time reasoning improvements—that are central to contemporary LLM evaluation and deployment.

The assumption implicit in this design is that zero-shot generation is the appropriate unit of analysis for assessing whether models "possess" reasoning capabilities. This assumption is debatable: a model that fails to solve a Sudoku puzzle in a single forward pass but succeeds when allowed to self-correct, backtrack, or maintain an explicit scratchpad might still be considered to "possess reasoning capability" in a practically meaningful sense. The literature on test-time compute scaling (discussed extensively in the reference example paper provided with these instructions) has demonstrated that inference-time strategies can dramatically improve reasoning performance—sometimes by factors of 4× or more. MMGR provides no evidence on whether the reasoning failures it documents are fundamental capability gaps or gaps that could be closed with better test-time strategies.

The Consequence. The benchmark's negative conclusions ("models fail catastrophically on Abstract Reasoning"; "models cannot execute genuine multi-step deduction"; "the temporal tax on reasoning") may overstate the severity of reasoning deficits if inference-time techniques can substantially improve performance. Conversely, if test-time compute does not help, the conclusions would be strengthened. The paper provides no evidence either way.

Concretely, several of the documented failure modes might be addressable through inference-time techniques:

Context drift in ARC-AGI (Section 7.3). Video models corrupt static demonstration examples across frames. A test-time strategy that explicitly separates "static context preservation" from "dynamic solution generation"—for instance, by generating the solution separately and compositing it with frozen demonstration frames—could eliminate this failure mode without requiring improved reasoning capability.
The hallucination of competence in Math (Section 8.3.1). Video models produce correct final answers through invalid intermediate steps. A test-time strategy that uses a verifier to check intermediate step correctness, or that generates multiple candidate chains and selects the one with consistent step-by-step logic, could filter out invalid-reasoning completions and improve process-outcome alignment.
Self-correction failures in Sudoku (Section 6.4). Veo-3 exhibits self-reflective edits that sometimes help but often introduce new violations. A test-time strategy that explicitly verifies constraint satisfaction after each editing step (using a symbolic Sudoku solver rather than visual judgment) could prevent the cascade of errors documented in Figure 8.
Trajectory alignment failures in SLAG (Section 13.3). Video models fail to align the 2D trajectory projection with 3D motion. A test-time strategy that generates the 2D map and 3D navigation separately with explicit alignment constraints could address this.

What Evidence Exists in the Paper. The paper's evaluation protocol (Section 4.2) specifies: "We strictly adhere to the default API parameters... and recommended configurations... to guarantee a fair, zero-shot comparison without task-specific fine-tuning." This is a deliberate design choice. The paper provides no experiments with chain-of-thought prompting, self-consistency, majority voting, verifier-guided selection, or any other test-time compute scaling technique. The action reflection metrics (Maze Action Reflection, Sudoku Action Reflection, Math Action Reflection) measure whether models naturally exhibit self-corrective or exploratory behavior, but they do not test whether explicitly prompting for such behavior would improve performance.

Section 15 (Conclusion) gestures toward the need for "architectures that decouple reasoning states from visual rendering and integrate auxiliary objectives for causal consistency"—this acknowledges that architectural innovation, not just test-time strategies, may be needed. But the paper does not distinguish between "capability is absent" and "capability is present but not elicited by the evaluation protocol."

Mitigation Status. Not addressed. The paper's zero-shot protocol is a legitimate methodological choice—it establishes a clean baseline and enables fair comparison across models—but the limitation is that the baseline may substantially underestimate models' reasoning capabilities if those capabilities require appropriate test-time scaffolding to manifest. The paper does not discuss this tradeoff, does not suggest follow-up experiments with inference-time techniques, and does not qualify its negative conclusions with the caveat that they apply specifically to zero-shot generation. For a practitioner deciding whether to deploy these models for reasoning tasks, the distinction between "model X cannot reason" (fundamental limitation) and "model X does not reason under zero-shot generation but might reason with appropriate test-time compute" (protocol limitation) is practically crucial.

This limitation is particularly significant because the paper positions itself as characterizing the "reasoning deficits of modern generative models" (Section 1, abstract) in general, not just their zero-shot generation behavior. The stronger claim ("these models cannot reason") is not warranted by the zero-shot-only evidence, and the paper does not acknowledge this gap between claim scope and experimental scope.

The Benchmark Scope Is Limited to Tasks with Objective Correctness Criteria, Excluding Open-Ended Reasoning

The Assumption or Constraint. Every task in MMGR has a well-defined, objectively verifiable correctness criterion: whether the maze path reaches the goal without crossing walls, whether the Sudoku grid satisfies digit-uniqueness constraints, whether the ARC-AGI output exactly matches the ground-truth grid, whether the math solution produces the correct numeric answer, whether the navigation trajectory ends in the specified destination region, and whether the physics video obeys identifiable physical laws. This design choice enables the binary, strict-conjunction evaluation that is central to MMGR's methodology—every metric is a 0/1 judgment, and overall success requires all sub-metrics to be simultaneously satisfied.

The paper explicitly grounds this design in the need to avoid the "illusion of competence" where partial success masks fundamental failure (Section 4.3): "We prioritize this holistic measure to address the disparity between partial success and complete correctness—a gap that typically inflates performance estimates by 1.2–4× when ignored." The reliance on objective correctness is therefore a deliberate feature of the benchmark, not an oversight.

However, this reliance on objective correctness limits the benchmark to tasks where "correctness" is well-defined in a symbolic or geometric sense. Many reasoning tasks of practical importance—explaining a scientific concept, constructing a persuasive argument, generating a coherent narrative, performing creative problem-solving with multiple valid solutions—do not admit binary correctness judgments. Extending MMGR-style evaluation to these domains would require fundamentally different metrics (e.g., human preference judgments, rubric-based scoring with inter-annotator agreement) that are incompatible with the strict-conjunction framework.

The Consequence. The paper's conclusions about which reasoning abilities models possess and which they lack are necessarily conditioned on the specific operationalization of "reasoning" that the benchmark tasks embody. A model that fails MMGR's Maze task (because it cannot maintain a valid path without wall-crossing) has demonstrably failed at that specific reasoning challenge. Generalizing this to "the model lacks 2D spatial reasoning" or "the model cannot perform logical deduction" requires the additional assumption that the task validly measures the underlying ability—an assumption that the paper grounds in cognitive science (Spelke & Kinzler, 2007; Lake et al., 2017) but that is inherently limited by the narrowness of the task set relative to the breadth of human spatial and logical reasoning.

Specifically, the following forms of reasoning are systematically excluded or underrepresented:

Probabilistic reasoning. All MMGR tasks have deterministic answers. There is no task testing whether models can reason about uncertainty, generate calibrated confidence estimates, or make decisions under incomplete information—abilities central to real-world reasoning.
Causal reasoning beyond physical dynamics. Physical Commonsense tests causal reasoning about object interactions, but there is no task testing causal reasoning about social situations, historical events, economic mechanisms, or other non-physical causal domains.
Moral reasoning, ethical reasoning, value-laden reasoning. These are entirely absent from the benchmark.
Creative reasoning with multiple valid solutions. Tasks like "design a chair that is both comfortable and aesthetically pleasing" or "write a poem that conveys melancholy" require reasoning but have no unique correct answer. MMGR excludes these entirely.
Multi-turn, interactive reasoning. All tasks are single-turn: the model receives a prompt and generates an output. There is no evaluation of whether models can engage in dialogue, incorporate feedback, or revise their reasoning based on new information across multiple interactions.

What Evidence Exists in the Paper. The task designs are documented in Sections 5–14. The mapping of tasks to reasoning abilities is provided in Table 1. The paper does not discuss the exclusion of the reasoning types listed above, nor does it claim that MMGR covers all forms of reasoning. The five-ability framework is presented as a comprehensive taxonomy of the abilities needed for "robust world simulation" (Section 1), not for general intelligence. The paper's scope is explicitly bounded: "Our benchmark targets this dichotomy to enable fine-grained diagnosis of model capabilities: abstract tasks like Sudoku and ARC-AGI probe 2D grid-based logic, whereas embodied navigation tasks demand coherent 3D spatial understanding."

Mitigation Status. Acknowledged implicitly through the scope definition. Section 1 frames the five abilities as constituting the requirements for world modeling, not for general reasoning. The paper does not claim that MMGR evaluates all reasoning abilities, and the explicit mapping of tasks to abilities (Table 1) makes clear which abilities are tested and which are not. Section 15 (Conclusion) suggests that future work should extend the evaluation framework, but does not specify which reasoning types are priorities for inclusion.

For a practitioner, the limitation is practical: if you care about a model's reasoning capabilities in a domain not represented in MMGR (e.g., legal reasoning, medical diagnosis, strategic planning), MMGR provides no direct evidence. The indirect evidence—that models show systematic failures on structured symbolic reasoning tasks—suggests caution in deploying these models for other reasoning-intensive applications, but the cross-domain generalization of MMGR's findings is not established.

This limitation is less severe than the others discussed above because it is inherent in the benchmark format and is partially mitigated by the clarity of the task-to-ability mapping. A practitioner can read the MMGR results as characterizing reasoning capabilities specifically within the domains tested, without assuming coverage of all reasoning types. The paper does not overclaim on this point.

7. Implications and Future Directions

How This Work Changes the Landscape

MMGR does not introduce a new model or training method—it changes the field by changing what we measure, and therefore what we will optimize for. The paper's core reframing is to treat "reasoning correctness" as a dimension of generative model evaluation that is orthogonal to visual quality, supported by extensive evidence that the two can vary independently (Wan-2.2's 84–96% Visual Realism with 24% Overall on Physical Commonsense, Table 35; Veo-3's 74% Outcome Success but 12% Process Success on GSM8K, Table 20). This is not an incremental improvement in evaluation methodology—it is a diagnostic intervention that redirects attention from "how good does it look?" to "does the model understand what it is generating?"

Prior to MMGR, the dominant evaluation paradigm for generative models was quality-centric: FVD, IS, CLIP-score, and VBench-style multi-dimensional quality metrics all measure some variant of "this looks like a real video/satisfies the prompt visually." The implicit assumption—visible in the "world simulator" framing used by major labs (OpenAI, 2024b; DeepMind, 2024)—was that models sophisticated enough to produce photorealistic video would, by virtue of their scale and the richness of their training data, internalize the causal, physical, and logical constraints governing that video. MMGR's results systematically falsify this assumption. A model can produce visually flawless billiard shots where balls interpenetrate, or mathematically correct answers derived through invalid visual morphing, because the training objective rewarded visual accuracy, not causal accuracy.

The practical consequence of this reframing is that evaluation will increasingly need to include reasoning diagnostics alongside quality metrics. The paper's finding that partial success inflates performance estimates by 1.2–4× (and up to 25× in the worst navigation cases, Section 9.4) means that ignoring process-level evaluation produces systematically misleading claims about model capability. A lab reporting 74% on GSM8K based on answer accuracy alone (as is standard in LLM evaluation) would miss that Veo-3's process success is only 12%, and that the model is pattern-matching answers rather than reasoning. MMGR establishes that process-level evaluation is not a luxury for detailed analysis—it is a necessity for truthful capability assessment.

MMGR also resolves a genuine contradiction in prior work about whether video models possess emergent reasoning capabilities. Wiedemer et al. (2025) identified "Chain-of-Frames" reasoning in Veo-3, suggesting nascent world modeling. Tong et al. (2025) found competitive performance by Sora-2 on vision tasks when using video generation as a reasoning paradigm. Guo et al. (2025) found both strengths and specific geometric consistency failures. These findings appeared contradictory because different papers tested different tasks with different metrics. MMGR provides the unified framework that reconciles them: the answer is not a binary "yes, video models can reason" or "no, they cannot," but a domain-dependent capability profile. Video models show genuine physical reasoning capability (Sora-2 70% Overall on Physical Commonsense, Table 35) and ego-centric spatial perception (Veo-3 93.3% Physical Understanding on Panoramic Last-Mile Navigation, Table 24), but collapse on abstract symbolic reasoning (Veo-3 <7% Overall on Sudoku, Table 6; <5% on ARC-AGI v1, Table 9) and long-horizon spatial planning (Veo-3 ~3% Holistic Overall on SLAG, Table 25). The apparent contradiction in prior work reflected task-specific evaluation without a shared taxonomy; MMGR's five-ability framework resolves it by mapping specific abilities to specific tasks, revealing that video models have islands of genuine capability surrounded by seas of pattern-matching.

The paper also changes the landscape for evaluation methodology itself. The systematic comparison of AutoEval against HumanEval across multiple tasks (Tables 5, 7, 16, 17, 25, 27, 29, 31, 33, 38, 39, 40) establishes that VLM-based evaluation for generative reasoning is not a solved problem—it introduces systematic, task-dependent, and metric-dependent biases of 2–5× magnitude. For maze navigation, AutoEval overestimates overall success by 3–5× (Table 5: 42–52% Auto vs. 0–20% Human). For physical commonsense, it underestimates success by ~1.6× (Table 38: 51% Auto vs. 80% Human). The direction of error varies with the type of metric (temporal violations are missed; color precision is over-penalized; action reflection is under-detected). This finding raises the bar for evaluation rigor across the field: a benchmark that reports VLM-based evaluation without human calibration is, based on these results, reporting numbers of unknown accuracy for its most diagnostically interesting metrics. MMGR models a more rigorous approach—side-by-side presentation of AutoEval and HumanEval—that should become standard practice for generative reasoning evaluation.

The research directions that become more attractive in light of MMGR include: (1) training generative models with auxiliary objectives that reward logical consistency and physical accuracy, not just visual fidelity—the paper's finding that current objectives systematically reward the wrong thing makes this a high-priority intervention; (2) developing architectures that decouple reasoning state from visual rendering, given the evidence that temporal processing in current video models actively corrupts logical constraints; (3) building more robust automated evaluators that can detect the transient violations (wall-crossing, context drift, physical impossibility) that current VLMs miss; and (4) constructing training datasets with proportionally more structured symbolic reasoning content, given the domain asymmetry (video models do well where training data is rich—physical dynamics, sports—and poorly where it is scarce—abstract logic, symbolic manipulation) that the paper documents.

Directions that become less attractive include: (1) pursuing ever-larger video generation models under the assumption that scale alone will produce reasoning capabilities—the paper shows that Sora-2 and Veo-3, among the largest video models ever built, still exhibit pattern-matching rather than genuine reasoning on abstract tasks, and that Sora-2's ARC-AGI performance collapses 93% from v1 to v2 (Tables 9, 10), indicating memorization rather than generalization; (2) relying on VLM-based evaluation without task-specific human calibration for reasoning benchmarks—the evidence that AutoEval bias is large and directionally inconsistent undermines the practice of reporting AutoEval-only results as ground truth; and (3) treating "world simulator" claims at face value without diagnostic reasoning evaluation—MMGR provides a concrete, replicable methodology for stress-testing such claims, and the results suggest current models are far from the threshold where such claims are warranted.

Follow-Up Research This Work Enables

Architectures that decouple reasoning state from visual rendering. The paper's central negative result—the "temporal tax" where video generation actively competes with logical consistency—identifies a specific architectural bottleneck. Current video models generate each frame conditioned on previous frames' visual features, meaning that logical constraints (digit uniqueness in Sudoku, static demonstration examples in ARC-AGI, obstacle locations in Maze) are represented only implicitly in pixel-space and subject to visual drift. A direct follow-up would be to design a video generation architecture with an explicit, persistent symbolic state that is maintained independently of the visual stream—for example, a transformer that generates a symbolic representation of the solution (digit grid, path plan, transformation rule) in a first pass, then renders that representation as video frames in a second pass, with the symbolic state remaining invariant across frames. The Sudoku results provide a clean testbed: the model should never overwrite clues or introduce constraint violations during rendering because the symbolic state is immutable. Performance on MMGR's Sudoku task (currently <7% Overall for video models, Table 6) would be the primary evaluation metric.

Training data interventions to correct the symbolic-physical imbalance. The paper documents a sharp domain asymmetry: video models perform reasonably on Physical Commonsense (Sora-2: 70%, Veo-3: 51%) but near-zero on abstract symbolic reasoning tasks (Sudoku, ARC-AGI, Math process correctness). The post-hoc explanation—that video training corpora are rich in physical dynamics but poor in structured symbolic reasoning data (Section 15)—is plausible but untested. A direct follow-up would train video models on datasets where the proportion of symbolic reasoning content (maze-solving trajectories, Sudoku completion sequences, step-by-step math derivations rendered as video) is systematically varied and measure MMGR performance. The key experiment compares two models with identical architecture and total training FLOPs, but with 10% vs. 50% symbolic reasoning data in their training mix. If the symbolic-enriched model's abstract reasoning improves without degrading physical reasoning, the data imbalance hypothesis is supported and a concrete recipe for data curation is established. If abstract reasoning does not improve, the bottleneck is architectural rather than data-driven.

Test-time compute scaling for generative reasoning. The paper evaluates all models in a strict zero-shot, single-pass generation setting (Section 4.2). This leaves open the critical question: are the documented reasoning failures fundamental capability gaps, or gaps that can be closed with inference-time computation? The hallucination of competence pattern—Veo-3 achieving 74% Outcome Success but 12% Process Success on GSM8K (Table 20)—is particularly amenable to test-time intervention. A direct follow-up would apply best-of-N selection with a process-reward model (PRM) that scores solution steps for logical validity, selecting the generated video with the highest intermediate correctness score rather than the first sample. The baseline is Veo-3's 12% Process Success on GSM8K; the target to beat is Nano-banana Pro's 97.83%. If best-of-N with PRM selection can close even half the gap, it would establish that the capability is latent but unexpressed in zero-shot generation—a practically important distinction for deployment. A complementary experiment would test whether explicit chain-of-thought prompting ("show each step of the derivation, checking constraint satisfaction at every step") reduces context drift in ARC-AGI and Sudoku.

Multi-evaluator calibration to establish whether AutoEval bias is evaluator-specific or fundamental. The paper's human evaluation calibration is performed exclusively with Gemini-2.5-Pro as the AutoEval model. The documented biases—temporal resolution limitations, color sensitivity, failure to detect subtle exploratory behaviors—could be specific to this Gemini version or could be symptomatic of fundamental limitations in current VLM architectures for temporally dense evaluation. A direct follow-up would replicate the AutoEval-HumanEval comparison on a subset of MMGR tasks (Maze, ARC-AGI, Physical Commonsense) using GPT-4V, Claude 3.5, and an open-weight VLM (e.g., LLaVA-NeXT) as alternative evaluators. If all evaluators show similar bias patterns (all miss transient wall-crossings; all underestimate Action Reflection), the limitation is fundamental to current VLM capabilities and improved evaluation requires architectural innovation in evaluator models. If some evaluators are substantially more accurate on specific metrics, the finding guides evaluator selection for future benchmarking and establishes that evaluation reliability is an evaluator-model-choice problem rather than a fundamental unsolved problem.

Cross-domain generalization of MMGR's findings to code, audio, and 3D generation. MMGR's scope is limited to video and image generation, but the core diagnostic—separating process correctness from outcome correctness, measuring difficulty-dependent scaling curves, calibrating automated evaluation against human judgment—is modality-agnostic. A direct follow-up would adapt the benchmark's methodology to code generation from video (generating an animation that visualizes an algorithm's execution), audio generation (generating a sequence of notes that follows harmonic rules), or 3D scene generation (generating a geometrically consistent multi-room environment). The key question is whether the temporal tax, the hallucination of competence, and the decoupling of quality from correctness replicate across modalities. For code visualization, one could adapt MMGR's Maze task to a "visualize this sorting algorithm" task where the generated video must correctly animate each step of the sort—directly testing whether video models can maintain algorithmic constraints across frames.

Adversarial difficulty design to stress-test the boundary between generalization and memorization. The ARC-AGI v1-to-v2 comparison (Tables 9, 10) provides a template but conflates difficulty shift with novelty shift. A direct follow-up would construct a "v3" ARC-AGI benchmark where difficulty is held constant relative to v1 (same distribution of grid sizes, color counts, object counts) but the transformation rules are systematically novel—designed to be structurally analogous to v1 rules without being retrievable through pattern matching. If Nano-banana Pro's performance remains stable (30.54% → ~30%) while Sora-2 collapses further, the memorization-vs.-generalization diagnosis is strengthened. If Nano-banana Pro also collapses (e.g., to <10%), its apparent robustness was partly memorization of v1-specific rule families, and the genuine generalization ceiling is lower than MMGR currently estimates. This stress-test is critical because it establishes the true reasoning frontier—the difficulty beyond which even the best current models are pattern-matching rather than rule-inducing.

Practical Applications and Downstream Use Cases

Screening models before deployment as simulators or planning backends. Organizations building robotic planning systems, scientific visualization tools, or educational content generators on top of video generation models can use MMGR's task suite as a diagnostic gate. The finding that Veo-3 achieves 70% Target Achievement on maze navigation but 0–20% Overall under human evaluation (Table 5) because it systematically crosses walls demonstrates that raw success rates on navigation-like tasks are dangerously misleading. A robotics team considering Veo-3 for trajectory planning should run the MMGR Embodied Navigation subtasks specific to their deployment environment (Last-Mile Navigation for short-range manipulation planning; 3D Real-World Navigation for multi-room navigation) and require not just high Success Score 3D but high Overall scores (with Scene Consistency and Destination Integrity metrics satisfied) before trusting the model's outputs. The benchmark's 24-configuration hard-level design (Section 9.2) provides a template for constructing environment-specific validation suites: vary environmental complexity to match the target deployment, test both visual and textual goal specifications to match the command interface, and calibrate with human evaluation if AutoEval bias in the specific metric regime is unknown.

Guiding training data curation for improved reasoning. The domain asymmetry documented in MMGR—video models performing well on tasks with abundant training data (Sports: Veo-3 60%, Sora-2 64%, Table 35) and poorly on tasks with scarce training data (Sudoku <7%, ARC-AGI <10%)—provides a data-driven signal for training data curation teams. A lab training the next generation of video models can use MMGR's task-level scores as a diagnostic for which reasoning abilities their current training data mixture supports and which it neglects. If a model achieves >50% on Physical Commonsense but <5% on Sudoku, the training data likely lacks structured symbolic reasoning content. The concrete intervention is to synthetically generate video training examples for the underrepresented reasoning types—for instance, rendering Maze solution animations, Sudoku completion sequences, and step-by-step math derivations as training videos using programmatic generation pipelines—and incorporate them into the next training run. The target is not to solve MMGR (which would constitute benchmark contamination if done naively) but to shift the model's capability profile toward the balanced five-ability profile that world modeling requires. The performance improvement on held-out reasoning tasks from the enriched training distribution would quantify the causal effect of training data composition on reasoning capability.

Cost-efficient batch evaluation for model selection. For organizations comparing generative models for procurement or deployment decisions, MMGR provides a more diagnostic alternative to standard quality metrics at comparable evaluation cost. The automated VLM-based evaluation pipeline (Section 4.3) can process thousands of generated samples without human intervention, producing fine-grained reasoning profiles that a quality-only benchmark (FVD, VBench) cannot provide. A media company choosing between Veo-3 and Sora-2 for generating sports highlight animations would see from Table 35 that Sora-2 achieves 70% Overall on Physical Commonsense (vs. Veo-3's 51%), with particularly strong Physics Accuracy (86% vs. 71%)—a directly actionable signal that Sora-2 is more likely to produce physically plausible sports videos. The key caveat from the paper's own findings is that the AutoEval numbers need calibration: for Physical Commonsense, HumanEval rates Veo-3 substantially higher than AutoEval (80% vs. 51%, Table 38), so the decision-maker should commission a small human evaluation on their specific use case to verify the AutoEval rankings before committing. The benchmark's task diversity enables targeted evaluation: only run the tasks relevant to the deployment domain (e.g., Sports for a sports application, Embodied Navigation for a robotics application) rather than the full 1,853-sample suite.

Benchmarking evaluation methodologies and evaluator models. The paper's finding that VLM-based evaluation is systematically unreliable for reasoning assessment—with bias magnitudes of 2–5× and directions that vary by task (Section 5, human evaluation tables)—creates a new use case for MMGR as a meta-benchmark for evaluation quality. A team developing an improved automated evaluator (e.g., a VLM fine-tuned for temporal violation detection, or a multi-frame reasoning architecture) can use MMGR's human-annotated subsets as ground truth to measure their evaluator's accuracy on specific failure modes. The Maze task's human-annotated Cross Wall labels (Table 5) provide a testbed for evaluating whether an improved VLM can detect the transient wall-crossing events that Gemini-2.5-Pro misses. The ARC-AGI human-annotated Valid Solution labels (Tables 16, 17) test whether an evaluator can distinguish visually plausible but logically incorrect outputs from exact matches. The target metric is not model performance but evaluator-model agreement with human judgments—a direct measure of evaluation reliability. An evaluator that achieves >90% agreement with human annotators on Maze Cross Wall and ARC-AGI Valid Solution would represent a meaningful advance in automated reasoning assessment and could be deployed with greater confidence for large-scale benchmarking where human evaluation is infeasible.